New Filebeat subcommand: generate

[kvch]

This PR exposes the existing Filebeat module generators as generate subcommand.

$ ./filebeat generate -h
Generate Filebeat modules, filesets and fields.yml

Usage:
  filebeat generate [command]

Available Commands:
  fields      Generates a new fields.yml file for fileset
  fileset     Generates a new fileset
  module      Generates a new module

The subcommands module, fileset and fields use the same code as the scripts.

I added E2E tests to validate the functionality.

Blocked by #9147 as it contains the refactorings of that PR.

[ph]

Do we really need to expose both, because by default both of the flags will point to the same place?

[kvch]

Hmm... Not yet. Btu when community Beats will support a similar module-handling, we will need this.

[kvch]

jenkins test this

[ph]

LGTM, WFG

[kvch]

Rebased

[ruflin]

If we expose this to the user, I think we should also add some docs on how we expect the users to use this. How will the user handle the new fields?

The PR has needs_backport label. Is there a reason we need this in 6.x?

[kvch]

👍 on the docs, that's why the PR has the needs_docs label. What do you mean first question?

Obviously, we don't "need" it, but we planned to release it in 6.x. Why do you think it is problem?

[ruflin]

A user creates a module with this command, how will he load / configure the additional fields that are used for it? If the user upgrades to a newer version, will the module stay intact? What if he wants the module on multiple machines?

[kvch]

I am still not sure if I follow you. The behaviour of the generators has not changed. In this PR I only added new commands and subcommands to Filebeat to make existing functionality accessible to more people.

After a user generates a module and a fileset, he/she can add the fields of the fileset to the appropriate fields.yml file. generate fields is not able to produce a fields.yml which is 100% guaranteed to match the actual fields. It is just a "crutch" for a user to get started with the file. The generator explicitly asks the user to check the result. It also generates fields with the description which need to be filled in.

Why wouldn't the module stay intact during an upgrade? If the name of the user's module is the same as a new module coming with the new release, it gets overwritten. But this has been the behaviour since introducing module generators in Filebeat.

If he/she wants to have this module on multiple hosts, it is possible to copy them as before. Nothing changes here, also.

[andrewkroh]

Nice addition to Filebeat. An accompanying filebeat test module command that uses the _ingest/pipeline/_simulate API to test the pipeline and validate the resulting document would also help make module development more accessible.

[ruflin]

So far this functionality was only accessible to devs in the Beats repo and was not really intended for production but more for module development. With the above we make it accessible to all users and production use cases.

One question I don't fully understand yet is what happens with the fields.yml. If the above commands are used in a dev environment and make update is run, they are put into the global fields.yml file and fields.go inside the binary. If someone uses the above command outside the dev environment, he gets the fields.yml in the module (which we don't ship) but AFAIK it's not really usable and will not have any effect on the template.

To be clear: I really like to make the above commands available to all our users but I want to make sure they have the experience they expect when they run the commands.

[kvch]

Some parts of make update are not yet ported to Golang. So it is not yet possible to generate everything for a module.
@ruflin I am OK with putting this PR on hold until all generators can be exposed. WDYT?

[ruflin]

@kvch Even if mage update would be all in Go, there would be still an issue here. The reason is that by default even if there is a fields.yml, the fields from the binary will taken. Even if a user adds his fields to the global fields.yml, it will not read by default.

To keep this moving forward few ideas:

1. We recommend users to copy their fields to append_fields
2. We tell users that fields.yml does not work yet
3. We introduce some logic in the module that can read the local fields.yml in the module and append it to the template if it is there.

To get this PR in quickly, we could start with 2 and see what the user feedback will be and work on 3. I personally like 3 because it will make modules "just work".

[kvch]

@ruflin Indeed. I like what you have proposed in option 3. I will open a new PR with the changes which lets users read their local fields.yml file. That is a nice addition on its own.

[kvch]

I only need to investigate why the test cannot find the test pipeline on Windows.... >.<

[kvch]

jenkins test this

[kvch]

jenkins test this

[kvch]

Failing tests are unrelated.

[kvch]

Needs backport to 7.x branch...