[CI] Set MODULE per relevant stage

[ycombinator]

Follow up to #18592. In #18592, we set the MODULE environment variable at a global scope. As such, once the MODULE is determined and set, it gets used by all stages of the Jenkins CI pipeline. This can cause errors when the value of MODULE isn't valid for a a stage (e.g. X-Pack Filebeat doesn't implement the module specified in MODULE).

One fix would be to move the setting of MODULE from the global scope into each relevant stage of the Jenkins CI pipeline. We would only set it in those stages that implement Beats modules, viz. OSS Filebeat, OSS Metricbeat, OSS Auditbeat, X-Pack Filebeat, X-Pack Metricbeat, X-Pack Auditbeat, and X-Pack Winlogbeat. In each stage, the pattern used to determine whether module files have changed would need to be scoped to files relevant to that stage, e.g. x-pack/filebeat/module/* for the X-Pack Filebeat stage.

[v1v]

@ycombinator, got a question regarding the description:

(e.g. X-Pack Filebeat doesn't implement the module specified in MODULE).

..., OSS Auditbeat, X-Pack Filebeat, X-Pack Metricbeat,

There is a contradiction with X-Pack Filebeat as far as I understood.

BTW, I just coded some changes in the CI Pipeline, to explicitly enable the MODULE env variable in the above relevant stages, the stages I did enable are the ones that are running the tests targets, please let me know if I missed any other stage:

#18777

See the changes with , withModule: true)

[ycombinator]

There is a contradiction with X-Pack Filebeat as far as I understood.

Ah, sorry for the confusion. X-Pack FIlebeat does have modules so the MODULE environment variable should be set for it's build stage.

I was trying to give an example of the current state of affairs where the MODULE is being set at the global level. Today, MODULE could get set to a value that is invalid for certain stages. Using the same example that @jsoriano gave in #18652

For example in this build MODULE seems to be set to kafka, what is correct because a test is modified in the kafka module in OSS metricbeat, but x-pack metricbeat build fails because it doesn't have a kafka module.

I was just trying to illustrate the same situation, but with X-Pack Filebeat and OSS Filebeat instead of X-Pack Metricbeat and OSS Metricbeat in @jsoriano's example. Sorry for causing confusion!

[jsoriano]

Another example of this happened in #18802, a change for googlecloud module triggered all Metricbeat builds, but only x-pack Metricbeat has this module, then in OSS Metricbeat:

Error: no module googlecloud