Updates to pre-built Security ML jobs

[ajosh0504]

Summary

This PR makes the following updates to the pre-built Security ML jobs:

• Making the security-packetbeat compatible with Agent
• Removing superfluous fields from the job configurations to make them consistent
• Updating the detector_description field for almost all jobs
• Adding influencers where missing and/or relevant
• Adding a job_revision custom setting similar to the Logs jobs. Moving forward, this number will be updated each time a job is updated. We are starting with 4 since the linux and windows jobs are at v3 right now
• Adding a managed: true tag to indicate that these jobs are pre-configured by Elastic and so users will see the warnings added in this PR if users choose to delete, or modify these jobs

[pjhampton]

Looks good from a security data analytics perspective

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[peteharverson]

Added a few suggestions for minor edits to job and detector descriptions, but otherwise overall looks good.

Tested creation of most of these jobs in the ML UI:

• Security: Authentication
• Security: Cloudtrail
• Security: Linux
• Security: Packetbeat
• Security: Windows

[ajosh0504]

@elasticmachine merge upstream

[ajosh0504]

Tested the Packetbeat jobs locally to ensure that the datafeed update is working as expected:

• Posted documents as follows into a test index

- Noted that the datafeed for the `packetbeat_rare_dns_question` job recognized the two documents corresponding to DNS datasets:

[peteharverson]

@ajosh0504 as mentioned, note that the purpose of these job_tags properties was to allow searching on these properties in the ML jobs list, for example for jobs with specific versions

or specific euid:

and these tags are also displayed in a dedicated section in the expanded job row:

See original issue, and the support that was added for them in 7.14: #102099

Note that the Logs UI use an alternative approach for versioning jobs - https://github.com/elastic/kibana/blob/main/x-pack/plugins/ml/server/models/data_recognizer/modules/logs_ui_categories/ml/log_entry_categories_count.json#L38, but this job_revision property is not searchable in the ML job management page.

I am ok with you removing these job_tags if they are no longer providing value, but wanted to provide context on why we originally added them to many of the security jobs.

[ajosh0504]

@peteharverson Thanks for providing context. My thoughts on the above:

• Afaik we aren't using these fields internally.
• In terms of searching in the UI, I'm not sure how much of a value searching for v3 jobs is providing. Typically, we see our users upgrading to the latest version of jobs, rather than keeping both old as well as new jobs. Also having versions only for some jobs is a bit odd, given that other jobs (that don't have the version tags) have also undergone updates. I would like to have some versioning for all jobs, but I'd like to discuss this further outside of this PR, given that we are unofficially already several versions along across all our modules.
• I don't understand the value provided by the euid field, specifically given that it is retained when the job is cloned. I'd understand if it was a unique identifier for a job so as to distinguish between different copies of the same job, but that not being the case, I don't see a ton of value in it.

[peteharverson]

I'm happy to defer to you on this one @ajosh0504 and remove these job_tags.

The thinking behind euid was that this field couldn't (easily) be changed by the user, whereas they are able to completely change the job ID when cloning, so it could be used to track how many of this type of job had been created.

Makes sense to address a way to version all the security jobs outside of this PR.

[peteharverson]

Changes look good for ML.

[ajosh0504]

@pheyos Tagging you to see if anything at y'all's end needs updating given the changes in this PR.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 101bfd7

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	432	435	+3

Total ESLint disabled count

id	before	after	diff
securitySolution	512	515	+3

History

• 💛 Build #119538 was flaky 58ce02a
• 💚 Build #119096 succeeded f4f5987
• 💛 Build #118461 was flaky 8655ab2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ajosh0504

[pheyos]

Thanks for the heads-up @ajosh0504! Indeed, we'll have to do some follow-up adjustments to our tests in the ML QA framework once your PR is merged , I'll take care of that.