[ML] Update Security ML jobs to use entity analytics fields for host and user fields by sodhikirti07 · Pull Request #255339 · elastic/kibana

sodhikirti07 · 2026-02-27T17:12:12Z

Summary

This PR updates the Security ML jobs and datafeeds to use Entity Analytics fields that are supported by the Entity store.

Changes included:

Renames the ML jobs and datafeeds, adding a suffix of "_ea".
Add EA fields for host and user influencers
Modified the manifest.json files within the respective modules to reflect these changes.

Testing:
Tested the jobs in Kibana with test data

Checklist

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

jmcarlock

Requesting changes to block merging ATM

…fest.json

sodhikirti07 · 2026-03-19T13:43:08Z

@rylnd @abhishekbhatia1710 @dplumlee Could you please review this PR for the changes relevant to your team?

ymao1

Entity analytics changes LGTM

rylnd

With the understanding that requiring users to update their ML Rules/Jobs in 9.4 is an intentional choice, here: this looks good to me! For DE code this is mostly just a find/replace for those job_ids, which is a pretty straightforward review.

I did have one note on an anomalies archive that has a few job references that weren't updated, but other than that: LGTM

rylnd · 2026-03-19T16:18:45Z

x-pack/solutions/security/plugins/security_solution/common/machine_learning/affected_job_ids.ts

+// Note: In 9.4 the V3 jobs were replaced by Entity Analytics (_ea) variants that use
+// Entity Analytics fields (host.id, user.id, event.module, etc.) as influencers.
+// V3 non-EA jobs are now also considered affected.
 export const affectedJobIds: string[] = [


Thanks for updating this! I verified that users will continue to see the "new jobs available" callout in these new v3 cases 👍

Note: I think this could be more accurately called:

Suggested change

export const affectedJobIds: string[] = [

export const outdatedJobIds: string[] = [

And the description should be updated to reflect this new purpose, but: out of scope for this PR 👍

I've made a note (created a SecML issue) to review how this affect_job_ids file works as we now have more precedences - but for now will leave this const as-is, thanks for pointing it out!

rylnd · 2026-03-19T16:28:28Z

x-pack/solutions/security/test/fixtures/es_archives/security_solution/anomalies/mappings.json

        "is_hidden": true
      }
    },
-    "index": ".ml-anomalies-custom-v3_linux_anomalous_network_activity",


This mappings file (and corresponding data.json.gz) also contains references to

"index": ".ml-anomalies-custom-rare_process_by_host_linux_ecs",

which from my understanding is also migrating here to rare_process_by_host_linux_ea.

Do you want to change all of these references, or leave some "old" job data? I could see arguments either way; just want to make sure this is intentional/desired.

Thanks for flagging this Ryland, I will test this again; my original intent was to change all references, but now I'm wondering if this isn't actually all used by the test cases, as the tests pass with the stuff that I did change (as of the last change on the tests)

So let me verify... if it's just that the test files have some extra info/mappings/fields, etc. that aren't actually used, then I may leave it for now 🤔 but depends on what I find

We'll keep you posted

I think it’s fine to keep the reference rare_process_by_host_linux_ecs as is, since it appears to be older than the current version anyway.

I don't recall if that data is used in any test (except perhaps to verify that rules are properly partitioning by job_id), but: I agree that having the "old" data is good and representative of what actual users would have, so I'm all for keeping it as is 👍

For learning purposes I went to check the history and confirmed that this PR's change to this file follows the same patterns as the last time the jobs were changed in this file

And on the side, as you mentioned Kirti the last time this file was changed the rare_process_by_host_linux_ecs wasn't changed

So will leave it as is! I do appreciate the comment, Ryland, as you've worked with these files before

spong

Just a single test file change on the SecurityGenAI side -- LGTM! 👍

peteharverson · 2026-03-19T16:59:52Z

x-pack/solutions/security/plugins/security_solution/common/machine_learning/affected_job_ids.ts

+//
+// Note: In 9.4 the V3 jobs were replaced by Entity Analytics (_ea) variants that use
+// Entity Analytics fields (host.id, user.id, event.module, etc.) as influencers.
+// V3 non-EA jobs are now also considered affected.


If the change below is accepted, should this comment read V3 non-EA jobs are now also considered outdated.?

For now we'll leave this file as-is, but I've created an internal issue to re-examine how this file works as there's now more precedence of replacing/deprecating ML jobs. We can discuss if we can make this affected/outdated jobs file cleaner

peteharverson · 2026-03-19T17:33:24Z

.../server/models/data_recognizer/modules/security_auth/ml/auth_high_count_logon_events_ea.json

    "created_by": "ml-module-security-auth",
    "security_app_display_name": "Spike in Logon Events",
    "managed": true,
    "job_revision": 4


Is it worth incrementing the job_revision field for all the jobs edited in this PR? I don't know if it's used in any telemetry collected by Security? On the ML side we only use the created_by field for usage tracking of the jobs, and not currently job_revision.

Thanks for mentioning this! I think that for this PR, since we’re already changing the job_id, it might not be necessary to bump the job_revision. This field could be useful for notifying users of job updates, as it’s already used by Observability jobs in a similar process. We’ll open a separate investigation to explore enabling a similar process under security and leveraging this field for user notifications.

peteharverson

ML changes LGTM

florent-leborgne

LGTM for docs

nastasha-solomon

API docs lgtm

maximpn

Rule Management Area LGTM

elasticmachine · 2026-03-24T13:44:25Z

⏳ Build in-progress

Buildkite Build
Commit: 3f2fbdf
Storybooks Preview
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-255339-3f2fbdf881d9

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #8 / discover/context_awareness telemetry field usage events should track field usage on surrounding documents page
[job] [logs] FTR Configs #105 / machine learning - data visualizer - group1 field statistics in Discover when enabled with geo point field displays the 'Field statistics' table content correctly

History

💛 Build #413728 was flaky d3935c7
💔 Build #413121 failed f20f51a
💚 Build #412896 succeeded 7c9a4af
💚 Build #410890 succeeded c3886c4
💛 Build #409702 was flaky 59aeff0
💔 Build #409356 failed 91ce942

…ra/kibana into dashboard_align_attachment_to_api * 'dashboard_align_attachment_to_api' of github.com:mbondyra/kibana: (45 commits) [OTel Tracing] HTTP instrumentation (elastic#258663) Replace deprecated EUI icons in files owned by @elastic/ml-ui (elastic#255624) [Codeowners] add missing codeowners for security_solution_api_integration tests (elastic#259223) [CI] fix bad imports that came from a merge-race (elastic#259383) Add `.claude/worktrees/` to `.gitignore` (elastic#259192) Improve unknown-key validation error message in @kbn/config-schema (elastic#258633) [ML] Update Security ML jobs to use entity analytics fields for host and user fields (elastic#255339) [Table sweep] Update table columns responsiveness in Index Management and Dashboards (elastic#259340) skip failing test suite (elastic#258790) skip failing test suite (elastic#259261) chore: util to clean cached images (elastic#259335) [Entity Store] Use last_seen for automated resolution watermark (elastic#258574) [One Workflow] Fix flaky alert trigger Scout test by removing order-dependent assertions (elastic#259299) Skip serverless Discover request counts tests for MKI (elastic#259333) [Security Solution] render header title in new document flyout in Security Solution and Discover (elastic#258166) [Agent Builder] register inference endpoint feature (elastic#259259) [Agent Builder] Skills Command Menu - Add descriptions and scope options to agent (elastic#258964) [Streams][Streamlang][API] Fully use meta({id}) to reuse schema partials in OAS output (elastic#259275) fix(files_example): add tableCaption to EuiInMemoryTable for a11y (elastic#258289) [Entity Store] Adding list endpoint with query filter (elastic#258320) ...

…and user fields (elastic#255339) ## Summary This PR updates the Security ML jobs and datafeeds to use Entity Analytics fields that are supported by the Entity store. **Changes included**: - Renames the ML jobs and datafeeds, adding a suffix of "_ea". - Add EA fields for host and user influencers - Modified the `manifest.json` files within the respective modules to reflect these changes. **Testing:** Tested the jobs in Kibana with test data <img width="1512" height="776" alt="image" src="https://github.com/user-attachments/assets/ed290458-031d-4d85-a33a-00454efa3cb4" /> ### Checklist Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Susan Chang <shuhsuan.chang@elastic.co> Co-authored-by: Susan <23287722+susan-shu-c@users.noreply.github.com>

## Summary Update the list of influencers for Security ML jobs Related PR adding EA changes to ML jobs: #255339 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...

## Summary Update the list of influencers for Security ML jobs Related PR adding EA changes to ML jobs: elastic#255339 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...

… with EA changes (#5348)  ## Summary  - Updated the ML jobs ID, description to reflect the changes we made for the new [Entity Analytics (EA) in our ML jobs](elastic/kibana#255339). - Added Privileged Access Detection under Security Integrations Relates to: - elastic/security-ml#1044 These changes are staged for `9.4` release. ## Generative AI disclosure  1. Did you use a generative AI (GenAI) tool to assist in creating this contribution? - [ ] Yes - [ ] No  --------- Co-authored-by: natasha-moore-elastic <natasha.moore@elastic.co> Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com>

## Summary Update the list of influencers for Security ML jobs Related PR adding EA changes to ML jobs: elastic#255339 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...

sodhikirti07 added 8 commits February 25, 2026 16:23

add euid changes to security_auth jobs

c3e3432

updated job names in manifest for security_auth

d0338bb

updated azure jobs and manifest.json

b2bea94

updated cloudtrail jobs and manifest.json

793b01c

updated security host jobs and manifest.json

64882ca

updated security linux jobs and manifest.json

4f65695

updated security packetbeat jobs and manifest.json

9f5ae57

updated security windows jobs and manifest.json

d29ad73

sodhikirti07 added the backport:skip This PR does not require backporting label Mar 2, 2026

jmcarlock requested changes Mar 3, 2026

View reviewed changes

sodhikirti07 added v9.4.0 :ml Feature:Anomaly Detection ML anomaly detection Feature:Security ML Jobs Security Solution ML Jobs labels Mar 3, 2026

sodhikirti07 mentioned this pull request Mar 3, 2026

[Security] [Serverless: Mar 31] Update prebuilt ML jobs documentation with EA changes elastic/docs-content#5348

Merged

2 tasks

restore user.name and host.name to jobs influencers

02818c4

sodhikirti07 added the release_note:feature Makes this part of the condensed release notes label Mar 6, 2026

sodhikirti07 and others added 13 commits March 6, 2026 12:37

rename auth job

d127d7a

remove auth euid job suffix

e9cce7a

rename security windows

2fd0bee

Merge branch 'main' into update-ml-jobs-with-new-euids

765809d

update influencers

394a50e

rename security_auth jobs, datafeed, add influencers and update manifest

a453c01

update description of security_auth jobs

e1ba3df

rename security auth jobs, datafeeds, add influencers and update mani…

841207d

…fest.json

update security_cloudtrail jobs, datafeeds and manifest.json

d7f2e91

update security_gcp_audit jobs and datafeeds

88ff8fb

update datafeed job id for security auth, cloudtrail, azure

77a6d28

update security_host jobs, datafeeds

39d0c62

update security_linux jobs and datafeeds

23bdc06

jmcarlock approved these changes Mar 19, 2026

View reviewed changes

susan-shu-c approved these changes Mar 19, 2026

View reviewed changes

peteharverson self-requested a review March 19, 2026 14:53

ymao1 approved these changes Mar 19, 2026

View reviewed changes

rylnd approved these changes Mar 19, 2026

View reviewed changes

Merge branch 'main' into update-ml-jobs-with-new-euids

f20f51a

spong approved these changes Mar 19, 2026

View reviewed changes

peteharverson reviewed Mar 19, 2026

View reviewed changes

peteharverson approved these changes Mar 20, 2026

View reviewed changes

Merge branch 'main' into update-ml-jobs-with-new-euids

d3935c7

florent-leborgne approved these changes Mar 20, 2026

View reviewed changes

nastasha-solomon approved these changes Mar 20, 2026

View reviewed changes

maximpn requested review from maximpn and removed request for dplumlee March 23, 2026 13:43

maximpn approved these changes Mar 23, 2026

View reviewed changes

sodhikirti07 added 2 commits March 23, 2026 13:09

Merge branch 'main' into update-ml-jobs-with-new-euids

41de35f

Merge branch 'main' into update-ml-jobs-with-new-euids

3f2fbdf

sodhikirti07 merged commit e5b95bb into main Mar 24, 2026
19 checks passed

sodhikirti07 deleted the update-ml-jobs-with-new-euids branch March 24, 2026 15:01

ymao1 mentioned this pull request Mar 26, 2026

Adding V2 anomaly record data elastic/security-documents-generator#339

Merged

sodhikirti07 mentioned this pull request Mar 27, 2026

[ML] Update Security ML jobs influencers list #259833

Merged

10 tasks

	export const affectedJobIds: string[] = [
	export const outdatedJobIds: string[] = [

Conversation

sodhikirti07 commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Identify risks

Uh oh!

jmcarlock left a comment

Choose a reason for hiding this comment

Uh oh!

sodhikirti07 commented Mar 19, 2026

Uh oh!

ymao1 left a comment

Choose a reason for hiding this comment

Uh oh!

rylnd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

susan-shu-c Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sodhikirti07 Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peteharverson left a comment

Choose a reason for hiding this comment

Uh oh!

florent-leborgne left a comment

Choose a reason for hiding this comment

Uh oh!

nastasha-solomon left a comment

Choose a reason for hiding this comment

Uh oh!

maximpn left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 24, 2026

⏳ Build in-progress

Failed CI Steps

Test Failures

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

sodhikirti07 commented Feb 27, 2026 •

edited

Loading

susan-shu-c Mar 19, 2026 •

edited

Loading

sodhikirti07 Mar 19, 2026 •

edited

Loading