[Entity Store] Implement EUID

[romulets]

Introduces unique id identifier.

Decisions:

• Not create an abstraction for the evaluation logic yet. Pushing this as it's for faster development loop
• Always prefixing the ${type}: to ids
  • host.name@host.domain becomes host:host.name @ host.domain
  • user.name becomes user:user.name
  • user.name becomes user:user.name
  • service.name becomes service:service.name
  • user.entity.id becomes user:user.entity.id
  • host.entity.id becomes host:host.entity.id
  • service.entity.id becomes service:service.entity.id
  • entity.id becomes generic:entity.id

Decisions to be made

• Will generic be only on entity.id? I believe so. @opauloh do you agree?
• With always prepending things, we might temper with valuable data. Should we have another field to keep ids with the type?

Tests

1. Enable the v2 feature flag (via kibana.yaml or http call)

POST kbn:/internal/kibana/settings
{
  "changes": {
    "securitySolution:entityStoreEnableV2": true
  }
}

2. Install the entity store v2

POST kbn:/internal/security/entity-store/install?apiVersion=2
{}

3. Watch it run or force logs extraction via the internal api

POST kbn:/internal/security/entity-store/host/force-log-extraction?apiVersion=2
{
    "fromDateISO": "2026-01-20T11:00:00Z",
    "toDateISO": "2026-01-20T13:00:00Z"
}

I used this `_bulk` operation as testing data

POST /_bulk
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:00Z", "user": {"name": "john.doe"}, "host": {"entity": {"id": "host-123"}}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:01Z", "user": {"name": "jane.smith"}, "host": {"id": "host-456"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:02Z", "user": {"name": "bob.jones"}, "host": {"name": "server-01"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:03Z", "user": {"name": "alice.brown", "id": "user-789"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:04Z", "user": {"id": "user-101"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:04Z", "user": {"entity": {"id": "non-generated-user"}}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:05Z", "user": {"email": "test@example.com"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:06Z", "user": {"name": "charlie.wilson", "domain": "corp"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:07Z", "user": {"name": "david.lee"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:08Z", "user": {"name": "", "id": "user-202"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:09Z", "user": {"name": null, "id": "user-303"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:10Z", "user": {"name": "eve.martin"}, "host": {"entity": {"id": ""}, "id": "host-404"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:11Z", "user": {"name": "frank.taylor"}, "host": {"entity": {"id": null}, "id": "", "name": "workstation-05"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:12Z", "user": {"name": "grace.anderson", "email": "grace@example.com"}, "host": {"entity": {"id": null}, "id": null, "name": null}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:13Z", "user": {"name": "henry.clark"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:14Z", "user": {"name": "iris.davis", "domain": ""}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:15Z", "user": {"name": "jack.white", "domain": null}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:16Z", "user": {"name": "karen.green", "id": "user-505"}, "host": {"entity": {"id": "host-505"}}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:17Z", "user": {"name": "larry.black", "email": "larry@example.com"}, "host": {"id": "host-606"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:18Z", "user": {"name": "mary.blue", "domain": "corp"}, "host": {"name": "server-07"}}
{"create": {"_index": ".entities.v2.updates.security_user_default"}}
{"@timestamp": "2026-01-20T12:05:19Z"}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:00Z", "host": {"id": "host-123"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:01Z", "host": {"name": "server-01", "domain": "example.com"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:02Z", "host": {"hostname": "workstation-05", "domain": "corp.local"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:03Z", "host": {"domain": "test.org"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:04Z", "host": {"hostname": "laptop-01"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:05Z", "host": {"name": "desktop-02"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:06Z", "host": {"id": "host-456", "name": "server-02", "domain": "example.com"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:07Z", "host": {"id": "host-789", "hostname": "workstation-10", "domain": "corp.local"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:08Z", "host": {"name": "server-03", "hostname": "backup-server", "domain": "test.com"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:09Z", "host": {"id": "", "name": "server-04", "domain": "example.org"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:10Z", "host": {"id": null, "name": "server-05", "domain": "test.net"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:11Z", "host": {"name": "", "domain": "example.com"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:12Z", "host": {"name": null, "hostname": "workstation-20", "domain": "corp.local"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:13Z", "host": {"name": "server-06", "domain": ""}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:14Z", "host": {"name": "server-07", "domain": null}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:15Z", "host": {"hostname": "", "domain": "test.org"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:16Z", "host": {"hostname": null, "domain": "example.com"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:17Z", "host": {"hostname": "laptop-02", "name": "server-08", "domain": "corp.local"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:18Z", "host": {"hostname": "workstation-30"}}
{"create": {"_index": ".entities.v2.updates.security_host_default"}}
{"@timestamp": "2026-01-20T12:05:04Z", "host": {"entity": {"id": "non-generated-host"}}}

[uri-weisman]

🚀

[opauloh]

Will generic be only on entity.id? I believe so. @opauloh do you agree?

I agree, we don't have a clear guideline yet on establishing additional fields for all the ECS root fields (i.e cloud.entity.id, orchestrator.entity.id), so generic will remain the umbrella for storing entity types that doesn't fall into host / user / service. Integrations that publishes entity.id for Host and Identity should be updated to publish host.entity.id and user.entity.id

[opauloh]

With always prepending things, we might temper with valuable data. Should we have another field to keep ids with the type?

That's is a valid concern, and since we are moving to store a typed entity.id (e.g. user:..., host:...), we should preserve provenance so we can trace back to the original events. I initially thought about having something like an entity.raw_id, but I think we can expand it to a solution that also allow tracing back the composed IDs (e.g. root@host-123, ADMIN@DOMAIN-ABC).

I suggest we reserve an entity.id_metadata field that records what the id was composed of:

"entity.id": "user:root@host-123",
"entity.id_metadata": {
  "composed_of": [
    { "field": "user.name", "value": "root" },
    { "field": "host.id", "value": "host-123" }
  ]
}

---

"entity.id": "host:host-123",
"entity.id_metadata": {
  "composed_of": [
    { "field": "host.id", "value": "host-123" }
  ]
}

---

"entity.id": "host:arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0",
"entity.id_metadata": {
  "composed_of": [
    { "field": "host.entity.id", "value": "arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0" }
  ]
}

---

"entity.id": "user:ADMIN@DOMAIN-ABC",
"entity.id_metadata": {
  "composed_of": [
    { "field": "user.name", "value": "ADMIN" },
    { "field": "user.domain", "value": "DOMAIN-ABC" }
  ]
}

This keeps the typed ids for uniqueness, and preserving enough context to re-query source events and debug id generation. The idea of the entity.id_metadata field is that in the future we can expand to include other fields (i.e entity.id_metadata.version to track ranking/composition changes).

Let me know what are your thoughts on that, this can be totally addressed in a follow-up ticket as well

[csr]

LGTM, thanks for working through our Scout feedback. Nice tests!

[romulets]

That's is a valid concern, and since we are moving to store a typed entity.id (e.g. user:..., host:...), we should preserve provenance so we can trace back to the original events. I initially thought about having something like an entity.raw_id, but I think we can expand it to a solution that also allow tracing back the composed IDs (e.g. root@host-123, ADMIN@DOMAIN-ABC).

@opauloh I think for simplicity of the query having just the raw_id right now might be the best approach. I worry about complexity and performance impact of having a fine grained detail of what was considered in which entity. I rather explore this in another ticket

[opauloh]

User / Host / Service Ranking implementation LGTM 💯

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: fdaee24
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-250566-fdaee24c04e0
• Security Deployment

Failed CI Steps

• Scout: [ platform / streams_app-stateful ] plugin

Test Failures

• [job] [logs] Scout: [ platform / streams_app-stateful ] plugin / should allow adding a new kql data source
• [job] [logs] Scout: [ platform / streams_app-stateful ] plugin / stateful - Stream data processing - data sources management - should allow adding a new kql data source

Metrics [docs]

✅ unchanged

History

• 💔 Build #386861 failed 5c61a48
• 💔 Build #386765 failed a7079d0
• 💔 Build #386596 failed cd99dc3
• 💔 Build #386583 failed 0b12cb9

cc @romulets