Update entities schema to support integrations data by uri-weisman · Pull Request #262242 · elastic/kibana

uri-weisman · 2026-04-09T11:18:57Z

Summary

Update entities schema to align with the populated integrations data.

Summary
Refines how relationship fields from logs are materialized on entity documents in the entity store v2 index:
each relationship besides resolution (e.g. owns, supervises) is now an object of { raw_identifiers, ids }.

We introduce a data structure that allows integrations and entity maintainers to populate as much identifiers as possible, it will provide us the ability, in the future, to run a maintainer on entity store indices and find a correlation to those identifiers.

What changed:

For each supported relationship, we collect identifier fields into entity.relationships.<relationship>.raw_identifiers.*. entity.relationships.<relationship>.ids - should represent EUIDs (will be updated by entity maintainers).

Integration data (source) example:

{
  "@timestamp": "2024-09-01T12:00:00.000Z",
  "event": { "module": "endpoint", "dataset": "endpoint.events.network" },
  "user": {
    "id": "S-1-5-21-…-1001",
    "name": "jdoe",
    "entity": {
      "id": "user:jdoe@corp",
      "type": "user",
      "relationships": {
        "owns": {
          "host.name": ["prod-db-01", "prod-db-02"],
          "host.id": ["h-aaa", "h-bbb"],
        },
        "supervises": {
          "user.email": ["delegate@corp"],
        }
      }
    }
  }
}

Stored shape in .entities.v2. without post processing:

"entity": {
  "namespace": "workday"
  "relationships": {
    "owns": {
      "raw_identifiers": {
        "host.name": ["prod-db-01", "prod-db-02"],
        "host.id": ["h-aaa", "h-bbb"]
      },
      "ids": []
    },
    "supervises": {
      "raw_identifiers": { "user.email": ["john123@corp"] },
      "ids": []
    }
  }
}

**Another entity in the store that the relationship's attributes refers to:**

"entity": {
  "id": "user:john123@corp@workday"
  "namespace": "workday"
  "user": {
    "email": "john123@corp"
  }
}

Stored shape in .entities.v2 after post-processing:

"entity": {
  "relationships": {
    "owns": {
      "raw_identifiers": {
        "host.name": ["prod-db-01", "prod-db-02"],
        "host.id": ["h-aaa", "h-bbb"]
      },
      "ids": ["host:prod-db-01@corp"]
    },
    "supervises": {
      "raw_identifiers": { "user.email": ["john123@corp"] },
      "ids": ["user:john123@corp@workday"]
    }
  }
}

Values in raw_identifiers and ids are aggregated over time per entity-store rules (collect / dedupe), so the index may show more than a single event contributed.

…hub.com/uri-weisman/kibana into feat/entity-store-ecs-relationship-bags

niros1 · 2026-04-12T14:09:39Z

Thanks Uri
Just want to pin point that ids can store "resolved" ids from the raw data OR from any other maintainer or source.

uri-weisman · 2026-04-12T15:58:34Z

@coderabbitai full review

coderabbitai · 2026-04-12T15:58:41Z

✅ Actions performed

Full review triggered.

uri-weisman · 2026-04-12T16:26:21Z

Just want to pin point that ids can store "resolved" ids from the raw data OR from any other maintainer or source.

@niros1
Entity maintainer will emit API request as follows:

{
  "user": {
    "id": "uri@elastic",
    "relationships": {
      "owns": {
        "ids": ["host:prod-db-01@corp"]
      },
      "supervises": {
        "ids": ["user:john123@corp@workday"]
      }
    }
  }
}

The Integration log will look as follows (integrations are less likely to populate entity.id):
(I don't think we should expose implementation details on ECS based schema, all identifiers including entity.id should be published under the same structure.)

{
  "@timestamp": "2024-09-01T12:00:00.000Z",
  "event": { "module": "endpoint", "dataset": "endpoint.events.network" },
  "user": {
    "id": "uri@elastic",
    "name": "jdoe",
    "entity": {
      "id": "user:jdoe@corp",
      "type": "user",
      "relationships": {
        "owns": {
          "host.name": "blabla"
          "entity.id" ["host:prod-db-01@corp"],
        },
        "supervises": {
          "entity.id": ["user:john123@corp@workday"],
        }
      }
    }
  }
}

coderabbitai · 2026-04-12T16:28:41Z

✅ Actions performed

Full review triggered.

…hub.com/uri-weisman/kibana into feat/entity-store-ecs-relationship-bags

romulets · 2026-04-13T08:20:06Z

+    return `${col} = COALESCE(\`entity.relationships.resolution.resolved_to\`, [""])`;
+  }
+
+  return `${col} = COALESCE(\`entity.relationships.${field}.ids\`, [""])`;


Is graph ok with having correlations for id only?

for my understanding that was the format before the change but it was directly under the {field}.
@kfirpeled ?

The graph works with ids only
In order to correlate between entities and events we pull relevant data per node. And based on the enginemetadata we build the filters using the DSL functions

it is now in review: #261420

uri-weisman · 2026-04-13T10:16:26Z

/ci

kfirpeled

Checked on existing environment
The graph fails to load due to this change

Tried to "clear all data" from entity store. But now entity store cannot recover due to another issue.

Providing here the error message from the API

Error log 📁

“message”: “verification_exception: Found 7 problems\nline 3:49: Unknown column [entity.relationships.accesses_frequently.ids], did you mean any of [entity.relationships.accesses_frequently, entity.relationships.accesses_infrequently, entity.relationships.owns_inferred, entity.relationships.supervises, entity.relationships.depends_on, entity.relationships.communicates_with, entity.relationships.resolution.resolved_to, entity.relationships.owns]?\nline 4:51: Unknown column [entity.relationships.accesses_infrequently.ids], did you mean any of [entity.relationships.accesses_infrequently, entity.relationships.accesses_frequently, entity.relationships.owns_inferred, entity.relationships.resolution.resolved_to, entity.relationships.communicates_with, entity.relationships.supervises, entity.relationships.depends_on, entity.relationships.resolution.risk.calculated_score, entity.relationships.owns]?\nline 5:47: Unknown column [entity.relationships.communicates_with.ids], did you mean any of [entity.relationships.communicates_with, entity.relationships.owns_inferred, entity.relationships.supervises, entity.relationships.owns, entity.relationships.depends_on, entity.relationships.resolution.resolved_to, entity.relationships.accesses_frequently, entity.relationships.accesses_infrequently, entity.relationships.resolution.risk.calculated_score_norm]?\nline 6:40: Unknown column [entity.relationships.depends_on.ids], did you mean any of [entity.relationships.depends_on, entity.relationships.supervises, entity.relationships.owns, entity.relationships.owns_inferred, entity.relationships.communicates_with, entity.relationships.resolution.resolved_to, entity.relationships.accesses_frequently, entity.relationships.accesses_infrequently, entity.relationships.resolution.risk.calculated_score, entity.relationships.resolution.risk.calculated_level]?\nline 7:34: Unknown column [entity.relationships.owns.ids], did you mean any of [entity.relationships.owns, entity.relationships.owns_inferred, entity.relationships.supervises, entity.relationships.depends_on, entity.relationships.communicates_with, entity.relationships.resolution.resolved_to, entity.relationships.accesses_frequently, entity.relationships.accesses_infrequently, entity.relationships.resolution.risk.calculated_score]?\nline 8:43: Unknown column [entity.relationships.owns_inferred.ids], did you mean any of [entity.relationships.owns_inferred, entity.relationships.owns, entity.relationships.supervises, entity.relationships.accesses_infrequently, entity.relationships.communicates_with, entity.relationships.depends_on, entity.relationships.accesses_frequently, entity.relationships.resolution.resolved_to]?\nline 10:40: Unknown column [entity.relationships.supervises.ids], did you mean any of [entity.relationships.supervises, entity.relationships.depends_on, entity.relationships.communicates_with, entity.relationships.owns_inferred, entity.relationships.owns, entity.relationships.resolution.resolved_to, entity.relationships.accesses_frequently, entity.relationships.accesses_infrequently, entity.relationships.resolution.risk.calculated_score]?”

uri-weisman · 2026-04-13T11:34:31Z

/ci

kfirpeled

Tested on existing serverless
Relationships Owns and resolution, it covers all edge cases

Both worked

uri-weisman · 2026-04-13T13:39:50Z

/ci

elasticmachine · 2026-04-13T16:19:00Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 99af2ad
Storybooks Preview
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-262242-99af2adbe18d

Failed CI Steps

FTR Configs #88

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`entityStore`	128.0KB	128.1KB	+103.0B

History

kibanamachine · 2026-04-13T16:33:55Z

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/24354816960

(cherry picked from commit 1270467)

kibanamachine · 2026-04-13T16:42:00Z

💚 All backports created successfully

Status	Branch	Result
✅	9.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…262856) # Backport This will backport the following commits from `main` to `9.4`: - [Update entities schema to support integrations data (#262242)](#262242)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Uri Weisman <68195305+uri-weisman@users.noreply.github.com>

Tracks the changes needed in communicates_with and accesses maintainers to align with the EntityRelationship schema introduced in elastic#262242. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

uri-weisman added 2 commits April 9, 2026 14:11

Update entity schema to support integrations data

19cb101

beautiy pr

0a0c384

macroscopeapp Bot reviewed Apr 9, 2026

View reviewed changes

Comment thread x-pack/solutions/security/plugins/entity_store/common/domain/definitions/common_fields.ts Outdated

uri-weisman added 2 commits April 9, 2026 14:37

add non relationship fields

0bad6b7

some fixes

64f6ff0

macroscopeapp Bot reviewed Apr 9, 2026

View reviewed changes

Comment thread x-pack/solutions/security/plugins/entity_store/common/domain/definitions/common_fields.ts Outdated

uri-weisman and others added 3 commits April 9, 2026 15:16

Merge branch 'main' into feat/entity-store-ecs-relationship-bags

3d3e7bf

update to ecs convention

21e30a4

Merge branch 'feat/entity-store-ecs-relationship-bags' of https://git…

3da4135

…hub.com/uri-weisman/kibana into feat/entity-store-ecs-relationship-bags

uri-weisman marked this pull request as ready for review April 9, 2026 18:12

uri-weisman requested a review from a team as a code owner April 9, 2026 18:12

uri-weisman and others added 3 commits April 9, 2026 21:12

Merge branch 'main' into feat/entity-store-ecs-relationship-bags

e2d21d3

fix graph impl

0d3c5d5

Merge branch 'feat/entity-store-ecs-relationship-bags' of https://git…

92170ea

…hub.com/uri-weisman/kibana into feat/entity-store-ecs-relationship-bags

uri-weisman requested a review from a team as a code owner April 12, 2026 08:45

revert install assets

95e208f

uri-weisman added release_note:skip Skip the PR/issue when compiling release notes backport:version Backport to applied version labels v9.4.0 labels Apr 12, 2026

uri-weisman and others added 4 commits April 12, 2026 11:59

Merge branch 'main' into feat/entity-store-ecs-relationship-bags

331693f

update to raw dst

a0f2ceb

Merge branch 'feat/entity-store-ecs-relationship-bags' of https://git…

6f4c7ad

…hub.com/uri-weisman/kibana into feat/entity-store-ecs-relationship-bags

revert install assets

178d6e4

uri-weisman requested review from chemamartinez and kfirpeled April 12, 2026 16:40

Merge branch 'feat/entity-store-ecs-relationship-bags' of https://git…

e5fa730

…hub.com/uri-weisman/kibana into feat/entity-store-ecs-relationship-bags

romulets reviewed Apr 13, 2026

View reviewed changes

cr fixes

caf14a2

romulets approved these changes Apr 13, 2026

View reviewed changes

kfirpeled requested changes Apr 13, 2026

View reviewed changes

kfirpeled approved these changes Apr 13, 2026

View reviewed changes

Merge branch 'main' into feat/entity-store-ecs-relationship-bags

99af2ad

uri-weisman enabled auto-merge (squash) April 13, 2026 16:08

uri-weisman merged commit 1270467 into elastic:main Apr 13, 2026
18 checks passed

uri-weisman deleted the feat/entity-store-ecs-relationship-bags branch April 13, 2026 16:33

kibanamachine added the v9.5.0 label Apr 13, 2026

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 13, 2026

Update entities schema to support integrations data (elastic#262242)

419b28f

(cherry picked from commit 1270467)

kibanamachine mentioned this pull request Apr 13, 2026

[9.4] Update entities schema to support integrations data (#262242) #262856

Merged

seanrathier mentioned this pull request Apr 13, 2026

[Entity Analytics] Update entity maintainers to use new relationship schema (ids object) #262893

Closed

4 tasks

seanrathier mentioned this pull request Apr 13, 2026

[Entity Analytics] Update entity maintainers to use new relationship schema (ids object) #262911

Merged

3 tasks

uri-weisman mentioned this pull request Apr 14, 2026

[Entity Analytics] Find matching entities from relationship raw_identifiers #262972

Open

This was referenced Apr 29, 2026

Supervises Relationship maintainer #266369

Open

Owns Relationship maintainer #266373

Open

Administers Relationship maintainer #266374

Open

Depends On Relationship maintainer #266375

Open

Conversation

uri-weisman commented Apr 9, 2026 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed:

Uh oh!

Uh oh!

Uh oh!

niros1 commented Apr 12, 2026

Uh oh!

uri-weisman commented Apr 12, 2026

Uh oh!

coderabbitai Bot commented Apr 12, 2026

Uh oh!

uri-weisman commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Apr 12, 2026

Uh oh!

romulets Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

niros1 Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

kfirpeled Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

uri-weisman commented Apr 13, 2026

Uh oh!

kfirpeled left a comment

Choose a reason for hiding this comment

Uh oh!

uri-weisman commented Apr 13, 2026

Uh oh!

kfirpeled left a comment

Choose a reason for hiding this comment

Uh oh!

uri-weisman commented Apr 13, 2026

Uh oh!

elasticmachine commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

History

Uh oh!

Uh oh!

kibanamachine commented Apr 13, 2026

Uh oh!

kibanamachine commented Apr 13, 2026

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

uri-weisman commented Apr 9, 2026 •

edited by kibanamachine

Loading

uri-weisman commented Apr 12, 2026 •

edited

Loading

elasticmachine commented Apr 13, 2026 •

edited

Loading