[Contextual Security] Replace ENRICH with LOOKUP JOIN queries when fetching entities#247815
Conversation
…RICH policy refactor graph api entity enrichment section to test both LOOKUP and ENRICH query flows update alerts and events graph FTR tests to test both LOOKUP and ENRICH flows
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10296[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10298[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#10299[✅] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 25/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10300[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10301[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#10302[✅] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 25/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10305[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10306[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10309[❌] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 24/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#10312[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#10433[✅] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 25/25 tests passed. |
| * Recursively removes properties with string value "undefined" from an object. | ||
| * This handles cases where ESQL COALESCE returns "undefined" as a fallback string. | ||
| */ | ||
| const filterUndefinedStringValues = <T extends Record<string, unknown>>(obj: T): T => { |
There was a problem hiding this comment.
Do we want to filter undefined from the response?
There was a problem hiding this comment.
i would like to keep the undefined properties to not be returned as we had before - there is no benefit IMO returning such values.
| | DROP entity.id | ||
| | DROP entity.target.id | ||
| // rename entity.*fields before next pipeline to avoid name collisions | ||
| | EVAL entity.id = actorEntityId | ||
| | LOOKUP JOIN ${getEntitiesLatestIndexName(spaceId)} ON entity.id | ||
| | RENAME actorEntityName = entity.name | ||
| | RENAME actorEntityType = entity.type | ||
| | RENAME actorEntitySubType = entity.sub_type | ||
| | RENAME actorHostIp = host.ip | ||
| | RENAME actorLookupEntityId = entity.id | ||
|
|
||
| | EVAL entity.id = targetEntityId | ||
| | LOOKUP JOIN ${getEntitiesLatestIndexName(spaceId)} ON entity.id | ||
| | RENAME targetEntityName = entity.name | ||
| | RENAME targetEntityType = entity.type | ||
| | RENAME targetEntitySubType = entity.sub_type | ||
| | RENAME targetHostIp = host.ip | ||
| | RENAME targetLookupEntityId = entity.id | ||
|
|
||
| ${buildEnrichedEntityFieldsEsql()} | ||
| ` | ||
| : isEnrichPolicyExists | ||
| ? ` | ||
| // Use ENRICH policy for entity enrichment (deprecated fallback) |
There was a problem hiding this comment.
nit: In a follow-up PR, I would split this whole conditional chunk into small functions, each of them returning the clauses for lookup, enrich & fallback. As of now, it's becoming a challenge to understand the query.
But for this PR, feel free to merge without refactoring
There was a problem hiding this comment.
Great point, extracted lookup and enrich query builders to utility functions.
There was a problem hiding this comment.
There's a miss understanding regarding how we should populate undefined. this is unnecessary. Lets fix how we return undefined first. And then you'll not need utility such as filterUndefinedStringValues
There was a problem hiding this comment.
Update: sorry for the confusion, replace undefined with null.
There was a problem hiding this comment.
It seems to me, when actorEntityType is null
the generated code is
,"type":"undefined"
while it is expected to be
,"type": undefined
There was a problem hiding this comment.
Update: sorry for the confusion, replace undefined with null.
| * Generates ESQL statements for building entity fields with enrichment data. | ||
| * This is used when entity store enrichment is available (via LOOKUP JOIN or ENRICH). | ||
| */ | ||
| const buildEnrichedEntityFieldsEsql = (): string => { |
There was a problem hiding this comment.
I'd move this function to esql.utils.ts too since buildEnrichPolicyEsql and buildLookupJoinEsql live there already
add integration test for entities with partial data (name-only or type/sub_type-only)
kfirpeled
left a comment
There was a problem hiding this comment.
LGTM
Both ENRICH and LOOKUP JOIN worked locally
| | EVAL actorEntityField = CASE( | ||
| actorEntityName IS NOT NULL OR actorEntityType IS NOT NULL OR actorEntitySubType IS NOT NULL, | ||
| CONCAT(",\\"entity\\":", "{", | ||
| ${formatJsonProperty('name', 'actorEntityName', false)}, | ||
| REPLACE(CONCAT(",\\"entity\\":", "{", |
There was a problem hiding this comment.
instead of REPLACE
you can switch the order of the fields
so that availableInEntityStore and ecsParentField are first
Reduces the complexity
| CONCAT(",\\"entity\\":", "{", | ||
| "\\"availableInEntityStore\\":false", | ||
| ",\\"ecsParentField\\":\\"", actorEntityFieldHint, "\\"", | ||
| "}") | ||
| ) | ||
| | EVAL targetEntityField = CASE( | ||
| targetEntityName IS NOT NULL OR targetEntityType IS NOT NULL OR targetEntitySubType IS NOT NULL, | ||
| CONCAT(",\\"entity\\":", "{", | ||
| ${formatJsonProperty('name', 'targetEntityName', false)}, | ||
| REPLACE(CONCAT(",\\"entity\\":", "{", |
💚 Build Succeeded
Metrics [docs]History
|
…tching entities (elastic#247815) ## Summary This PR Introduces LOOKUP JOIN as the primary entity enrichment mechanism while maintaining backward compatibility with the deprecated ENRICH policy during the transition period. Closes [issue](elastic#232226) and multiple flaky tests due to entity store infra initialization instability. **Server-side changes (fetch_graph.ts)** - Implement LOOKUP JOIN query generation for entity enrichment - Add fallback logic: LOOKUP JOIN → ENRICH policy → no enrichment - Add `getEntitiesLatestIndexName` helper for v2 index names **Test infrastructure** - Add `executeEnrichPolicy` helper to entity_store.ts utils - Create entity_store_v2 test archives with lookup mode mappings - Create entity_store_v2_standard_mode for fallback scenario testing **API integration tests (graph.ts)** - Refactor 'Enrich graph with entity metadata' to test both flows - Add enrichmentConfigs array for ENRICH (v1) and LOOKUP JOIN (v2) - Add fallback test: v2 index exists but not in lookup mode **FTR functional tests** - Update alerts_flyout.ts with dual enrichment config support - Update events_flyout.ts with dual enrichment config support - Reuse entity_store_v2 archives across functional tests **Api/FTR tests coverage** Scenario | v2 Lookup Index | ENRICH Policy | Expected Path | Currently Tested? -- | -- | -- | -- | -- 1 | ✅ Exists in lookup mode | N/A | LOOKUP JOIN | ✅ v2 tests 2 | ❌ Doesn't exist | ✅ Exists | ENRICH | ✅ v1 tests 3 | ❌ Doesn't exist | ❌ Doesn't exist | No enrichment | ✅ All other tests (Happy flows, Validation, etc.) v2 - refers to the new mappings and data mocks we load to test the LOOKUP JOIN functionality - each test could be added just once and it will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we stop supporting querying enrich policies. ## How to test 1. Deploy a local env using the following command: `node scripts/es snapshot --license trial -E path.data=../default -E reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443 -E xpack.security.authc.api_key.enabled=true` 2. run kibana using `yarn start` 3. Go to `Advanced settings` and make sure`securitySolution:enableGraphVisualization` and `securitySolution:enableAssetInventory` features are toggled on. 4. Got to Security -> inventory -> click on 'Enable Asset Inventory'. 5. Install latest gcp-auditlogs integration (skip agent installation) v2.46.0 and above. 6. Install aws-cloudtrail integration (skip agent installation) v4.7.0 and above. 7. Install cloud asset discovery integration (skip agent installation). 8. reindex gcp-auditlogs data from long-live env: ``` POST _reindex { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "<api key>" } }, "index": "logs-*", "query": { "bool": { "must": [ { "term": { "data_stream.dataset": "gcp.audit" } }, { "bool": { "should": [ { "exists": { "field": "user.entity.id" } }, { "exists": { "field": "host.entity.id" } }, { "exists": { "field": "service.entity.id" } }, { "exists": { "field": "entity.id" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "exists": { "field": "user.target.entity.id" } }, { "exists": { "field": "host.target.entity.id" } }, { "exists": { "field": "service.target.entity.id" } }, { "exists": { "field": "entity.target.id" } } ], "minimum_should_match": 1 } } ] } } }, "dest": { "op_type": "create", "index": "logs-gcp.audit-default" } } ``` 9. reindex aws-cloudtrail data from long-live env: ``` POST _reindex { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "ApiKey YmNXcUNaZ0JYd1lMQmZkOEZ1bFc6TDZ3RFNVOXh2R2NEWV9Nb2YyTWxtQQ==" } }, "index": "logs-aws.cloudtrail-default", "query": { "bool": { "must": [ { "bool": { "should": [ { "exists": { "field": "user.entity.id" } }, { "exists": { "field": "host.entity.id" } }, { "exists": { "field": "service.entity.id" } }, { "exists": { "field": "entity.id" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "exists": { "field": "user.target.entity.id" } }, { "exists": { "field": "host.target.entity.id" } }, { "exists": { "field": "service.target.entity.id" } }, { "exists": { "field": "entity.target.id" } } ], "minimum_should_match": 1 } } ] } } }, "dest": { "op_type": "create", "index": "logs-aws.cloudtrail-default" } } ``` 10. reindex entities data from long-live env: ``` POST _reindex?wait_for_completion=true { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "message for api key" } }, "index": ".entities.v1.latest.security_generic_default", "query": { "bool": { "must": [], "filter": [ { "range": { "@timestamp": { "gte": "now-2y", "lte": "now" } } } ] } } }, "dest": { "op_type": "create", "index": ".entities.v1.latest.security_generic_default" }, "script": { "source": """ ctx._source.doc_id = ctx._id; ctx._source.doc_index = ctx._index; if (ctx._source.asset != null) { if (ctx._source.asset.containsKey('category')) { ctx._source['entity.category'] = ctx._source.asset.category; } if (ctx._source.asset.containsKey('name')) { ctx._source['entity.name'] = ctx._source.asset.name; } if (ctx._source.asset.containsKey('type')) { ctx._source['entity.type'] = ctx._source.asset.type; } if (ctx._source.asset.containsKey('sub_type')) { ctx._source['entity.sub_type'] = ctx._source.asset.sub_type; } if (ctx._source.asset.containsKey('sub_category')) { ctx._source['entity.sub_category'] = ctx._source.asset.sub_category; } } """ } } ``` 11. Create an entities v2 index with lookup mode: ``` PUT .entities.v2.latest.security_generic_default { "settings": { "index": { "mode": "lookup", "number_of_shards": 1, "number_of_replicas": 1 } }, "mappings": { "_meta": { "version": "1.6.0" }, "dynamic_templates": [ { "ecs_timestamp": { "match": "@timestamp", "mapping": { "ignore_malformed": false, "type": "date" } } }, { "ecs_message_match_only_text": { "path_match": [ "message", "*.message" ], "unmatch_mapping_type": "object", "mapping": { "type": "match_only_text" } } }, { "ecs_non_indexed_keyword": { "path_match": [ "*event.original", "*gen_ai.agent.description" ], "mapping": { "doc_values": false, "index": false, "type": "keyword" } } }, { "ecs_non_indexed_long": { "path_match": "*.x509.public_key_exponent", "mapping": { "doc_values": false, "index": false, "type": "long" } } }, { "ecs_ip": { "path_match": [ "ip", "*.ip", "*_ip" ], "match_mapping_type": "string", "mapping": { "type": "ip" } } }, { "ecs_wildcard": { "path_match": [ "*.io.text", "*.message_id", "*registry.data.strings", "*url.path" ], "unmatch_mapping_type": "object", "mapping": { "type": "wildcard" } } }, { "ecs_path_match_wildcard_and_match_only_text": { "path_match": [ "*.body.content", "*url.full", "*url.original" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "wildcard" } } }, { "ecs_match_wildcard_and_match_only_text": { "match": [ "*command_line", "*stack_trace" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "wildcard" } } }, { "ecs_path_match_keyword_and_match_only_text": { "path_match": [ "*.title", "*.executable", "*.name", "*.working_directory", "*.full_name", "*.display_name", "*file.path", "*file.target_path", "*os.full", "*email.subject", "*vulnerability.description", "*user_agent.original" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "keyword" } } }, { "ecs_date": { "path_match": [ "*.timestamp", "*_timestamp", "*.not_after", "*.not_before", "*.accessed", "created", "*.created", "*.installed", "*.creation_date", "*.ctime", "*.mtime", "ingested", "*.ingested", "*.start", "*.end", "*.indicator.first_seen", "*.indicator.last_seen", "*.indicator.modified_at", "*threat.enrichments.matched.occurred" ], "unmatch_mapping_type": "object", "mapping": { "type": "date" } } }, { "ecs_path_match_float": { "path_match": [ "*.score.*", "*_score*" ], "path_unmatch": "*.version", "unmatch_mapping_type": "object", "mapping": { "type": "float" } } }, { "ecs_usage_double_scaled_float": { "path_match": "*.usage", "match_mapping_type": [ "double", "long", "string" ], "mapping": { "scaling_factor": 1000, "type": "scaled_float" } } }, { "ecs_geo_point": { "path_match": "*.geo.location", "mapping": { "type": "geo_point" } } }, { "ecs_flattened": { "path_match": [ "*structured_data", "*exports", "*imports" ], "match_mapping_type": "object", "mapping": { "type": "flattened" } } }, { "ecs_gen_ai_integers": { "path_match": [ "*gen_ai.request.max_tokens", "*gen_ai.usage.input_tokens", "*gen_ai.usage.output_tokens", "*gen_ai.request.choice.count", "*gen_ai.request.seed" ], "mapping": { "type": "integer" } } }, { "ecs_gen_ai_doubles": { "path_match": [ "*gen_ai.request.temperature", "*gen_ai.request.top_k", "*gen_ai.request.frequency_penalty", "*gen_ai.request.presence_penalty", "*gen_ai.request.top_p" ], "mapping": { "type": "double" } } }, { "all_strings_to_keywords": { "match_mapping_type": "string", "mapping": { "ignore_above": 1024, "type": "keyword" } } }, { "strings_as_keyword": { "match_mapping_type": "string", "mapping": { "fields": { "text": { "type": "text" } }, "ignore_above": 1024, "type": "keyword" } } }, { "entity_metrics": { "path_match": "entity.metrics.*", "match_mapping_type": [ "long", "double" ], "mapping": { "type": "{dynamic_type}" } } } ], "date_detection": false, "properties": { "@timestamp": { "type": "date" }, "asset": { "properties": { "business_unit": { "type": "keyword" }, "criticality": { "type": "keyword" }, "environment": { "type": "keyword" }, "id": { "type": "keyword" }, "model": { "type": "keyword" }, "name": { "type": "keyword" }, "owner": { "type": "keyword" }, "serial_number": { "type": "keyword" }, "vendor": { "type": "keyword" } } }, "cloud": { "properties": { "account": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "availability_zone": { "type": "keyword" }, "instance": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "machine": { "properties": { "type": { "type": "keyword" } } }, "project": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "provider": { "type": "keyword" }, "region": { "type": "keyword" }, "service": { "properties": { "name": { "type": "keyword" } } } } }, "doc_id": { "type": "keyword", "ignore_above": 1024 }, "doc_index": { "type": "keyword", "ignore_above": 1024 }, "entity": { "properties": { "EngineMetadata": { "properties": { "Type": { "type": "keyword", "ignore_above": 1024 } } }, "attributes": { "properties": { "Asset": { "type": "boolean" }, "Managed": { "type": "boolean" }, "Mfa_enabled": { "type": "boolean" }, "Privileged": { "type": "boolean" } } }, "behaviors": { "properties": { "Brute_force_victim": { "type": "boolean" }, "New_country_login": { "type": "boolean" }, "Used_usb_device": { "type": "boolean" } } }, "definition_id": { "type": "keyword", "ignore_above": 1024 }, "definition_version": { "type": "keyword", "ignore_above": 1024 }, "display_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 1024 } } }, "id": { "type": "keyword" }, "identity_fields": { "type": "keyword" }, "last_seen_timestamp": { "type": "date" }, "lifecycle": { "properties": { "First_seen": { "type": "date" }, "Last_activity": { "type": "date" } } }, "name": { "type": "keyword" }, "risk": { "properties": { "calculated_level": { "type": "keyword" }, "calculated_score": { "type": "float" }, "calculated_score_norm": { "type": "float" } } }, "schema_version": { "type": "keyword", "ignore_above": 1024 }, "source": { "type": "keyword" }, "sub_type": { "type": "keyword" }, "type": { "type": "keyword" }, "url": { "type": "keyword" } } }, "event": { "properties": { "ingested": { "type": "date" } } }, "host": { "properties": { "architecture": { "type": "keyword" }, "boot": { "properties": { "id": { "type": "keyword" } } }, "cpu": { "properties": { "usage": { "type": "keyword" } } }, "disk": { "properties": { "read": { "properties": { "bytes": { "type": "keyword" } } }, "write": { "properties": { "bytes": { "type": "keyword" } } } } }, "domain": { "type": "keyword" }, "hostname": { "type": "keyword" }, "id": { "type": "keyword" }, "ip": { "type": "ip" }, "mac": { "type": "keyword" }, "name": { "type": "keyword" }, "network": { "properties": { "egress": { "properties": { "bytes": { "type": "keyword" }, "packets": { "type": "keyword" } } }, "ingress": { "properties": { "bytes": { "type": "keyword" }, "packets": { "type": "keyword" } } } } }, "pid_ns_ino": { "type": "keyword" }, "type": { "type": "keyword" }, "uptime": { "type": "keyword" } } }, "labels": { "type": "object" }, "orchestrator": { "properties": { "api_version": { "type": "keyword" }, "cluster": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "url": { "type": "keyword" }, "version": { "type": "keyword" } } }, "namespace": { "type": "keyword" }, "organization": { "type": "keyword" }, "resource": { "properties": { "annotation": { "type": "keyword" }, "id": { "type": "keyword" }, "ip": { "type": "keyword" }, "label": { "type": "keyword" }, "name": { "type": "keyword" }, "parent": { "properties": { "type": { "type": "keyword" } } }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" } } }, "tags": { "type": "keyword", "ignore_above": 1024 }, "user": { "properties": { "domain": { "type": "keyword" }, "email": { "type": "keyword" }, "full_name": { "type": "keyword", "fields": { "text": { "type": "match_only_text" } } }, "hash": { "type": "keyword" }, "id": { "type": "keyword" }, "name": { "type": "keyword", "fields": { "text": { "type": "match_only_text" } } }, "roles": { "type": "keyword" } } } } } } ``` 12. reindex data from v1 to v2 index: ``` POST _reindex { "source": { "index": ".entities.v1.latest.security_generic_default" }, "dest": { "index": ".entities.v2.latest.security_generic_default", "op_type": "create" } } ``` 13. go to security -> explore -> network/users/hosts. 14. apply filters to see only events containing graph representation. <img width="4074" height="818" alt="image" src="https://github.com/user-attachments/assets/46605770-73f3-41af-9241-f3013ccc5038" /> 15. open the graph and play with different filters and combinations to get nodes with entity data. 16. graph should work as expected. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…tching entities (elastic#247815) ## Summary This PR Introduces LOOKUP JOIN as the primary entity enrichment mechanism while maintaining backward compatibility with the deprecated ENRICH policy during the transition period. Closes [issue](elastic#232226) and multiple flaky tests due to entity store infra initialization instability. **Server-side changes (fetch_graph.ts)** - Implement LOOKUP JOIN query generation for entity enrichment - Add fallback logic: LOOKUP JOIN → ENRICH policy → no enrichment - Add `getEntitiesLatestIndexName` helper for v2 index names **Test infrastructure** - Add `executeEnrichPolicy` helper to entity_store.ts utils - Create entity_store_v2 test archives with lookup mode mappings - Create entity_store_v2_standard_mode for fallback scenario testing **API integration tests (graph.ts)** - Refactor 'Enrich graph with entity metadata' to test both flows - Add enrichmentConfigs array for ENRICH (v1) and LOOKUP JOIN (v2) - Add fallback test: v2 index exists but not in lookup mode **FTR functional tests** - Update alerts_flyout.ts with dual enrichment config support - Update events_flyout.ts with dual enrichment config support - Reuse entity_store_v2 archives across functional tests **Api/FTR tests coverage** Scenario | v2 Lookup Index | ENRICH Policy | Expected Path | Currently Tested? -- | -- | -- | -- | -- 1 | ✅ Exists in lookup mode | N/A | LOOKUP JOIN | ✅ v2 tests 2 | ❌ Doesn't exist | ✅ Exists | ENRICH | ✅ v1 tests 3 | ❌ Doesn't exist | ❌ Doesn't exist | No enrichment | ✅ All other tests (Happy flows, Validation, etc.) v2 - refers to the new mappings and data mocks we load to test the LOOKUP JOIN functionality - each test could be added just once and it will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we stop supporting querying enrich policies. ## How to test 1. Deploy a local env using the following command: `node scripts/es snapshot --license trial -E path.data=../default -E reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443 -E xpack.security.authc.api_key.enabled=true` 2. run kibana using `yarn start` 3. Go to `Advanced settings` and make sure`securitySolution:enableGraphVisualization` and `securitySolution:enableAssetInventory` features are toggled on. 4. Got to Security -> inventory -> click on 'Enable Asset Inventory'. 5. Install latest gcp-auditlogs integration (skip agent installation) v2.46.0 and above. 6. Install aws-cloudtrail integration (skip agent installation) v4.7.0 and above. 7. Install cloud asset discovery integration (skip agent installation). 8. reindex gcp-auditlogs data from long-live env: ``` POST _reindex { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "<api key>" } }, "index": "logs-*", "query": { "bool": { "must": [ { "term": { "data_stream.dataset": "gcp.audit" } }, { "bool": { "should": [ { "exists": { "field": "user.entity.id" } }, { "exists": { "field": "host.entity.id" } }, { "exists": { "field": "service.entity.id" } }, { "exists": { "field": "entity.id" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "exists": { "field": "user.target.entity.id" } }, { "exists": { "field": "host.target.entity.id" } }, { "exists": { "field": "service.target.entity.id" } }, { "exists": { "field": "entity.target.id" } } ], "minimum_should_match": 1 } } ] } } }, "dest": { "op_type": "create", "index": "logs-gcp.audit-default" } } ``` 9. reindex aws-cloudtrail data from long-live env: ``` POST _reindex { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "ApiKey YmNXcUNaZ0JYd1lMQmZkOEZ1bFc6TDZ3RFNVOXh2R2NEWV9Nb2YyTWxtQQ==" } }, "index": "logs-aws.cloudtrail-default", "query": { "bool": { "must": [ { "bool": { "should": [ { "exists": { "field": "user.entity.id" } }, { "exists": { "field": "host.entity.id" } }, { "exists": { "field": "service.entity.id" } }, { "exists": { "field": "entity.id" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "exists": { "field": "user.target.entity.id" } }, { "exists": { "field": "host.target.entity.id" } }, { "exists": { "field": "service.target.entity.id" } }, { "exists": { "field": "entity.target.id" } } ], "minimum_should_match": 1 } } ] } } }, "dest": { "op_type": "create", "index": "logs-aws.cloudtrail-default" } } ``` 10. reindex entities data from long-live env: ``` POST _reindex?wait_for_completion=true { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "message for api key" } }, "index": ".entities.v1.latest.security_generic_default", "query": { "bool": { "must": [], "filter": [ { "range": { "@timestamp": { "gte": "now-2y", "lte": "now" } } } ] } } }, "dest": { "op_type": "create", "index": ".entities.v1.latest.security_generic_default" }, "script": { "source": """ ctx._source.doc_id = ctx._id; ctx._source.doc_index = ctx._index; if (ctx._source.asset != null) { if (ctx._source.asset.containsKey('category')) { ctx._source['entity.category'] = ctx._source.asset.category; } if (ctx._source.asset.containsKey('name')) { ctx._source['entity.name'] = ctx._source.asset.name; } if (ctx._source.asset.containsKey('type')) { ctx._source['entity.type'] = ctx._source.asset.type; } if (ctx._source.asset.containsKey('sub_type')) { ctx._source['entity.sub_type'] = ctx._source.asset.sub_type; } if (ctx._source.asset.containsKey('sub_category')) { ctx._source['entity.sub_category'] = ctx._source.asset.sub_category; } } """ } } ``` 11. Create an entities v2 index with lookup mode: ``` PUT .entities.v2.latest.security_generic_default { "settings": { "index": { "mode": "lookup", "number_of_shards": 1, "number_of_replicas": 1 } }, "mappings": { "_meta": { "version": "1.6.0" }, "dynamic_templates": [ { "ecs_timestamp": { "match": "@timestamp", "mapping": { "ignore_malformed": false, "type": "date" } } }, { "ecs_message_match_only_text": { "path_match": [ "message", "*.message" ], "unmatch_mapping_type": "object", "mapping": { "type": "match_only_text" } } }, { "ecs_non_indexed_keyword": { "path_match": [ "*event.original", "*gen_ai.agent.description" ], "mapping": { "doc_values": false, "index": false, "type": "keyword" } } }, { "ecs_non_indexed_long": { "path_match": "*.x509.public_key_exponent", "mapping": { "doc_values": false, "index": false, "type": "long" } } }, { "ecs_ip": { "path_match": [ "ip", "*.ip", "*_ip" ], "match_mapping_type": "string", "mapping": { "type": "ip" } } }, { "ecs_wildcard": { "path_match": [ "*.io.text", "*.message_id", "*registry.data.strings", "*url.path" ], "unmatch_mapping_type": "object", "mapping": { "type": "wildcard" } } }, { "ecs_path_match_wildcard_and_match_only_text": { "path_match": [ "*.body.content", "*url.full", "*url.original" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "wildcard" } } }, { "ecs_match_wildcard_and_match_only_text": { "match": [ "*command_line", "*stack_trace" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "wildcard" } } }, { "ecs_path_match_keyword_and_match_only_text": { "path_match": [ "*.title", "*.executable", "*.name", "*.working_directory", "*.full_name", "*.display_name", "*file.path", "*file.target_path", "*os.full", "*email.subject", "*vulnerability.description", "*user_agent.original" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "keyword" } } }, { "ecs_date": { "path_match": [ "*.timestamp", "*_timestamp", "*.not_after", "*.not_before", "*.accessed", "created", "*.created", "*.installed", "*.creation_date", "*.ctime", "*.mtime", "ingested", "*.ingested", "*.start", "*.end", "*.indicator.first_seen", "*.indicator.last_seen", "*.indicator.modified_at", "*threat.enrichments.matched.occurred" ], "unmatch_mapping_type": "object", "mapping": { "type": "date" } } }, { "ecs_path_match_float": { "path_match": [ "*.score.*", "*_score*" ], "path_unmatch": "*.version", "unmatch_mapping_type": "object", "mapping": { "type": "float" } } }, { "ecs_usage_double_scaled_float": { "path_match": "*.usage", "match_mapping_type": [ "double", "long", "string" ], "mapping": { "scaling_factor": 1000, "type": "scaled_float" } } }, { "ecs_geo_point": { "path_match": "*.geo.location", "mapping": { "type": "geo_point" } } }, { "ecs_flattened": { "path_match": [ "*structured_data", "*exports", "*imports" ], "match_mapping_type": "object", "mapping": { "type": "flattened" } } }, { "ecs_gen_ai_integers": { "path_match": [ "*gen_ai.request.max_tokens", "*gen_ai.usage.input_tokens", "*gen_ai.usage.output_tokens", "*gen_ai.request.choice.count", "*gen_ai.request.seed" ], "mapping": { "type": "integer" } } }, { "ecs_gen_ai_doubles": { "path_match": [ "*gen_ai.request.temperature", "*gen_ai.request.top_k", "*gen_ai.request.frequency_penalty", "*gen_ai.request.presence_penalty", "*gen_ai.request.top_p" ], "mapping": { "type": "double" } } }, { "all_strings_to_keywords": { "match_mapping_type": "string", "mapping": { "ignore_above": 1024, "type": "keyword" } } }, { "strings_as_keyword": { "match_mapping_type": "string", "mapping": { "fields": { "text": { "type": "text" } }, "ignore_above": 1024, "type": "keyword" } } }, { "entity_metrics": { "path_match": "entity.metrics.*", "match_mapping_type": [ "long", "double" ], "mapping": { "type": "{dynamic_type}" } } } ], "date_detection": false, "properties": { "@timestamp": { "type": "date" }, "asset": { "properties": { "business_unit": { "type": "keyword" }, "criticality": { "type": "keyword" }, "environment": { "type": "keyword" }, "id": { "type": "keyword" }, "model": { "type": "keyword" }, "name": { "type": "keyword" }, "owner": { "type": "keyword" }, "serial_number": { "type": "keyword" }, "vendor": { "type": "keyword" } } }, "cloud": { "properties": { "account": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "availability_zone": { "type": "keyword" }, "instance": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "machine": { "properties": { "type": { "type": "keyword" } } }, "project": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "provider": { "type": "keyword" }, "region": { "type": "keyword" }, "service": { "properties": { "name": { "type": "keyword" } } } } }, "doc_id": { "type": "keyword", "ignore_above": 1024 }, "doc_index": { "type": "keyword", "ignore_above": 1024 }, "entity": { "properties": { "EngineMetadata": { "properties": { "Type": { "type": "keyword", "ignore_above": 1024 } } }, "attributes": { "properties": { "Asset": { "type": "boolean" }, "Managed": { "type": "boolean" }, "Mfa_enabled": { "type": "boolean" }, "Privileged": { "type": "boolean" } } }, "behaviors": { "properties": { "Brute_force_victim": { "type": "boolean" }, "New_country_login": { "type": "boolean" }, "Used_usb_device": { "type": "boolean" } } }, "definition_id": { "type": "keyword", "ignore_above": 1024 }, "definition_version": { "type": "keyword", "ignore_above": 1024 }, "display_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 1024 } } }, "id": { "type": "keyword" }, "identity_fields": { "type": "keyword" }, "last_seen_timestamp": { "type": "date" }, "lifecycle": { "properties": { "First_seen": { "type": "date" }, "Last_activity": { "type": "date" } } }, "name": { "type": "keyword" }, "risk": { "properties": { "calculated_level": { "type": "keyword" }, "calculated_score": { "type": "float" }, "calculated_score_norm": { "type": "float" } } }, "schema_version": { "type": "keyword", "ignore_above": 1024 }, "source": { "type": "keyword" }, "sub_type": { "type": "keyword" }, "type": { "type": "keyword" }, "url": { "type": "keyword" } } }, "event": { "properties": { "ingested": { "type": "date" } } }, "host": { "properties": { "architecture": { "type": "keyword" }, "boot": { "properties": { "id": { "type": "keyword" } } }, "cpu": { "properties": { "usage": { "type": "keyword" } } }, "disk": { "properties": { "read": { "properties": { "bytes": { "type": "keyword" } } }, "write": { "properties": { "bytes": { "type": "keyword" } } } } }, "domain": { "type": "keyword" }, "hostname": { "type": "keyword" }, "id": { "type": "keyword" }, "ip": { "type": "ip" }, "mac": { "type": "keyword" }, "name": { "type": "keyword" }, "network": { "properties": { "egress": { "properties": { "bytes": { "type": "keyword" }, "packets": { "type": "keyword" } } }, "ingress": { "properties": { "bytes": { "type": "keyword" }, "packets": { "type": "keyword" } } } } }, "pid_ns_ino": { "type": "keyword" }, "type": { "type": "keyword" }, "uptime": { "type": "keyword" } } }, "labels": { "type": "object" }, "orchestrator": { "properties": { "api_version": { "type": "keyword" }, "cluster": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "url": { "type": "keyword" }, "version": { "type": "keyword" } } }, "namespace": { "type": "keyword" }, "organization": { "type": "keyword" }, "resource": { "properties": { "annotation": { "type": "keyword" }, "id": { "type": "keyword" }, "ip": { "type": "keyword" }, "label": { "type": "keyword" }, "name": { "type": "keyword" }, "parent": { "properties": { "type": { "type": "keyword" } } }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" } } }, "tags": { "type": "keyword", "ignore_above": 1024 }, "user": { "properties": { "domain": { "type": "keyword" }, "email": { "type": "keyword" }, "full_name": { "type": "keyword", "fields": { "text": { "type": "match_only_text" } } }, "hash": { "type": "keyword" }, "id": { "type": "keyword" }, "name": { "type": "keyword", "fields": { "text": { "type": "match_only_text" } } }, "roles": { "type": "keyword" } } } } } } ``` 12. reindex data from v1 to v2 index: ``` POST _reindex { "source": { "index": ".entities.v1.latest.security_generic_default" }, "dest": { "index": ".entities.v2.latest.security_generic_default", "op_type": "create" } } ``` 13. go to security -> explore -> network/users/hosts. 14. apply filters to see only events containing graph representation. <img width="4074" height="818" alt="image" src="https://github.com/user-attachments/assets/46605770-73f3-41af-9241-f3013ccc5038" /> 15. open the graph and play with different filters and combinations to get nodes with entity data. 16. graph should work as expected. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…tching entities (elastic#247815) ## Summary This PR Introduces LOOKUP JOIN as the primary entity enrichment mechanism while maintaining backward compatibility with the deprecated ENRICH policy during the transition period. Closes [issue](elastic#232226) and multiple flaky tests due to entity store infra initialization instability. **Server-side changes (fetch_graph.ts)** - Implement LOOKUP JOIN query generation for entity enrichment - Add fallback logic: LOOKUP JOIN → ENRICH policy → no enrichment - Add `getEntitiesLatestIndexName` helper for v2 index names **Test infrastructure** - Add `executeEnrichPolicy` helper to entity_store.ts utils - Create entity_store_v2 test archives with lookup mode mappings - Create entity_store_v2_standard_mode for fallback scenario testing **API integration tests (graph.ts)** - Refactor 'Enrich graph with entity metadata' to test both flows - Add enrichmentConfigs array for ENRICH (v1) and LOOKUP JOIN (v2) - Add fallback test: v2 index exists but not in lookup mode **FTR functional tests** - Update alerts_flyout.ts with dual enrichment config support - Update events_flyout.ts with dual enrichment config support - Reuse entity_store_v2 archives across functional tests **Api/FTR tests coverage** Scenario | v2 Lookup Index | ENRICH Policy | Expected Path | Currently Tested? -- | -- | -- | -- | -- 1 | ✅ Exists in lookup mode | N/A | LOOKUP JOIN | ✅ v2 tests 2 | ❌ Doesn't exist | ✅ Exists | ENRICH | ✅ v1 tests 3 | ❌ Doesn't exist | ❌ Doesn't exist | No enrichment | ✅ All other tests (Happy flows, Validation, etc.) v2 - refers to the new mappings and data mocks we load to test the LOOKUP JOIN functionality - each test could be added just once and it will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we stop supporting querying enrich policies. ## How to test 1. Deploy a local env using the following command: `node scripts/es snapshot --license trial -E path.data=../default -E reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443 -E xpack.security.authc.api_key.enabled=true` 2. run kibana using `yarn start` 3. Go to `Advanced settings` and make sure`securitySolution:enableGraphVisualization` and `securitySolution:enableAssetInventory` features are toggled on. 4. Got to Security -> inventory -> click on 'Enable Asset Inventory'. 5. Install latest gcp-auditlogs integration (skip agent installation) v2.46.0 and above. 6. Install aws-cloudtrail integration (skip agent installation) v4.7.0 and above. 7. Install cloud asset discovery integration (skip agent installation). 8. reindex gcp-auditlogs data from long-live env: ``` POST _reindex { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "<api key>" } }, "index": "logs-*", "query": { "bool": { "must": [ { "term": { "data_stream.dataset": "gcp.audit" } }, { "bool": { "should": [ { "exists": { "field": "user.entity.id" } }, { "exists": { "field": "host.entity.id" } }, { "exists": { "field": "service.entity.id" } }, { "exists": { "field": "entity.id" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "exists": { "field": "user.target.entity.id" } }, { "exists": { "field": "host.target.entity.id" } }, { "exists": { "field": "service.target.entity.id" } }, { "exists": { "field": "entity.target.id" } } ], "minimum_should_match": 1 } } ] } } }, "dest": { "op_type": "create", "index": "logs-gcp.audit-default" } } ``` 9. reindex aws-cloudtrail data from long-live env: ``` POST _reindex { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "ApiKey YmNXcUNaZ0JYd1lMQmZkOEZ1bFc6TDZ3RFNVOXh2R2NEWV9Nb2YyTWxtQQ==" } }, "index": "logs-aws.cloudtrail-default", "query": { "bool": { "must": [ { "bool": { "should": [ { "exists": { "field": "user.entity.id" } }, { "exists": { "field": "host.entity.id" } }, { "exists": { "field": "service.entity.id" } }, { "exists": { "field": "entity.id" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "exists": { "field": "user.target.entity.id" } }, { "exists": { "field": "host.target.entity.id" } }, { "exists": { "field": "service.target.entity.id" } }, { "exists": { "field": "entity.target.id" } } ], "minimum_should_match": 1 } } ] } } }, "dest": { "op_type": "create", "index": "logs-aws.cloudtrail-default" } } ``` 10. reindex entities data from long-live env: ``` POST _reindex?wait_for_completion=true { "conflicts": "proceed", "source": { "remote": { "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443", "socket_timeout": "30s", "connect_timeout": "30s", "headers": { "Authorization": "message for api key" } }, "index": ".entities.v1.latest.security_generic_default", "query": { "bool": { "must": [], "filter": [ { "range": { "@timestamp": { "gte": "now-2y", "lte": "now" } } } ] } } }, "dest": { "op_type": "create", "index": ".entities.v1.latest.security_generic_default" }, "script": { "source": """ ctx._source.doc_id = ctx._id; ctx._source.doc_index = ctx._index; if (ctx._source.asset != null) { if (ctx._source.asset.containsKey('category')) { ctx._source['entity.category'] = ctx._source.asset.category; } if (ctx._source.asset.containsKey('name')) { ctx._source['entity.name'] = ctx._source.asset.name; } if (ctx._source.asset.containsKey('type')) { ctx._source['entity.type'] = ctx._source.asset.type; } if (ctx._source.asset.containsKey('sub_type')) { ctx._source['entity.sub_type'] = ctx._source.asset.sub_type; } if (ctx._source.asset.containsKey('sub_category')) { ctx._source['entity.sub_category'] = ctx._source.asset.sub_category; } } """ } } ``` 11. Create an entities v2 index with lookup mode: ``` PUT .entities.v2.latest.security_generic_default { "settings": { "index": { "mode": "lookup", "number_of_shards": 1, "number_of_replicas": 1 } }, "mappings": { "_meta": { "version": "1.6.0" }, "dynamic_templates": [ { "ecs_timestamp": { "match": "@timestamp", "mapping": { "ignore_malformed": false, "type": "date" } } }, { "ecs_message_match_only_text": { "path_match": [ "message", "*.message" ], "unmatch_mapping_type": "object", "mapping": { "type": "match_only_text" } } }, { "ecs_non_indexed_keyword": { "path_match": [ "*event.original", "*gen_ai.agent.description" ], "mapping": { "doc_values": false, "index": false, "type": "keyword" } } }, { "ecs_non_indexed_long": { "path_match": "*.x509.public_key_exponent", "mapping": { "doc_values": false, "index": false, "type": "long" } } }, { "ecs_ip": { "path_match": [ "ip", "*.ip", "*_ip" ], "match_mapping_type": "string", "mapping": { "type": "ip" } } }, { "ecs_wildcard": { "path_match": [ "*.io.text", "*.message_id", "*registry.data.strings", "*url.path" ], "unmatch_mapping_type": "object", "mapping": { "type": "wildcard" } } }, { "ecs_path_match_wildcard_and_match_only_text": { "path_match": [ "*.body.content", "*url.full", "*url.original" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "wildcard" } } }, { "ecs_match_wildcard_and_match_only_text": { "match": [ "*command_line", "*stack_trace" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "wildcard" } } }, { "ecs_path_match_keyword_and_match_only_text": { "path_match": [ "*.title", "*.executable", "*.name", "*.working_directory", "*.full_name", "*.display_name", "*file.path", "*file.target_path", "*os.full", "*email.subject", "*vulnerability.description", "*user_agent.original" ], "unmatch_mapping_type": "object", "mapping": { "fields": { "text": { "type": "match_only_text" } }, "type": "keyword" } } }, { "ecs_date": { "path_match": [ "*.timestamp", "*_timestamp", "*.not_after", "*.not_before", "*.accessed", "created", "*.created", "*.installed", "*.creation_date", "*.ctime", "*.mtime", "ingested", "*.ingested", "*.start", "*.end", "*.indicator.first_seen", "*.indicator.last_seen", "*.indicator.modified_at", "*threat.enrichments.matched.occurred" ], "unmatch_mapping_type": "object", "mapping": { "type": "date" } } }, { "ecs_path_match_float": { "path_match": [ "*.score.*", "*_score*" ], "path_unmatch": "*.version", "unmatch_mapping_type": "object", "mapping": { "type": "float" } } }, { "ecs_usage_double_scaled_float": { "path_match": "*.usage", "match_mapping_type": [ "double", "long", "string" ], "mapping": { "scaling_factor": 1000, "type": "scaled_float" } } }, { "ecs_geo_point": { "path_match": "*.geo.location", "mapping": { "type": "geo_point" } } }, { "ecs_flattened": { "path_match": [ "*structured_data", "*exports", "*imports" ], "match_mapping_type": "object", "mapping": { "type": "flattened" } } }, { "ecs_gen_ai_integers": { "path_match": [ "*gen_ai.request.max_tokens", "*gen_ai.usage.input_tokens", "*gen_ai.usage.output_tokens", "*gen_ai.request.choice.count", "*gen_ai.request.seed" ], "mapping": { "type": "integer" } } }, { "ecs_gen_ai_doubles": { "path_match": [ "*gen_ai.request.temperature", "*gen_ai.request.top_k", "*gen_ai.request.frequency_penalty", "*gen_ai.request.presence_penalty", "*gen_ai.request.top_p" ], "mapping": { "type": "double" } } }, { "all_strings_to_keywords": { "match_mapping_type": "string", "mapping": { "ignore_above": 1024, "type": "keyword" } } }, { "strings_as_keyword": { "match_mapping_type": "string", "mapping": { "fields": { "text": { "type": "text" } }, "ignore_above": 1024, "type": "keyword" } } }, { "entity_metrics": { "path_match": "entity.metrics.*", "match_mapping_type": [ "long", "double" ], "mapping": { "type": "{dynamic_type}" } } } ], "date_detection": false, "properties": { "@timestamp": { "type": "date" }, "asset": { "properties": { "business_unit": { "type": "keyword" }, "criticality": { "type": "keyword" }, "environment": { "type": "keyword" }, "id": { "type": "keyword" }, "model": { "type": "keyword" }, "name": { "type": "keyword" }, "owner": { "type": "keyword" }, "serial_number": { "type": "keyword" }, "vendor": { "type": "keyword" } } }, "cloud": { "properties": { "account": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "availability_zone": { "type": "keyword" }, "instance": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "machine": { "properties": { "type": { "type": "keyword" } } }, "project": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "provider": { "type": "keyword" }, "region": { "type": "keyword" }, "service": { "properties": { "name": { "type": "keyword" } } } } }, "doc_id": { "type": "keyword", "ignore_above": 1024 }, "doc_index": { "type": "keyword", "ignore_above": 1024 }, "entity": { "properties": { "EngineMetadata": { "properties": { "Type": { "type": "keyword", "ignore_above": 1024 } } }, "attributes": { "properties": { "Asset": { "type": "boolean" }, "Managed": { "type": "boolean" }, "Mfa_enabled": { "type": "boolean" }, "Privileged": { "type": "boolean" } } }, "behaviors": { "properties": { "Brute_force_victim": { "type": "boolean" }, "New_country_login": { "type": "boolean" }, "Used_usb_device": { "type": "boolean" } } }, "definition_id": { "type": "keyword", "ignore_above": 1024 }, "definition_version": { "type": "keyword", "ignore_above": 1024 }, "display_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 1024 } } }, "id": { "type": "keyword" }, "identity_fields": { "type": "keyword" }, "last_seen_timestamp": { "type": "date" }, "lifecycle": { "properties": { "First_seen": { "type": "date" }, "Last_activity": { "type": "date" } } }, "name": { "type": "keyword" }, "risk": { "properties": { "calculated_level": { "type": "keyword" }, "calculated_score": { "type": "float" }, "calculated_score_norm": { "type": "float" } } }, "schema_version": { "type": "keyword", "ignore_above": 1024 }, "source": { "type": "keyword" }, "sub_type": { "type": "keyword" }, "type": { "type": "keyword" }, "url": { "type": "keyword" } } }, "event": { "properties": { "ingested": { "type": "date" } } }, "host": { "properties": { "architecture": { "type": "keyword" }, "boot": { "properties": { "id": { "type": "keyword" } } }, "cpu": { "properties": { "usage": { "type": "keyword" } } }, "disk": { "properties": { "read": { "properties": { "bytes": { "type": "keyword" } } }, "write": { "properties": { "bytes": { "type": "keyword" } } } } }, "domain": { "type": "keyword" }, "hostname": { "type": "keyword" }, "id": { "type": "keyword" }, "ip": { "type": "ip" }, "mac": { "type": "keyword" }, "name": { "type": "keyword" }, "network": { "properties": { "egress": { "properties": { "bytes": { "type": "keyword" }, "packets": { "type": "keyword" } } }, "ingress": { "properties": { "bytes": { "type": "keyword" }, "packets": { "type": "keyword" } } } } }, "pid_ns_ino": { "type": "keyword" }, "type": { "type": "keyword" }, "uptime": { "type": "keyword" } } }, "labels": { "type": "object" }, "orchestrator": { "properties": { "api_version": { "type": "keyword" }, "cluster": { "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "url": { "type": "keyword" }, "version": { "type": "keyword" } } }, "namespace": { "type": "keyword" }, "organization": { "type": "keyword" }, "resource": { "properties": { "annotation": { "type": "keyword" }, "id": { "type": "keyword" }, "ip": { "type": "keyword" }, "label": { "type": "keyword" }, "name": { "type": "keyword" }, "parent": { "properties": { "type": { "type": "keyword" } } }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" } } }, "tags": { "type": "keyword", "ignore_above": 1024 }, "user": { "properties": { "domain": { "type": "keyword" }, "email": { "type": "keyword" }, "full_name": { "type": "keyword", "fields": { "text": { "type": "match_only_text" } } }, "hash": { "type": "keyword" }, "id": { "type": "keyword" }, "name": { "type": "keyword", "fields": { "text": { "type": "match_only_text" } } }, "roles": { "type": "keyword" } } } } } } ``` 12. reindex data from v1 to v2 index: ``` POST _reindex { "source": { "index": ".entities.v1.latest.security_generic_default" }, "dest": { "index": ".entities.v2.latest.security_generic_default", "op_type": "create" } } ``` 13. go to security -> explore -> network/users/hosts. 14. apply filters to see only events containing graph representation. <img width="4074" height="818" alt="image" src="https://github.com/user-attachments/assets/46605770-73f3-41af-9241-f3013ccc5038" /> 15. open the graph and play with different filters and combinations to get nodes with entity data. 16. graph should work as expected. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Summary
This PR Introduces LOOKUP JOIN as the primary entity enrichment mechanism while maintaining backward compatibility with the deprecated ENRICH policy during the transition period.
Closes issue and multiple flaky tests due to entity store infra initialization instability.
Server-side changes (fetch_graph.ts)
getEntitiesLatestIndexNamehelper for v2 index namesTest infrastructure
executeEnrichPolicyhelper to entity_store.ts utilsAPI integration tests (graph.ts)
FTR functional tests
Api/FTR tests coverage
v2 - refers to the new mappings and data mocks we load to test the LOOKUP JOIN functionality - each test could be added just once and it will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we stop supporting querying enrich policies.
How to test
node scripts/es snapshot --license trial -E path.data=../default -E reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443 -E xpack.security.authc.api_key.enabled=trueyarn startAdvanced settingsand make suresecuritySolution:enableGraphVisualizationandsecuritySolution:enableAssetInventoryfeatures are toggled on.Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
release_note:breakinglabel should be applied in these situations.release_note:*label is applied per the guidelinesbackport:*labels.Identify risks
Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.