Skip to content

[Contextual Security] Replace ENRICH with LOOKUP JOIN queries when fetching entities#247815

Merged
alexreal1314 merged 36 commits intoelastic:mainfrom
alexreal1314:232226-lookup-entity-enrichment
Jan 21, 2026
Merged

[Contextual Security] Replace ENRICH with LOOKUP JOIN queries when fetching entities#247815
alexreal1314 merged 36 commits intoelastic:mainfrom
alexreal1314:232226-lookup-entity-enrichment

Conversation

@alexreal1314
Copy link
Copy Markdown
Contributor

@alexreal1314 alexreal1314 commented Jan 5, 2026

Summary

This PR Introduces LOOKUP JOIN as the primary entity enrichment mechanism while maintaining backward compatibility with the deprecated ENRICH policy during the transition period.
Closes issue and multiple flaky tests due to entity store infra initialization instability.

Server-side changes (fetch_graph.ts)

  • Implement LOOKUP JOIN query generation for entity enrichment
  • Add fallback logic: LOOKUP JOIN → ENRICH policy → no enrichment
  • Add getEntitiesLatestIndexName helper for v2 index names

Test infrastructure

  • Add executeEnrichPolicy helper to entity_store.ts utils
  • Create entity_store_v2 test archives with lookup mode mappings
  • Create entity_store_v2_standard_mode for fallback scenario testing

API integration tests (graph.ts)

  • Refactor 'Enrich graph with entity metadata' to test both flows
  • Add enrichmentConfigs array for ENRICH (v1) and LOOKUP JOIN (v2)
  • Add fallback test: v2 index exists but not in lookup mode

FTR functional tests

  • Update alerts_flyout.ts with dual enrichment config support
  • Update events_flyout.ts with dual enrichment config support
  • Reuse entity_store_v2 archives across functional tests

Api/FTR tests coverage

Scenario v2 Lookup Index ENRICH Policy Expected Path Currently Tested?
1 ✅ Exists in lookup mode N/A LOOKUP JOIN ✅ v2 tests
2 ❌ Doesn't exist ✅ Exists ENRICH ✅ v1 tests
3 ❌ Doesn't exist ❌ Doesn't exist No enrichment ✅ All other tests (Happy flows, Validation, etc.)

v2 - refers to the new mappings and data mocks we load to test the LOOKUP JOIN functionality - each test could be added just once and it will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we stop supporting querying enrich policies.

How to test

  1. Deploy a local env using the following command:
    node scripts/es snapshot --license trial -E path.data=../default -E reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443 -E xpack.security.authc.api_key.enabled=true
  2. run kibana using yarn start
  3. Go to Advanced settings and make suresecuritySolution:enableGraphVisualization and securitySolution:enableAssetInventory features are toggled on.
  4. Got to Security -> inventory -> click on 'Enable Asset Inventory'.
  5. Install latest gcp-auditlogs integration (skip agent installation) v2.46.0 and above.
  6. Install aws-cloudtrail integration (skip agent installation) v4.7.0 and above.
  7. Install cloud asset discovery integration (skip agent installation).
  8. reindex gcp-auditlogs data from long-live env:
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "<api key>"
      }
    },
    "index": "logs-*",
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "data_stream.dataset": "gcp.audit"
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-gcp.audit-default"
  }
}
  1. reindex aws-cloudtrail data from long-live env:
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "<api key>"
      }
    },
    "index": "logs-aws.cloudtrail-default",
        "query": {
      "bool": {
        "must": [
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-aws.cloudtrail-default"
  }
}
  1. reindex entities data from long-live env:
POST _reindex?wait_for_completion=true
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "<api key>"
      }
    },
    "index": ".entities.v1.latest.security_generic_default",
    "query": {
      "bool": {
        "must": [],
        "filter": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-2y",
                "lte": "now"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": ".entities.v1.latest.security_generic_default"
  },
  "script": {
    "source": """
      ctx._source.doc_id = ctx._id;
      ctx._source.doc_index = ctx._index;

      if (ctx._source.asset != null) {
        if (ctx._source.asset.containsKey('category')) {
          ctx._source['entity.category'] = ctx._source.asset.category;
        }
        if (ctx._source.asset.containsKey('name')) {
          ctx._source['entity.name'] = ctx._source.asset.name;
        }
        if (ctx._source.asset.containsKey('type')) {
          ctx._source['entity.type'] = ctx._source.asset.type;
        }
        if (ctx._source.asset.containsKey('sub_type')) {
          ctx._source['entity.sub_type'] = ctx._source.asset.sub_type;
        }
        if (ctx._source.asset.containsKey('sub_category')) {
          ctx._source['entity.sub_category'] = ctx._source.asset.sub_category;
        }
      }
    """
  }
}
  1. Create an entities v2 index with lookup mode:
PUT .entities.v2.latest.security_generic_default
{
  "settings": {
    "index": {
      "mode": "lookup",
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  },
  "mappings": {
      "_meta": {
        "version": "1.6.0"
      },
      "dynamic_templates": [
        {
          "ecs_timestamp": {
            "match": "@timestamp",
            "mapping": {
              "ignore_malformed": false,
              "type": "date"
            }
          }
        },
        {
          "ecs_message_match_only_text": {
            "path_match": [
              "message",
              "*.message"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "ecs_non_indexed_keyword": {
            "path_match": [
              "*event.original",
              "*gen_ai.agent.description"
            ],
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "keyword"
            }
          }
        },
        {
          "ecs_non_indexed_long": {
            "path_match": "*.x509.public_key_exponent",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "long"
            }
          }
        },
        {
          "ecs_ip": {
            "path_match": [
              "ip",
              "*.ip",
              "*_ip"
            ],
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "ecs_wildcard": {
            "path_match": [
              "*.io.text",
              "*.message_id",
              "*registry.data.strings",
              "*url.path"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_wildcard_and_match_only_text": {
            "path_match": [
              "*.body.content",
              "*url.full",
              "*url.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_match_wildcard_and_match_only_text": {
            "match": [
              "*command_line",
              "*stack_trace"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "*.title",
              "*.executable",
              "*.name",
              "*.working_directory",
              "*.full_name",
              "*.display_name",
              "*file.path",
              "*file.target_path",
              "*os.full",
              "*email.subject",
              "*vulnerability.description",
              "*user_agent.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        },
        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end",
              "*.indicator.first_seen",
              "*.indicator.last_seen",
              "*.indicator.modified_at",
              "*threat.enrichments.matched.occurred"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        },
        {
          "ecs_path_match_float": {
            "path_match": [
              "*.score.*",
              "*_score*"
            ],
            "path_unmatch": "*.version",
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "float"
            }
          }
        },
        {
          "ecs_usage_double_scaled_float": {
            "path_match": "*.usage",
            "match_mapping_type": [
              "double",
              "long",
              "string"
            ],
            "mapping": {
              "scaling_factor": 1000,
              "type": "scaled_float"
            }
          }
        },
        {
          "ecs_geo_point": {
            "path_match": "*.geo.location",
            "mapping": {
              "type": "geo_point"
            }
          }
        },
        {
          "ecs_flattened": {
            "path_match": [
              "*structured_data",
              "*exports",
              "*imports"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
        {
          "ecs_gen_ai_integers": {
            "path_match": [
              "*gen_ai.request.max_tokens",
              "*gen_ai.usage.input_tokens",
              "*gen_ai.usage.output_tokens",
              "*gen_ai.request.choice.count",
              "*gen_ai.request.seed"
            ],
            "mapping": {
              "type": "integer"
            }
          }
        },
        {
          "ecs_gen_ai_doubles": {
            "path_match": [
              "*gen_ai.request.temperature",
              "*gen_ai.request.top_k",
              "*gen_ai.request.frequency_penalty",
              "*gen_ai.request.presence_penalty",
              "*gen_ai.request.top_p"
            ],
            "mapping": {
              "type": "double"
            }
          }
        },
        {
          "all_strings_to_keywords": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "text": {
                  "type": "text"
                }
              },
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "entity_metrics": {
            "path_match": "entity.metrics.*",
            "match_mapping_type": [
              "long",
              "double"
            ],
            "mapping": {
              "type": "{dynamic_type}"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "asset": {
          "properties": {
            "business_unit": {
              "type": "keyword"
            },
            "criticality": {
              "type": "keyword"
            },
            "environment": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "model": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "owner": {
              "type": "keyword"
            },
            "serial_number": {
              "type": "keyword"
            },
            "vendor": {
              "type": "keyword"
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "availability_zone": {
              "type": "keyword"
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "keyword"
                }
              }
            },
            "project": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "provider": {
              "type": "keyword"
            },
            "region": {
              "type": "keyword"
            },
            "service": {
              "properties": {
                "name": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "doc_id": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "doc_index": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "entity": {
          "properties": {
            "EngineMetadata": {
              "properties": {
                "Type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "attributes": {
              "properties": {
                "Asset": {
                  "type": "boolean"
                },
                "Managed": {
                  "type": "boolean"
                },
                "Mfa_enabled": {
                  "type": "boolean"
                },
                "Privileged": {
                  "type": "boolean"
                }
              }
            },
            "behaviors": {
              "properties": {
                "Brute_force_victim": {
                  "type": "boolean"
                },
                "New_country_login": {
                  "type": "boolean"
                },
                "Used_usb_device": {
                  "type": "boolean"
                }
              }
            },
            "definition_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "definition_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "display_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "id": {
              "type": "keyword"
            },
            "identity_fields": {
              "type": "keyword"
            },
            "last_seen_timestamp": {
              "type": "date"
            },
            "lifecycle": {
              "properties": {
                "First_seen": {
                  "type": "date"
                },
                "Last_activity": {
                  "type": "date"
                }
              }
            },
            "name": {
              "type": "keyword"
            },
            "risk": {
              "properties": {
                "calculated_level": {
                  "type": "keyword"
                },
                "calculated_score": {
                  "type": "float"
                },
                "calculated_score_norm": {
                  "type": "float"
                }
              }
            },
            "schema_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "source": {
              "type": "keyword"
            },
            "sub_type": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "url": {
              "type": "keyword"
            }
          }
        },
        "event": {
          "properties": {
            "ingested": {
              "type": "date"
            }
          }
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "keyword"
            },
            "boot": {
              "properties": {
                "id": {
                  "type": "keyword"
                }
              }
            },
            "cpu": {
              "properties": {
                "usage": {
                  "type": "keyword"
                }
              }
            },
            "disk": {
              "properties": {
                "read": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                },
                "write": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "domain": {
              "type": "keyword"
            },
            "hostname": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            },
            "mac": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "network": {
              "properties": {
                "egress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                },
                "ingress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "pid_ns_ino": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "uptime": {
              "type": "keyword"
            }
          }
        },
        "labels": {
          "type": "object"
        },
        "orchestrator": {
          "properties": {
            "api_version": {
              "type": "keyword"
            },
            "cluster": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "url": {
                  "type": "keyword"
                },
                "version": {
                  "type": "keyword"
                }
              }
            },
            "namespace": {
              "type": "keyword"
            },
            "organization": {
              "type": "keyword"
            },
            "resource": {
              "properties": {
                "annotation": {
                  "type": "keyword"
                },
                "id": {
                  "type": "keyword"
                },
                "ip": {
                  "type": "keyword"
                },
                "label": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "parent": {
                  "properties": {
                    "type": {
                      "type": "keyword"
                    }
                  }
                },
                "type": {
                  "type": "keyword"
                }
              }
            },
            "type": {
              "type": "keyword"
            }
          }
        },
        "tags": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "user": {
          "properties": {
            "domain": {
              "type": "keyword"
            },
            "email": {
              "type": "keyword"
            },
            "full_name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "hash": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "roles": {
              "type": "keyword"
            }
          }
        }
      }
    }
}

  1. reindex data from v1 to v2 index:
POST _reindex
{
  "source": { "index": ".entities.v1.latest.security_generic_default" },
  "dest":   { "index": ".entities.v2.latest.security_generic_default", "op_type": "create" }
}
  1. go to security -> explore -> network/users/hosts.
  2. apply filters to see only events containing graph representation.
image 15. open the graph and play with different filters and combinations to get nodes with entity data. 16. graph should work as expected.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

…RICH policy

refactor graph api entity enrichment section to test both LOOKUP and ENRICH query flows

update alerts and events graph FTR tests to test both LOOKUP and ENRICH flows
@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10296

[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed.
[❌] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 23/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10298

[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed.
[❌] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 23/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10299

[✅] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 25/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10300

[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10301

[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10302

[✅] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 25/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10305

[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10306

[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10309

[❌] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 24/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10312

[❌] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 0/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10433

[✅] x-pack/solutions/security/test/cloud_security_posture_api/config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/cloud_security_posture_functional/config.ts: 25/25 tests passed.

see run history

* Recursively removes properties with string value "undefined" from an object.
* This handles cases where ESQL COALESCE returns "undefined" as a fallback string.
*/
const filterUndefinedStringValues = <T extends Record<string, unknown>>(obj: T): T => {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to filter undefined from the response?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would like to keep the undefined properties to not be returned as we had before - there is no benefit IMO returning such values.

Comment on lines +332 to +355
| DROP entity.id
| DROP entity.target.id
// rename entity.*fields before next pipeline to avoid name collisions
| EVAL entity.id = actorEntityId
| LOOKUP JOIN ${getEntitiesLatestIndexName(spaceId)} ON entity.id
| RENAME actorEntityName = entity.name
| RENAME actorEntityType = entity.type
| RENAME actorEntitySubType = entity.sub_type
| RENAME actorHostIp = host.ip
| RENAME actorLookupEntityId = entity.id

| EVAL entity.id = targetEntityId
| LOOKUP JOIN ${getEntitiesLatestIndexName(spaceId)} ON entity.id
| RENAME targetEntityName = entity.name
| RENAME targetEntityType = entity.type
| RENAME targetEntitySubType = entity.sub_type
| RENAME targetHostIp = host.ip
| RENAME targetLookupEntityId = entity.id

${buildEnrichedEntityFieldsEsql()}
`
: isEnrichPolicyExists
? `
// Use ENRICH policy for entity enrichment (deprecated fallback)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: In a follow-up PR, I would split this whole conditional chunk into small functions, each of them returning the clauses for lookup, enrich & fallback. As of now, it's becoming a challenge to understand the query.

But for this PR, feel free to merge without refactoring

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point, extracted lookup and enrich query builders to utility functions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a miss understanding regarding how we should populate undefined. this is unnecessary. Lets fix how we return undefined first. And then you'll not need utility such as filterUndefinedStringValues

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: sorry for the confusion, replace undefined with null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to me, when actorEntityType is null
the generated code is
,"type":"undefined"
while it is expected to be
,"type": undefined

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: sorry for the confusion, replace undefined with null.

* Generates ESQL statements for building entity fields with enrichment data.
* This is used when entity store enrichment is available (via LOOKUP JOIN or ENRICH).
*/
const buildEnrichedEntityFieldsEsql = (): string => {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd move this function to esql.utils.ts too since buildEnrichPolicyEsql and buildLookupJoinEsql live there already

add integration test for entities with partial data (name-only or
type/sub_type-only)
Copy link
Copy Markdown
Contributor

@albertoblaz albertoblaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! 👏

Copy link
Copy Markdown
Contributor

@kfirpeled kfirpeled left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Both ENRICH and LOOKUP JOIN worked locally

| EVAL actorEntityField = CASE(
actorEntityName IS NOT NULL OR actorEntityType IS NOT NULL OR actorEntitySubType IS NOT NULL,
CONCAT(",\\"entity\\":", "{",
${formatJsonProperty('name', 'actorEntityName', false)},
REPLACE(CONCAT(",\\"entity\\":", "{",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of REPLACE

you can switch the order of the fields
so that availableInEntityStore and ecsParentField are first

Reduces the complexity

CONCAT(",\\"entity\\":", "{",
"\\"availableInEntityStore\\":false",
",\\"ecsParentField\\":\\"", actorEntityFieldHint, "\\"",
"}")
)
| EVAL targetEntityField = CASE(
targetEntityName IS NOT NULL OR targetEntityType IS NOT NULL OR targetEntitySubType IS NOT NULL,
CONCAT(",\\"entity\\":", "{",
${formatJsonProperty('name', 'targetEntityName', false)},
REPLACE(CONCAT(",\\"entity\\":", "{",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 21, 2026

💚 Build Succeeded

Metrics [docs]

Unknown metric groups

API count

id before after diff
@kbn/cloud-security-posture-common 233 234 +1

History

cc @alexreal1314

@alexreal1314 alexreal1314 merged commit ebbaa4c into elastic:main Jan 21, 2026
16 checks passed
yuliia-fryshko pushed a commit to yuliia-fryshko/kibana that referenced this pull request Jan 22, 2026
…tching entities (elastic#247815)

## Summary

This PR Introduces LOOKUP JOIN as the primary entity enrichment
mechanism while maintaining backward compatibility with the deprecated
ENRICH policy during the transition period.
Closes [issue](elastic#232226) and
multiple flaky tests due to entity store infra initialization
instability.


**Server-side changes (fetch_graph.ts)**
- Implement LOOKUP JOIN query generation for entity enrichment
- Add fallback logic: LOOKUP JOIN → ENRICH policy → no enrichment
- Add `getEntitiesLatestIndexName` helper for v2 index names

**Test infrastructure**
- Add `executeEnrichPolicy` helper to entity_store.ts utils
- Create entity_store_v2 test archives with lookup mode mappings
- Create entity_store_v2_standard_mode for fallback scenario testing

**API integration tests (graph.ts)**
- Refactor 'Enrich graph with entity metadata' to test both flows
- Add enrichmentConfigs array for ENRICH (v1) and LOOKUP JOIN (v2)
- Add fallback test: v2 index exists but not in lookup mode

**FTR functional tests**
- Update alerts_flyout.ts with dual enrichment config support
- Update events_flyout.ts with dual enrichment config support
- Reuse entity_store_v2 archives across functional tests

**Api/FTR tests coverage**

Scenario | v2 Lookup Index | ENRICH Policy | Expected Path | Currently
Tested?
-- | -- | -- | -- | --
1 | ✅ Exists in lookup mode | N/A | LOOKUP JOIN | ✅ v2 tests
2 | ❌ Doesn't exist | ✅ Exists | ENRICH | ✅ v1 tests
3 | ❌ Doesn't exist | ❌ Doesn't exist | No enrichment | ✅ All other
tests (Happy flows, Validation, etc.)

v2 - refers to the new mappings and data mocks we load to test the
LOOKUP JOIN functionality - each test could be added just once and it
will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we
stop supporting querying enrich policies.




## How to test
1. Deploy a local env using the following command:
`node scripts/es snapshot --license trial -E path.data=../default -E
reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443
-E xpack.security.authc.api_key.enabled=true`
2. run kibana using `yarn start`
3. Go to `Advanced settings` and make
sure`securitySolution:enableGraphVisualization` and
`securitySolution:enableAssetInventory` features are toggled on.
4. Got to Security -> inventory -> click on 'Enable Asset Inventory'.
5. Install latest gcp-auditlogs integration (skip agent installation)
v2.46.0 and above.
6. Install aws-cloudtrail integration (skip agent installation) v4.7.0
and above.
7. Install cloud asset discovery integration (skip agent installation).
8. reindex gcp-auditlogs data from long-live env:
```
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "<api key>"
      }
    },
    "index": "logs-*",
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "data_stream.dataset": "gcp.audit"
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-gcp.audit-default"
  }
}
```

9. reindex aws-cloudtrail data from long-live env:
```
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "ApiKey YmNXcUNaZ0JYd1lMQmZkOEZ1bFc6TDZ3RFNVOXh2R2NEWV9Nb2YyTWxtQQ=="
      }
    },
    "index": "logs-aws.cloudtrail-default",
        "query": {
      "bool": {
        "must": [
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-aws.cloudtrail-default"
  }
}
```

10. reindex entities data from long-live env:

```
POST _reindex?wait_for_completion=true
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "message for api key"
      }
    },
    "index": ".entities.v1.latest.security_generic_default",
    "query": {
      "bool": {
        "must": [],
        "filter": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-2y",
                "lte": "now"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": ".entities.v1.latest.security_generic_default"
  },
  "script": {
    "source": """
      ctx._source.doc_id = ctx._id;
      ctx._source.doc_index = ctx._index;

      if (ctx._source.asset != null) {
        if (ctx._source.asset.containsKey('category')) {
          ctx._source['entity.category'] = ctx._source.asset.category;
        }
        if (ctx._source.asset.containsKey('name')) {
          ctx._source['entity.name'] = ctx._source.asset.name;
        }
        if (ctx._source.asset.containsKey('type')) {
          ctx._source['entity.type'] = ctx._source.asset.type;
        }
        if (ctx._source.asset.containsKey('sub_type')) {
          ctx._source['entity.sub_type'] = ctx._source.asset.sub_type;
        }
        if (ctx._source.asset.containsKey('sub_category')) {
          ctx._source['entity.sub_category'] = ctx._source.asset.sub_category;
        }
      }
    """
  }
}
```

11. Create an entities v2 index with lookup mode:

```
PUT .entities.v2.latest.security_generic_default
{
  "settings": {
    "index": {
      "mode": "lookup",
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  },
  "mappings": {
      "_meta": {
        "version": "1.6.0"
      },
      "dynamic_templates": [
        {
          "ecs_timestamp": {
            "match": "@timestamp",
            "mapping": {
              "ignore_malformed": false,
              "type": "date"
            }
          }
        },
        {
          "ecs_message_match_only_text": {
            "path_match": [
              "message",
              "*.message"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "ecs_non_indexed_keyword": {
            "path_match": [
              "*event.original",
              "*gen_ai.agent.description"
            ],
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "keyword"
            }
          }
        },
        {
          "ecs_non_indexed_long": {
            "path_match": "*.x509.public_key_exponent",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "long"
            }
          }
        },
        {
          "ecs_ip": {
            "path_match": [
              "ip",
              "*.ip",
              "*_ip"
            ],
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "ecs_wildcard": {
            "path_match": [
              "*.io.text",
              "*.message_id",
              "*registry.data.strings",
              "*url.path"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_wildcard_and_match_only_text": {
            "path_match": [
              "*.body.content",
              "*url.full",
              "*url.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_match_wildcard_and_match_only_text": {
            "match": [
              "*command_line",
              "*stack_trace"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "*.title",
              "*.executable",
              "*.name",
              "*.working_directory",
              "*.full_name",
              "*.display_name",
              "*file.path",
              "*file.target_path",
              "*os.full",
              "*email.subject",
              "*vulnerability.description",
              "*user_agent.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        },
        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end",
              "*.indicator.first_seen",
              "*.indicator.last_seen",
              "*.indicator.modified_at",
              "*threat.enrichments.matched.occurred"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        },
        {
          "ecs_path_match_float": {
            "path_match": [
              "*.score.*",
              "*_score*"
            ],
            "path_unmatch": "*.version",
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "float"
            }
          }
        },
        {
          "ecs_usage_double_scaled_float": {
            "path_match": "*.usage",
            "match_mapping_type": [
              "double",
              "long",
              "string"
            ],
            "mapping": {
              "scaling_factor": 1000,
              "type": "scaled_float"
            }
          }
        },
        {
          "ecs_geo_point": {
            "path_match": "*.geo.location",
            "mapping": {
              "type": "geo_point"
            }
          }
        },
        {
          "ecs_flattened": {
            "path_match": [
              "*structured_data",
              "*exports",
              "*imports"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
        {
          "ecs_gen_ai_integers": {
            "path_match": [
              "*gen_ai.request.max_tokens",
              "*gen_ai.usage.input_tokens",
              "*gen_ai.usage.output_tokens",
              "*gen_ai.request.choice.count",
              "*gen_ai.request.seed"
            ],
            "mapping": {
              "type": "integer"
            }
          }
        },
        {
          "ecs_gen_ai_doubles": {
            "path_match": [
              "*gen_ai.request.temperature",
              "*gen_ai.request.top_k",
              "*gen_ai.request.frequency_penalty",
              "*gen_ai.request.presence_penalty",
              "*gen_ai.request.top_p"
            ],
            "mapping": {
              "type": "double"
            }
          }
        },
        {
          "all_strings_to_keywords": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "text": {
                  "type": "text"
                }
              },
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "entity_metrics": {
            "path_match": "entity.metrics.*",
            "match_mapping_type": [
              "long",
              "double"
            ],
            "mapping": {
              "type": "{dynamic_type}"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "asset": {
          "properties": {
            "business_unit": {
              "type": "keyword"
            },
            "criticality": {
              "type": "keyword"
            },
            "environment": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "model": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "owner": {
              "type": "keyword"
            },
            "serial_number": {
              "type": "keyword"
            },
            "vendor": {
              "type": "keyword"
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "availability_zone": {
              "type": "keyword"
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "keyword"
                }
              }
            },
            "project": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "provider": {
              "type": "keyword"
            },
            "region": {
              "type": "keyword"
            },
            "service": {
              "properties": {
                "name": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "doc_id": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "doc_index": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "entity": {
          "properties": {
            "EngineMetadata": {
              "properties": {
                "Type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "attributes": {
              "properties": {
                "Asset": {
                  "type": "boolean"
                },
                "Managed": {
                  "type": "boolean"
                },
                "Mfa_enabled": {
                  "type": "boolean"
                },
                "Privileged": {
                  "type": "boolean"
                }
              }
            },
            "behaviors": {
              "properties": {
                "Brute_force_victim": {
                  "type": "boolean"
                },
                "New_country_login": {
                  "type": "boolean"
                },
                "Used_usb_device": {
                  "type": "boolean"
                }
              }
            },
            "definition_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "definition_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "display_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "id": {
              "type": "keyword"
            },
            "identity_fields": {
              "type": "keyword"
            },
            "last_seen_timestamp": {
              "type": "date"
            },
            "lifecycle": {
              "properties": {
                "First_seen": {
                  "type": "date"
                },
                "Last_activity": {
                  "type": "date"
                }
              }
            },
            "name": {
              "type": "keyword"
            },
            "risk": {
              "properties": {
                "calculated_level": {
                  "type": "keyword"
                },
                "calculated_score": {
                  "type": "float"
                },
                "calculated_score_norm": {
                  "type": "float"
                }
              }
            },
            "schema_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "source": {
              "type": "keyword"
            },
            "sub_type": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "url": {
              "type": "keyword"
            }
          }
        },
        "event": {
          "properties": {
            "ingested": {
              "type": "date"
            }
          }
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "keyword"
            },
            "boot": {
              "properties": {
                "id": {
                  "type": "keyword"
                }
              }
            },
            "cpu": {
              "properties": {
                "usage": {
                  "type": "keyword"
                }
              }
            },
            "disk": {
              "properties": {
                "read": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                },
                "write": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "domain": {
              "type": "keyword"
            },
            "hostname": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            },
            "mac": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "network": {
              "properties": {
                "egress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                },
                "ingress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "pid_ns_ino": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "uptime": {
              "type": "keyword"
            }
          }
        },
        "labels": {
          "type": "object"
        },
        "orchestrator": {
          "properties": {
            "api_version": {
              "type": "keyword"
            },
            "cluster": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "url": {
                  "type": "keyword"
                },
                "version": {
                  "type": "keyword"
                }
              }
            },
            "namespace": {
              "type": "keyword"
            },
            "organization": {
              "type": "keyword"
            },
            "resource": {
              "properties": {
                "annotation": {
                  "type": "keyword"
                },
                "id": {
                  "type": "keyword"
                },
                "ip": {
                  "type": "keyword"
                },
                "label": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "parent": {
                  "properties": {
                    "type": {
                      "type": "keyword"
                    }
                  }
                },
                "type": {
                  "type": "keyword"
                }
              }
            },
            "type": {
              "type": "keyword"
            }
          }
        },
        "tags": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "user": {
          "properties": {
            "domain": {
              "type": "keyword"
            },
            "email": {
              "type": "keyword"
            },
            "full_name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "hash": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "roles": {
              "type": "keyword"
            }
          }
        }
      }
    }
}

```

12. reindex data from v1 to v2 index:

```
POST _reindex
{
  "source": { "index": ".entities.v1.latest.security_generic_default" },
  "dest":   { "index": ".entities.v2.latest.security_generic_default", "op_type": "create" }
}
```
13. go to security -> explore -> network/users/hosts.
14. apply filters to see only events containing graph representation.
<img width="4074" height="818" alt="image"
src="https://github.com/user-attachments/assets/46605770-73f3-41af-9241-f3013ccc5038"
/>
15. open the graph and play with different filters and combinations to
get nodes with entity data.
16. graph should work as expected.




### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
qn895 pushed a commit to qn895/kibana that referenced this pull request Jan 22, 2026
…tching entities (elastic#247815)

## Summary

This PR Introduces LOOKUP JOIN as the primary entity enrichment
mechanism while maintaining backward compatibility with the deprecated
ENRICH policy during the transition period.
Closes [issue](elastic#232226) and
multiple flaky tests due to entity store infra initialization
instability.


**Server-side changes (fetch_graph.ts)**
- Implement LOOKUP JOIN query generation for entity enrichment
- Add fallback logic: LOOKUP JOIN → ENRICH policy → no enrichment
- Add `getEntitiesLatestIndexName` helper for v2 index names

**Test infrastructure**
- Add `executeEnrichPolicy` helper to entity_store.ts utils
- Create entity_store_v2 test archives with lookup mode mappings
- Create entity_store_v2_standard_mode for fallback scenario testing

**API integration tests (graph.ts)**
- Refactor 'Enrich graph with entity metadata' to test both flows
- Add enrichmentConfigs array for ENRICH (v1) and LOOKUP JOIN (v2)
- Add fallback test: v2 index exists but not in lookup mode

**FTR functional tests**
- Update alerts_flyout.ts with dual enrichment config support
- Update events_flyout.ts with dual enrichment config support
- Reuse entity_store_v2 archives across functional tests

**Api/FTR tests coverage**

Scenario | v2 Lookup Index | ENRICH Policy | Expected Path | Currently
Tested?
-- | -- | -- | -- | --
1 | ✅ Exists in lookup mode | N/A | LOOKUP JOIN | ✅ v2 tests
2 | ❌ Doesn't exist | ✅ Exists | ENRICH | ✅ v1 tests
3 | ❌ Doesn't exist | ❌ Doesn't exist | No enrichment | ✅ All other
tests (Happy flows, Validation, etc.)

v2 - refers to the new mappings and data mocks we load to test the
LOOKUP JOIN functionality - each test could be added just once and it
will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we
stop supporting querying enrich policies.




## How to test
1. Deploy a local env using the following command:
`node scripts/es snapshot --license trial -E path.data=../default -E
reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443
-E xpack.security.authc.api_key.enabled=true`
2. run kibana using `yarn start`
3. Go to `Advanced settings` and make
sure`securitySolution:enableGraphVisualization` and
`securitySolution:enableAssetInventory` features are toggled on.
4. Got to Security -> inventory -> click on 'Enable Asset Inventory'.
5. Install latest gcp-auditlogs integration (skip agent installation)
v2.46.0 and above.
6. Install aws-cloudtrail integration (skip agent installation) v4.7.0
and above.
7. Install cloud asset discovery integration (skip agent installation).
8. reindex gcp-auditlogs data from long-live env:
```
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "<api key>"
      }
    },
    "index": "logs-*",
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "data_stream.dataset": "gcp.audit"
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-gcp.audit-default"
  }
}
```

9. reindex aws-cloudtrail data from long-live env:
```
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "ApiKey YmNXcUNaZ0JYd1lMQmZkOEZ1bFc6TDZ3RFNVOXh2R2NEWV9Nb2YyTWxtQQ=="
      }
    },
    "index": "logs-aws.cloudtrail-default",
        "query": {
      "bool": {
        "must": [
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-aws.cloudtrail-default"
  }
}
```

10. reindex entities data from long-live env:

```
POST _reindex?wait_for_completion=true
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "message for api key"
      }
    },
    "index": ".entities.v1.latest.security_generic_default",
    "query": {
      "bool": {
        "must": [],
        "filter": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-2y",
                "lte": "now"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": ".entities.v1.latest.security_generic_default"
  },
  "script": {
    "source": """
      ctx._source.doc_id = ctx._id;
      ctx._source.doc_index = ctx._index;

      if (ctx._source.asset != null) {
        if (ctx._source.asset.containsKey('category')) {
          ctx._source['entity.category'] = ctx._source.asset.category;
        }
        if (ctx._source.asset.containsKey('name')) {
          ctx._source['entity.name'] = ctx._source.asset.name;
        }
        if (ctx._source.asset.containsKey('type')) {
          ctx._source['entity.type'] = ctx._source.asset.type;
        }
        if (ctx._source.asset.containsKey('sub_type')) {
          ctx._source['entity.sub_type'] = ctx._source.asset.sub_type;
        }
        if (ctx._source.asset.containsKey('sub_category')) {
          ctx._source['entity.sub_category'] = ctx._source.asset.sub_category;
        }
      }
    """
  }
}
```

11. Create an entities v2 index with lookup mode:

```
PUT .entities.v2.latest.security_generic_default
{
  "settings": {
    "index": {
      "mode": "lookup",
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  },
  "mappings": {
      "_meta": {
        "version": "1.6.0"
      },
      "dynamic_templates": [
        {
          "ecs_timestamp": {
            "match": "@timestamp",
            "mapping": {
              "ignore_malformed": false,
              "type": "date"
            }
          }
        },
        {
          "ecs_message_match_only_text": {
            "path_match": [
              "message",
              "*.message"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "ecs_non_indexed_keyword": {
            "path_match": [
              "*event.original",
              "*gen_ai.agent.description"
            ],
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "keyword"
            }
          }
        },
        {
          "ecs_non_indexed_long": {
            "path_match": "*.x509.public_key_exponent",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "long"
            }
          }
        },
        {
          "ecs_ip": {
            "path_match": [
              "ip",
              "*.ip",
              "*_ip"
            ],
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "ecs_wildcard": {
            "path_match": [
              "*.io.text",
              "*.message_id",
              "*registry.data.strings",
              "*url.path"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_wildcard_and_match_only_text": {
            "path_match": [
              "*.body.content",
              "*url.full",
              "*url.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_match_wildcard_and_match_only_text": {
            "match": [
              "*command_line",
              "*stack_trace"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "*.title",
              "*.executable",
              "*.name",
              "*.working_directory",
              "*.full_name",
              "*.display_name",
              "*file.path",
              "*file.target_path",
              "*os.full",
              "*email.subject",
              "*vulnerability.description",
              "*user_agent.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        },
        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end",
              "*.indicator.first_seen",
              "*.indicator.last_seen",
              "*.indicator.modified_at",
              "*threat.enrichments.matched.occurred"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        },
        {
          "ecs_path_match_float": {
            "path_match": [
              "*.score.*",
              "*_score*"
            ],
            "path_unmatch": "*.version",
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "float"
            }
          }
        },
        {
          "ecs_usage_double_scaled_float": {
            "path_match": "*.usage",
            "match_mapping_type": [
              "double",
              "long",
              "string"
            ],
            "mapping": {
              "scaling_factor": 1000,
              "type": "scaled_float"
            }
          }
        },
        {
          "ecs_geo_point": {
            "path_match": "*.geo.location",
            "mapping": {
              "type": "geo_point"
            }
          }
        },
        {
          "ecs_flattened": {
            "path_match": [
              "*structured_data",
              "*exports",
              "*imports"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
        {
          "ecs_gen_ai_integers": {
            "path_match": [
              "*gen_ai.request.max_tokens",
              "*gen_ai.usage.input_tokens",
              "*gen_ai.usage.output_tokens",
              "*gen_ai.request.choice.count",
              "*gen_ai.request.seed"
            ],
            "mapping": {
              "type": "integer"
            }
          }
        },
        {
          "ecs_gen_ai_doubles": {
            "path_match": [
              "*gen_ai.request.temperature",
              "*gen_ai.request.top_k",
              "*gen_ai.request.frequency_penalty",
              "*gen_ai.request.presence_penalty",
              "*gen_ai.request.top_p"
            ],
            "mapping": {
              "type": "double"
            }
          }
        },
        {
          "all_strings_to_keywords": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "text": {
                  "type": "text"
                }
              },
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "entity_metrics": {
            "path_match": "entity.metrics.*",
            "match_mapping_type": [
              "long",
              "double"
            ],
            "mapping": {
              "type": "{dynamic_type}"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "asset": {
          "properties": {
            "business_unit": {
              "type": "keyword"
            },
            "criticality": {
              "type": "keyword"
            },
            "environment": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "model": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "owner": {
              "type": "keyword"
            },
            "serial_number": {
              "type": "keyword"
            },
            "vendor": {
              "type": "keyword"
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "availability_zone": {
              "type": "keyword"
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "keyword"
                }
              }
            },
            "project": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "provider": {
              "type": "keyword"
            },
            "region": {
              "type": "keyword"
            },
            "service": {
              "properties": {
                "name": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "doc_id": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "doc_index": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "entity": {
          "properties": {
            "EngineMetadata": {
              "properties": {
                "Type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "attributes": {
              "properties": {
                "Asset": {
                  "type": "boolean"
                },
                "Managed": {
                  "type": "boolean"
                },
                "Mfa_enabled": {
                  "type": "boolean"
                },
                "Privileged": {
                  "type": "boolean"
                }
              }
            },
            "behaviors": {
              "properties": {
                "Brute_force_victim": {
                  "type": "boolean"
                },
                "New_country_login": {
                  "type": "boolean"
                },
                "Used_usb_device": {
                  "type": "boolean"
                }
              }
            },
            "definition_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "definition_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "display_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "id": {
              "type": "keyword"
            },
            "identity_fields": {
              "type": "keyword"
            },
            "last_seen_timestamp": {
              "type": "date"
            },
            "lifecycle": {
              "properties": {
                "First_seen": {
                  "type": "date"
                },
                "Last_activity": {
                  "type": "date"
                }
              }
            },
            "name": {
              "type": "keyword"
            },
            "risk": {
              "properties": {
                "calculated_level": {
                  "type": "keyword"
                },
                "calculated_score": {
                  "type": "float"
                },
                "calculated_score_norm": {
                  "type": "float"
                }
              }
            },
            "schema_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "source": {
              "type": "keyword"
            },
            "sub_type": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "url": {
              "type": "keyword"
            }
          }
        },
        "event": {
          "properties": {
            "ingested": {
              "type": "date"
            }
          }
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "keyword"
            },
            "boot": {
              "properties": {
                "id": {
                  "type": "keyword"
                }
              }
            },
            "cpu": {
              "properties": {
                "usage": {
                  "type": "keyword"
                }
              }
            },
            "disk": {
              "properties": {
                "read": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                },
                "write": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "domain": {
              "type": "keyword"
            },
            "hostname": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            },
            "mac": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "network": {
              "properties": {
                "egress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                },
                "ingress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "pid_ns_ino": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "uptime": {
              "type": "keyword"
            }
          }
        },
        "labels": {
          "type": "object"
        },
        "orchestrator": {
          "properties": {
            "api_version": {
              "type": "keyword"
            },
            "cluster": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "url": {
                  "type": "keyword"
                },
                "version": {
                  "type": "keyword"
                }
              }
            },
            "namespace": {
              "type": "keyword"
            },
            "organization": {
              "type": "keyword"
            },
            "resource": {
              "properties": {
                "annotation": {
                  "type": "keyword"
                },
                "id": {
                  "type": "keyword"
                },
                "ip": {
                  "type": "keyword"
                },
                "label": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "parent": {
                  "properties": {
                    "type": {
                      "type": "keyword"
                    }
                  }
                },
                "type": {
                  "type": "keyword"
                }
              }
            },
            "type": {
              "type": "keyword"
            }
          }
        },
        "tags": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "user": {
          "properties": {
            "domain": {
              "type": "keyword"
            },
            "email": {
              "type": "keyword"
            },
            "full_name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "hash": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "roles": {
              "type": "keyword"
            }
          }
        }
      }
    }
}

```

12. reindex data from v1 to v2 index:

```
POST _reindex
{
  "source": { "index": ".entities.v1.latest.security_generic_default" },
  "dest":   { "index": ".entities.v2.latest.security_generic_default", "op_type": "create" }
}
```
13. go to security -> explore -> network/users/hosts.
14. apply filters to see only events containing graph representation.
<img width="4074" height="818" alt="image"
src="https://github.com/user-attachments/assets/46605770-73f3-41af-9241-f3013ccc5038"
/>
15. open the graph and play with different filters and combinations to
get nodes with entity data.
16. graph should work as expected.




### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
dennis-tismenko pushed a commit to dennis-tismenko/kibana that referenced this pull request Jan 22, 2026
…tching entities (elastic#247815)

## Summary

This PR Introduces LOOKUP JOIN as the primary entity enrichment
mechanism while maintaining backward compatibility with the deprecated
ENRICH policy during the transition period.
Closes [issue](elastic#232226) and
multiple flaky tests due to entity store infra initialization
instability.


**Server-side changes (fetch_graph.ts)**
- Implement LOOKUP JOIN query generation for entity enrichment
- Add fallback logic: LOOKUP JOIN → ENRICH policy → no enrichment
- Add `getEntitiesLatestIndexName` helper for v2 index names

**Test infrastructure**
- Add `executeEnrichPolicy` helper to entity_store.ts utils
- Create entity_store_v2 test archives with lookup mode mappings
- Create entity_store_v2_standard_mode for fallback scenario testing

**API integration tests (graph.ts)**
- Refactor 'Enrich graph with entity metadata' to test both flows
- Add enrichmentConfigs array for ENRICH (v1) and LOOKUP JOIN (v2)
- Add fallback test: v2 index exists but not in lookup mode

**FTR functional tests**
- Update alerts_flyout.ts with dual enrichment config support
- Update events_flyout.ts with dual enrichment config support
- Reuse entity_store_v2 archives across functional tests

**Api/FTR tests coverage**

Scenario | v2 Lookup Index | ENRICH Policy | Expected Path | Currently
Tested?
-- | -- | -- | -- | --
1 | ✅ Exists in lookup mode | N/A | LOOKUP JOIN | ✅ v2 tests
2 | ❌ Doesn't exist | ✅ Exists | ENRICH | ✅ v1 tests
3 | ❌ Doesn't exist | ❌ Doesn't exist | No enrichment | ✅ All other
tests (Happy flows, Validation, etc.)

v2 - refers to the new mappings and data mocks we load to test the
LOOKUP JOIN functionality - each test could be added just once and it
will be tested in both scenarios - using ENRICH and LOOKUP JOIN until we
stop supporting querying enrich policies.




## How to test
1. Deploy a local env using the following command:
`node scripts/es snapshot --license trial -E path.data=../default -E
reindex.remote.whitelist=kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443
-E xpack.security.authc.api_key.enabled=true`
2. run kibana using `yarn start`
3. Go to `Advanced settings` and make
sure`securitySolution:enableGraphVisualization` and
`securitySolution:enableAssetInventory` features are toggled on.
4. Got to Security -> inventory -> click on 'Enable Asset Inventory'.
5. Install latest gcp-auditlogs integration (skip agent installation)
v2.46.0 and above.
6. Install aws-cloudtrail integration (skip agent installation) v4.7.0
and above.
7. Install cloud asset discovery integration (skip agent installation).
8. reindex gcp-auditlogs data from long-live env:
```
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "<api key>"
      }
    },
    "index": "logs-*",
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "data_stream.dataset": "gcp.audit"
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-gcp.audit-default"
  }
}
```

9. reindex aws-cloudtrail data from long-live env:
```
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "ApiKey YmNXcUNaZ0JYd1lMQmZkOEZ1bFc6TDZ3RFNVOXh2R2NEWV9Nb2YyTWxtQQ=="
      }
    },
    "index": "logs-aws.cloudtrail-default",
        "query": {
      "bool": {
        "must": [
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.entity.id" } },
                { "exists": { "field": "host.entity.id" } },
                { "exists": { "field": "service.entity.id" } },
                { "exists": { "field": "entity.id" } }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                { "exists": { "field": "user.target.entity.id" } },
                { "exists": { "field": "host.target.entity.id" } },
                { "exists": { "field": "service.target.entity.id" } },
                { "exists": { "field": "entity.target.id" } }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-aws.cloudtrail-default"
  }
}
```

10. reindex entities data from long-live env:

```
POST _reindex?wait_for_completion=true
{
  "conflicts": "proceed",
  "source": {
    "remote": {
      "host": "https://kfir-graph-viz-wip-ba715e.es.eu-west-1.aws.qa.elastic.cloud:443",
      "socket_timeout": "30s",
      "connect_timeout": "30s",
      "headers": {
        "Authorization": "message for api key"
      }
    },
    "index": ".entities.v1.latest.security_generic_default",
    "query": {
      "bool": {
        "must": [],
        "filter": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-2y",
                "lte": "now"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": ".entities.v1.latest.security_generic_default"
  },
  "script": {
    "source": """
      ctx._source.doc_id = ctx._id;
      ctx._source.doc_index = ctx._index;

      if (ctx._source.asset != null) {
        if (ctx._source.asset.containsKey('category')) {
          ctx._source['entity.category'] = ctx._source.asset.category;
        }
        if (ctx._source.asset.containsKey('name')) {
          ctx._source['entity.name'] = ctx._source.asset.name;
        }
        if (ctx._source.asset.containsKey('type')) {
          ctx._source['entity.type'] = ctx._source.asset.type;
        }
        if (ctx._source.asset.containsKey('sub_type')) {
          ctx._source['entity.sub_type'] = ctx._source.asset.sub_type;
        }
        if (ctx._source.asset.containsKey('sub_category')) {
          ctx._source['entity.sub_category'] = ctx._source.asset.sub_category;
        }
      }
    """
  }
}
```

11. Create an entities v2 index with lookup mode:

```
PUT .entities.v2.latest.security_generic_default
{
  "settings": {
    "index": {
      "mode": "lookup",
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  },
  "mappings": {
      "_meta": {
        "version": "1.6.0"
      },
      "dynamic_templates": [
        {
          "ecs_timestamp": {
            "match": "@timestamp",
            "mapping": {
              "ignore_malformed": false,
              "type": "date"
            }
          }
        },
        {
          "ecs_message_match_only_text": {
            "path_match": [
              "message",
              "*.message"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "ecs_non_indexed_keyword": {
            "path_match": [
              "*event.original",
              "*gen_ai.agent.description"
            ],
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "keyword"
            }
          }
        },
        {
          "ecs_non_indexed_long": {
            "path_match": "*.x509.public_key_exponent",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "long"
            }
          }
        },
        {
          "ecs_ip": {
            "path_match": [
              "ip",
              "*.ip",
              "*_ip"
            ],
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "ecs_wildcard": {
            "path_match": [
              "*.io.text",
              "*.message_id",
              "*registry.data.strings",
              "*url.path"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_wildcard_and_match_only_text": {
            "path_match": [
              "*.body.content",
              "*url.full",
              "*url.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_match_wildcard_and_match_only_text": {
            "match": [
              "*command_line",
              "*stack_trace"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "*.title",
              "*.executable",
              "*.name",
              "*.working_directory",
              "*.full_name",
              "*.display_name",
              "*file.path",
              "*file.target_path",
              "*os.full",
              "*email.subject",
              "*vulnerability.description",
              "*user_agent.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        },
        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end",
              "*.indicator.first_seen",
              "*.indicator.last_seen",
              "*.indicator.modified_at",
              "*threat.enrichments.matched.occurred"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        },
        {
          "ecs_path_match_float": {
            "path_match": [
              "*.score.*",
              "*_score*"
            ],
            "path_unmatch": "*.version",
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "float"
            }
          }
        },
        {
          "ecs_usage_double_scaled_float": {
            "path_match": "*.usage",
            "match_mapping_type": [
              "double",
              "long",
              "string"
            ],
            "mapping": {
              "scaling_factor": 1000,
              "type": "scaled_float"
            }
          }
        },
        {
          "ecs_geo_point": {
            "path_match": "*.geo.location",
            "mapping": {
              "type": "geo_point"
            }
          }
        },
        {
          "ecs_flattened": {
            "path_match": [
              "*structured_data",
              "*exports",
              "*imports"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
        {
          "ecs_gen_ai_integers": {
            "path_match": [
              "*gen_ai.request.max_tokens",
              "*gen_ai.usage.input_tokens",
              "*gen_ai.usage.output_tokens",
              "*gen_ai.request.choice.count",
              "*gen_ai.request.seed"
            ],
            "mapping": {
              "type": "integer"
            }
          }
        },
        {
          "ecs_gen_ai_doubles": {
            "path_match": [
              "*gen_ai.request.temperature",
              "*gen_ai.request.top_k",
              "*gen_ai.request.frequency_penalty",
              "*gen_ai.request.presence_penalty",
              "*gen_ai.request.top_p"
            ],
            "mapping": {
              "type": "double"
            }
          }
        },
        {
          "all_strings_to_keywords": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "text": {
                  "type": "text"
                }
              },
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        {
          "entity_metrics": {
            "path_match": "entity.metrics.*",
            "match_mapping_type": [
              "long",
              "double"
            ],
            "mapping": {
              "type": "{dynamic_type}"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "asset": {
          "properties": {
            "business_unit": {
              "type": "keyword"
            },
            "criticality": {
              "type": "keyword"
            },
            "environment": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "model": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "owner": {
              "type": "keyword"
            },
            "serial_number": {
              "type": "keyword"
            },
            "vendor": {
              "type": "keyword"
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "availability_zone": {
              "type": "keyword"
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "keyword"
                }
              }
            },
            "project": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "provider": {
              "type": "keyword"
            },
            "region": {
              "type": "keyword"
            },
            "service": {
              "properties": {
                "name": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "doc_id": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "doc_index": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "entity": {
          "properties": {
            "EngineMetadata": {
              "properties": {
                "Type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "attributes": {
              "properties": {
                "Asset": {
                  "type": "boolean"
                },
                "Managed": {
                  "type": "boolean"
                },
                "Mfa_enabled": {
                  "type": "boolean"
                },
                "Privileged": {
                  "type": "boolean"
                }
              }
            },
            "behaviors": {
              "properties": {
                "Brute_force_victim": {
                  "type": "boolean"
                },
                "New_country_login": {
                  "type": "boolean"
                },
                "Used_usb_device": {
                  "type": "boolean"
                }
              }
            },
            "definition_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "definition_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "display_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "id": {
              "type": "keyword"
            },
            "identity_fields": {
              "type": "keyword"
            },
            "last_seen_timestamp": {
              "type": "date"
            },
            "lifecycle": {
              "properties": {
                "First_seen": {
                  "type": "date"
                },
                "Last_activity": {
                  "type": "date"
                }
              }
            },
            "name": {
              "type": "keyword"
            },
            "risk": {
              "properties": {
                "calculated_level": {
                  "type": "keyword"
                },
                "calculated_score": {
                  "type": "float"
                },
                "calculated_score_norm": {
                  "type": "float"
                }
              }
            },
            "schema_version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "source": {
              "type": "keyword"
            },
            "sub_type": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "url": {
              "type": "keyword"
            }
          }
        },
        "event": {
          "properties": {
            "ingested": {
              "type": "date"
            }
          }
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "keyword"
            },
            "boot": {
              "properties": {
                "id": {
                  "type": "keyword"
                }
              }
            },
            "cpu": {
              "properties": {
                "usage": {
                  "type": "keyword"
                }
              }
            },
            "disk": {
              "properties": {
                "read": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                },
                "write": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "domain": {
              "type": "keyword"
            },
            "hostname": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            },
            "mac": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "network": {
              "properties": {
                "egress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                },
                "ingress": {
                  "properties": {
                    "bytes": {
                      "type": "keyword"
                    },
                    "packets": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "pid_ns_ino": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "uptime": {
              "type": "keyword"
            }
          }
        },
        "labels": {
          "type": "object"
        },
        "orchestrator": {
          "properties": {
            "api_version": {
              "type": "keyword"
            },
            "cluster": {
              "properties": {
                "id": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "url": {
                  "type": "keyword"
                },
                "version": {
                  "type": "keyword"
                }
              }
            },
            "namespace": {
              "type": "keyword"
            },
            "organization": {
              "type": "keyword"
            },
            "resource": {
              "properties": {
                "annotation": {
                  "type": "keyword"
                },
                "id": {
                  "type": "keyword"
                },
                "ip": {
                  "type": "keyword"
                },
                "label": {
                  "type": "keyword"
                },
                "name": {
                  "type": "keyword"
                },
                "parent": {
                  "properties": {
                    "type": {
                      "type": "keyword"
                    }
                  }
                },
                "type": {
                  "type": "keyword"
                }
              }
            },
            "type": {
              "type": "keyword"
            }
          }
        },
        "tags": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "user": {
          "properties": {
            "domain": {
              "type": "keyword"
            },
            "email": {
              "type": "keyword"
            },
            "full_name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "hash": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "roles": {
              "type": "keyword"
            }
          }
        }
      }
    }
}

```

12. reindex data from v1 to v2 index:

```
POST _reindex
{
  "source": { "index": ".entities.v1.latest.security_generic_default" },
  "dest":   { "index": ".entities.v2.latest.security_generic_default", "op_type": "create" }
}
```
13. go to security -> explore -> network/users/hosts.
14. apply filters to see only events containing graph representation.
<img width="4074" height="818" alt="image"
src="https://github.com/user-attachments/assets/46605770-73f3-41af-9241-f3013ccc5038"
/>
15. open the graph and play with different filters and combinations to
get nodes with entity data.
16. graph should work as expected.




### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting ci:build-serverless-image ci:cloud-deploy Create or update a Cloud deployment ci:cloud-redeploy Always create a new Cloud deployment ci:project-deploy-security Create a Security Serverless Project release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failing test: X-Pack Cloud Security Posture API Tests.x-pack/solutions/security/test/cloud_security_posture_api/routes/graph·ts - Cloud Security Posture POST /internal/cloud_security_posture/graph Happy flows Enrich graph with entity metadata should enrich graph with multiple targets from different fields with mixed grouping Failing test: X-Pack Cloud Security Posture Functional Tests.x-pack/solutions/security/test/cloud_security_posture_functional/pages/alerts_flyout·ts - Cloud Security Posture Security Alerts Page - Graph visualization ECS fields only expanded flyout - entity enrichment for multiple generic targets - single target field Failing test: X-Pack Cloud Security Posture API Tests.x-pack/solutions/security/test/cloud_security_posture_api/routes/graph·ts - Cloud Security Posture POST /internal/cloud_security_posture/graph Happy flows Enrich graph with entity metadata should contain entity data when asset inventory is enabled Failing test: X-Pack Cloud Security Posture API Tests.x-pack/solutions/security/test/cloud_security_posture_api/routes/graph·ts - Cloud Security Posture POST /internal/cloud_security_posture/graph Happy flows Enrich graph with entity metadata should return enriched data when asset inventory is enabled in a different space - multi target Graph API - Replace usage of ENRICH with LOOKUP JOIN

5 participants