[Security Solution][Detection Engine] adds support of identical ids across multiple indices during ES|QL rule pagination requests

[vitaliidm]

Summary

• addresses [Security Solution][Detection Engine] rework detection rule ES|QL pagination for deduplicated values #216891
• follow up to [Security Solution][Detection Engine] refactors ES|QL pagination in case of alerts deduplication #217069
• accounts for index when excluding ids from subsequent pagination requests. It will prevent situations when identical event ids present in multiple indices and can be excluded when encountered first time for one of the indices
• changes state from excludedDocuments?: ExcludedDocument[]; to excludedDocuments?: Record<string, ExcludedDocument[]>; to store information of index. Where key is document index

Setup

Setup 2 indices with identical documents ids

PUT close_alerts_1
PUT close_alerts_1/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "agent": {
      "properties": {
        "name": {
          "type": "keyword"
        }
      }
    },
    "source": {
      "properties": {
        "ip": {
          "type": "ip"
        }
      }
    },
    "destination": {
      "properties": {
        "ip": {
          "type": "ip"
        },
        "port": {
          "type": "keyword"
        }
      }
    }
  }
}


PUT close_alerts_2
PUT close_alerts_2/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "agent": {
      "properties": {
        "name": {
          "type": "keyword"
        }
      }
    },
    "source": {
      "properties": {
        "ip": {
          "type": "ip"
        }
      }
    },
    "destination": {
      "properties": {
        "ip": {
          "type": "ip"
        },
        "port": {
          "type": "keyword"
        }
      }
    }
  }
}


POST close_alerts_1/_doc/CFnnxZcBgU3wmL9Lp7xG
{
 "@timestamp": "2025-07-01T09:26:30.425Z",
 "source.ip": "127.0.0.1",
 "destination.ip": "127.0.0.1",
 "destination.port": 1,
 "agent.name": "test-0"
}

POST close_alerts_2/_doc/CFnnxZcBgU3wmL9Lp7xG
{
 "@timestamp": "2025-07-01T09:26:30.425Z",
 "source.ip": "127.0.0.1",
 "destination.ip": "127.0.0.1",
 "destination.port": 1,
 "agent.name": "test-0"
}

Exported rule Test rule

query before

POST _query/async?drop_null_columns=true&allow_partial_results=true
{
  "query": "from close_alerts* METADATA _id, _index | limit 101",
  "filter": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "lte": "2025-07-01T12:26:45.110Z",
              "gte": "2025-06-30T11:56:45.110Z",
              "format": "strict_date_optional_time"
            }
          }
        },
        {
          "bool": {
            "must": [],
            "filter": [],
            "should": [],
            "must_not": []
          }
        }
      ],
      "must_not": {
        "ids": {
          "values": [
            "CFnnxZcBgU3wmL9Lp7xG"
          ]
        }
      }
    }
  },
  "wait_for_completion_timeout": "4m",
  "keep_alive": "60s"
}

query after

POST _query/async?drop_null_columns=true&allow_partial_results=true
{
  "query": "from close_alerts* METADATA _id, _index | limit 101",
  "filter": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "lte": "2025-07-01T12:27:57.472Z",
              "gte": "2025-06-30T11:57:57.472Z",
              "format": "strict_date_optional_time"
            }
          }
        },
        {
          "bool": {
            "must": [],
            "filter": [],
            "should": [],
            "must_not": []
          }
        }
      ],
      "must_not": [
        {
          "bool": {
            "filter": [
              {
                "ids": {
                  "values": [
                    "CFnnxZcBgU3wmL9Lp7xG"
                  ]
                }
              },
              {
                "term": {
                  "_index": "close_alerts_1"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "filter": [
              {
                "ids": {
                  "values": [
                    "CFnnxZcBgU3wmL9Lp7xG"
                  ]
                }
              },
              {
                "term": {
                  "_index": "close_alerts_2"
                }
              }
            ]
          }
        }
      ]
    }
  },
  "wait_for_completion_timeout": "4m",
  "keep_alive": "60s"
}

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#8503

[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/esql/trial_license_complete_tier/configs/ess.config.ts: 173/200 tests passed.

see run history

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8507

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/esql/trial_license_complete_tier/configs/ess.config.ts: 200/200 tests passed.

see run history

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#8511

[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/esql/trial_license_complete_tier/configs/ess.config.ts: 174/200 tests passed.

see run history

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[dhurley14]

Looks great!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 93e51bc

Metrics [docs]

✅ unchanged

History

• 💚 Build #316672 succeeded 7fa51e1
• 💛 Build #315001 was flaky 13ed623
• 💔 Build #314902 failed 1c294de
• 💔 Build #314856 failed 93f2159
• 💔 Build #314660 failed e1a2c66
• 💔 Build #314419 failed 8faf226

cc @vitaliidm

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 219278 locally
cc: @vitaliidm