[Entity Store] Implement logs pagination in CCS

[romulets]

Summary

This PR introduces two major improvements to CCS (cross-cluster search) logs extraction:
log-slice pagination (mirroring what local extraction already had) and independent timestamp
management so CCS no longer relies on the caller to supply its time window.

A third fix resolves a subtle boundary bug in the time-window filter that caused log documents
to be silently dropped when all remaining logs share the same millisecond timestamp.

---

1 — Log-slice pagination for CCS extraction

CCS extraction previously used a single-pass entity loop with no raw-log capping.
It now uses the same two-level pagination that local extraction uses:

Outer loop — log slices
A boundary probe (buildLogPaginationCursorProbeEsql) runs before each entity batch.
It sorts raw logs ascending by (@timestamp, _id), takes the first maxLogsPerPage
documents, and returns the last one as the inclusive slice end (sliceEnd) plus a
total_logs count. When total_logs ≤ maxLogsPerPage the window is exhausted and no
further probe is needed.

Inner loop — entity pages
Within each slice, entities are paginated by (_firstSeenLog, entity.id) up to docsLimit
per query. The slice boundary (sliceEnd) is applied as a compound inclusive upper bound on
every entity page.

State persistence
After each entity page, checkpointTimestamp and paginationRecoveryId are written so a
mid-slice crash can be resumed on the next run without re-processing already-ingested entities.
After a slice completes, checkpointTimestamp advances to the slice end and paginationRecoveryId
is cleared.

---

2 — Independent timestamp management for CCS

CCS extraction no longer receives fromDateISO/toDateISO from the caller.
It now computes and owns its own time window using a new CcsLogExtractionState saved object.

CcsExtractToUpdatesParams changes

Removed	Added
fromDateISO	lookbackPeriod — how far back to look on a fresh start (e.g. '3h')
toDateISO	delay — trailing-edge delay applied to now for toDateISO (e.g. '1m')
	windowOverride? — explicit { fromDateISO, toDateISO } for API-triggered runs

CcsLogExtractionState saved object (new)

Field	Purpose
checkpointTimestamp	_firstSeenLog of the last processed entity; used as fromDateISO on the next run
paginationRecoveryId	Entity ID cursor for mid-slice crash recovery

Window resolution (resolveExtractionWindow)

windowOverride set       →  use it directly; skip all state reads/writes (isOverride = true)
paginationRecoveryId set →  effectiveFrom = checkpointTimestamp, recoveryId = paginationRecoveryId
checkpointTimestamp set  →  effectiveFrom = checkpointTimestamp  (normal continuation)
otherwise                →  effectiveFrom = now − lookbackPeriod  (fresh start)
toDateISO                =  now − delay  (always, unless override)

API-triggered runs (windowOverride set) pass skipStateUpdates = true to both loops so
they never corrupt the scheduled-run checkpoint.

---

Callers updated

• LogsExtractionClient: removes fromDateISO/toDateISO from the CCS call; passes
  lookbackPeriod and delay from config.

• force_ccs_extract_to_updates route: keeps fromDateISO/toDateISO in the request
  body (explicit intent) and forwards them as windowOverride.

  ---

  Testing manually:

1. Start an ECH deployment on 9.4-SNAPSHOT
2. Go to Stack Management > API Keys and generate a new Cross-Cluster api key.

• Save the provided credentials

3. Start kibana and elasticsearch local
4. Add the stored credentials to the local deployment running this command in your CLI:.es/9.4.0/bin/elasticsearch-keystore add cluster.remote.${REMOTE_CLUSTER_NAME}.credentials. This command will prompt you to add the credential.
5. Reload security settings from kibana dev tools POST /_nodes/reload_secure_settings
6. Go to the cloud console of your deployment, under security, at the bottom of the page copy the proxy address

7. Register a new cluster with the proxy address

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.${REMOTE_CLUSTER_NAME}.mode": "proxy",
    "cluster.remote.${REMOTE_CLUSTER_NAME}.proxy_address": "${PROXY_ADDRESS}"
  }
}

8. Add data to the remote cluster observer it be ingested in your environment!

[macroscopeapp]

Catch flakiness early (recommended): run the flaky test runner against this PR before merging.

A new integration test (Should paginate correctly across outer log-slice loop and inner entity-page loop) is added with index creation, data ingestion, and multi-step API interactions — unknown stability.

Trigger a run with the Flaky Test Runner UI or post this comment on the PR:

/flaky scoutConfig:x-pack/solutions/security/plugins/entity_store/test/scout/api/playwright.config.ts:30

^{Share feedback in the #appex-qa channel.}

^{Posted via Macroscope — Flaky Test Runner nudge}

[romulets]

/flaky scoutConfig:x-pack/solutions/security/plugins/entity_store/test/scout/api/playwright.config.ts:30

[kibanamachine]

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#11983

• x-pack/solutions/security/plugins/entity_store/test/scout/api/playwright.config.ts x30

[rudolf]

Core changes lgtm

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#11983

[❌] x-pack/solutions/security/plugins/entity_store/test/scout/api/playwright.config.ts: 29/30 tests passed.

see run history

[florent-leborgne]

minimal docs change - LGTM

[kibanamachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 7df5af9
• Cloud Deployment

Failed CI Steps

• FTR Configs #219

Metrics [docs]

✅ unchanged

History

• 💔 Build #435704 failed f038a26
• 💛 Build #435485 was flaky 984968e

cc @romulets

[kibanamachine]

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/25119149957

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	9.4	Backport failed because of merge conflicts You might need to backport the following PRs to 9.4: - Change WatchlistConfigClient to use internal Elasticsearch client instead of current user (#265966) - [One Workflow] Add form_data support to kibana.request step (#265671) - [Entity Analytics][Leads generation] Improve error states for ES index-level permission denials (#265956) - Add aria-labelledby to Console Config settings controls (#265207) - [Ingest pipelines] Fix Ip location processor bug (#265740) - [Entity Analytics] Auditing usages of documentEntityIdentifiers in user/host flyout (#265887) - fix(zod): adopt upstream lazy-bind memory optimization (colinhacks/zod#5897) (#266343) - [SecuritySolution] [Dashboard Migrations] Add security automatic migrations evaluation suite (#261568)

Manual backport

To create the backport manually run:

node scripts/backport --pr 266307

Questions ?

Please refer to the Backport tool documentation

[romulets]

💚 All backports created successfully

Status	Branch	Result
✅	9.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation