[9.4] [Entity Store] Implement logs pagination in CCS (#266307)#266464
Merged
romulets merged 2 commits intoelastic:9.4from Apr 29, 2026
Merged
[9.4] [Entity Store] Implement logs pagination in CCS (#266307)#266464romulets merged 2 commits intoelastic:9.4from
romulets merged 2 commits intoelastic:9.4from
Conversation
## Summary This PR introduces two major improvements to CCS (cross-cluster search) logs extraction: log-slice pagination (mirroring what local extraction already had) and independent timestamp management so CCS no longer relies on the caller to supply its time window. A third fix resolves a subtle boundary bug in the time-window filter that caused log documents to be silently dropped when all remaining logs share the same millisecond timestamp. --- ## 1 — Log-slice pagination for CCS extraction CCS extraction previously used a single-pass entity loop with no raw-log capping. It now uses the same two-level pagination that local extraction uses: **Outer loop — log slices** A boundary probe (`buildLogPaginationCursorProbeEsql`) runs before each entity batch. It sorts raw logs ascending by `(@timestamp, _id)`, takes the first `maxLogsPerPage` documents, and returns the last one as the inclusive slice end (`sliceEnd`) plus a `total_logs` count. When `total_logs ≤ maxLogsPerPage` the window is exhausted and no further probe is needed. **Inner loop — entity pages** Within each slice, entities are paginated by `(_firstSeenLog, entity.id)` up to `docsLimit` per query. The slice boundary (`sliceEnd`) is applied as a compound inclusive upper bound on every entity page. **State persistence** After each entity page, `checkpointTimestamp` and `paginationRecoveryId` are written so a mid-slice crash can be resumed on the next run without re-processing already-ingested entities. After a slice completes, `checkpointTimestamp` advances to the slice end and `paginationRecoveryId` is cleared. --- ## 2 — Independent timestamp management for CCS CCS extraction no longer receives `fromDateISO`/`toDateISO` from the caller. It now computes and owns its own time window using a new `CcsLogExtractionState` saved object. **`CcsExtractToUpdatesParams` changes** | Removed | Added | |---|---| | `fromDateISO` | `lookbackPeriod` — how far back to look on a fresh start (e.g. `'3h'`) | | `toDateISO` | `delay` — trailing-edge delay applied to `now` for `toDateISO` (e.g. `'1m'`) | | | `windowOverride?` — explicit `{ fromDateISO, toDateISO }` for API-triggered runs | **`CcsLogExtractionState` saved object (new)** | Field | Purpose | |---|---| | `checkpointTimestamp` | `_firstSeenLog` of the last processed entity; used as `fromDateISO` on the next run | | `paginationRecoveryId` | Entity ID cursor for mid-slice crash recovery | **Window resolution (`resolveExtractionWindow`)** ``` windowOverride set → use it directly; skip all state reads/writes (isOverride = true) paginationRecoveryId set → effectiveFrom = checkpointTimestamp, recoveryId = paginationRecoveryId checkpointTimestamp set → effectiveFrom = checkpointTimestamp (normal continuation) otherwise → effectiveFrom = now − lookbackPeriod (fresh start) toDateISO = now − delay (always, unless override) ``` API-triggered runs (`windowOverride` set) pass `skipStateUpdates = true` to both loops so they never corrupt the scheduled-run checkpoint. --- ## Callers updated - **`LogsExtractionClient`**: removes `fromDateISO`/`toDateISO` from the CCS call; passes `lookbackPeriod` and `delay` from config. - **`force_ccs_extract_to_updates` route**: keeps `fromDateISO`/`toDateISO` in the request body (explicit intent) and forwards them as `windowOverride`. --- ## Testing manually: 1. Start an ECH deployment on 9.4-SNAPSHOT 2. Go to Stack Management > API Keys and generate a new `Cross-Cluster` api key. - Save the provided credentials 3. Start kibana and elasticsearch local 4. Add the stored credentials to the local deployment running this command in your CLI:`.es/9.4.0/bin/elasticsearch-keystore add cluster.remote.${REMOTE_CLUSTER_NAME}.credentials`. This command will prompt you to add the credential. 5. Reload security settings from kibana dev tools `POST /_nodes/reload_secure_settings` 6. Go to the cloud console of your deployment, under security, at the bottom of the page copy the proxy address <img width="1143" height="194" alt="image" src="https://github.com/user-attachments/assets/19ce4142-c184-466a-a1e1-a91ecdbec18f" /> 7. Register a new cluster with the proxy address ``` PUT _cluster/settings { "persistent": { "cluster.remote.${REMOTE_CLUSTER_NAME}.mode": "proxy", "cluster.remote.${REMOTE_CLUSTER_NAME}.proxy_address": "${PROXY_ADDRESS}" } } ``` 8. Add data to the remote cluster observer it be ingested in your environment! --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 90f1efb) # Conflicts: # x-pack/solutions/security/plugins/entity_store/server/domain/asset_manager/asset_manager_client.test.ts
orouz
approved these changes
Apr 29, 2026
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
💚 Build Succeeded
Metrics [docs]
History
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto9.4:Questions ?
Please refer to the Backport tool documentation