[AI4DSOC] Change the logic to query the last alert ingested in an integration#228602
Merged
PhilippeOberti merged 2 commits intoelastic:mainfrom Jul 28, 2025
Merged
Conversation
PhilippeOberti
added
the
release_note:skip
PhilippeOberti
added
Team:Threat Hunting:Investigations
Contributor
|
Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations) |
agusruidiazgd
approved these changes
Jul 28, 2025
Contributor
agusruidiazgd
left a comment
agusruidiazgd
left a comment
There was a problem hiding this comment.
Thanks for this fix Philippe! 💯
Contributor
|
Starting backport for target branches: 8.19, 9.1 https://github.com/elastic/kibana/actions/runs/16581398428 |
Contributor
💚 Build Succeeded
Metrics [docs]Async chunks
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
|
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jul 28, 2025
…egration (elastic#228602) ## Summary This PR changes the logic used in the AI4DSOC alert summary page integration section to display the last sync value for each integration card displayed at the top of the page In the previous logic we were fetching all dataStreams and use the `last_activity_ms` value to display as the last sync for the integration. We [realized](elastic#220617) that some dataStreams have event not related to ingested alert. For example it could be event mentioning that an error is happening... This meant that the alert summary UI was showing an incorrect last sync value. The new logic leverage a call using ESQL that works as follow: ``` FROM logs-integrationName.alert-default | WHERE event.kind == "alert" | SORT event.ingested DESC | KEEP event.ingested | LIMIT 1 ``` We fetch the information for each integration. We retrieve the last document with `event.kind == 'alert'` then we sort to get the most recent first, select only the `event.ingested` field/value pair and keep only the first document. The value from the `event.ingested` field is then passed to the integration card and displayed for the last sync. The PR also sets up a 30 second interval, to make sure that the values in the integration card last sync section are updated periodically. This will give the user an indication that the rest of the page below (charts and table) might be out of date. https://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios elastic#220617 (cherry picked from commit 5729233)
Contributor
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jul 28, 2025
…an integration (#228602) (#229723) # Backport This will backport the following commits from `main` to `9.1`: - [[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)](#228602) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Philippe Oberti","email":"philippe.oberti@elastic.co"},"sourceCommit":{"committedDate":"2025-07-28T22:05:38Z","message":"[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)\n\n## Summary\n\nThis PR changes the logic used in the AI4DSOC alert summary page\nintegration section to display the last sync value for each integration\ncard displayed at the top of the page\n\nIn the previous logic we were fetching all dataStreams and use the\n`last_activity_ms` value to display as the last sync for the\nintegration. We\n[realized](#220617) that some\ndataStreams have event not related to ingested alert. For example it\ncould be event mentioning that an error is happening... This meant that\nthe alert summary UI was showing an incorrect last sync value.\n\nThe new logic leverage a call using ESQL that works as follow:\n```\nFROM logs-integrationName.alert-default\n | WHERE event.kind == \"alert\"\n | SORT event.ingested DESC\n | KEEP event.ingested\n | LIMIT 1\n```\nWe fetch the information for each integration. We retrieve the last\ndocument with `event.kind == 'alert'` then we sort to get the most\nrecent first, select only the `event.ingested` field/value pair and keep\nonly the first document.\nThe value from the `event.ingested` field is then passed to the\nintegration card and displayed for the last sync.\n\nThe PR also sets up a 30 second interval, to make sure that the values\nin the integration card last sync section are updated periodically. This\nwill give the user an indication that the rest of the page below (charts\nand table) might be out of date.\n\n\nhttps://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892\n\n### Checklist\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\nhttps://github.com//issues/220617","sha":"57292335e3cb4011c71e62afa7b3aaea4c5ce034","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Threat Hunting:Investigations","backport:version","v9.1.0","v8.19.0","v9.2.0"],"title":"[AI4DSOC] Change the logic to query the last alert ingested in an integration","number":228602,"url":"https://github.com/elastic/kibana/pull/228602","mergeCommit":{"message":"[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)\n\n## Summary\n\nThis PR changes the logic used in the AI4DSOC alert summary page\nintegration section to display the last sync value for each integration\ncard displayed at the top of the page\n\nIn the previous logic we were fetching all dataStreams and use the\n`last_activity_ms` value to display as the last sync for the\nintegration. We\n[realized](#220617) that some\ndataStreams have event not related to ingested alert. For example it\ncould be event mentioning that an error is happening... This meant that\nthe alert summary UI was showing an incorrect last sync value.\n\nThe new logic leverage a call using ESQL that works as follow:\n```\nFROM logs-integrationName.alert-default\n | WHERE event.kind == \"alert\"\n | SORT event.ingested DESC\n | KEEP event.ingested\n | LIMIT 1\n```\nWe fetch the information for each integration. We retrieve the last\ndocument with `event.kind == 'alert'` then we sort to get the most\nrecent first, select only the `event.ingested` field/value pair and keep\nonly the first document.\nThe value from the `event.ingested` field is then passed to the\nintegration card and displayed for the last sync.\n\nThe PR also sets up a 30 second interval, to make sure that the values\nin the integration card last sync section are updated periodically. This\nwill give the user an indication that the rest of the page below (charts\nand table) might be out of date.\n\n\nhttps://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892\n\n### Checklist\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\nhttps://github.com//issues/220617","sha":"57292335e3cb4011c71e62afa7b3aaea4c5ce034"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/228602","number":228602,"mergeCommit":{"message":"[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)\n\n## Summary\n\nThis PR changes the logic used in the AI4DSOC alert summary page\nintegration section to display the last sync value for each integration\ncard displayed at the top of the page\n\nIn the previous logic we were fetching all dataStreams and use the\n`last_activity_ms` value to display as the last sync for the\nintegration. We\n[realized](#220617) that some\ndataStreams have event not related to ingested alert. For example it\ncould be event mentioning that an error is happening... This meant that\nthe alert summary UI was showing an incorrect last sync value.\n\nThe new logic leverage a call using ESQL that works as follow:\n```\nFROM logs-integrationName.alert-default\n | WHERE event.kind == \"alert\"\n | SORT event.ingested DESC\n | KEEP event.ingested\n | LIMIT 1\n```\nWe fetch the information for each integration. We retrieve the last\ndocument with `event.kind == 'alert'` then we sort to get the most\nrecent first, select only the `event.ingested` field/value pair and keep\nonly the first document.\nThe value from the `event.ingested` field is then passed to the\nintegration card and displayed for the last sync.\n\nThe PR also sets up a 30 second interval, to make sure that the values\nin the integration card last sync section are updated periodically. This\nwill give the user an indication that the rest of the page below (charts\nand table) might be out of date.\n\n\nhttps://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892\n\n### Checklist\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\nhttps://github.com//issues/220617","sha":"57292335e3cb4011c71e62afa7b3aaea4c5ce034"}}]}] BACKPORT--> Co-authored-by: Philippe Oberti <philippe.oberti@elastic.co>
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added
the
backport missing
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
1 similar comment
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
PhilippeOberti
added a commit
to PhilippeOberti/kibana
that referenced
this pull request
Aug 1, 2025
…egration (elastic#228602) ## Summary This PR changes the logic used in the AI4DSOC alert summary page integration section to display the last sync value for each integration card displayed at the top of the page In the previous logic we were fetching all dataStreams and use the `last_activity_ms` value to display as the last sync for the integration. We [realized](elastic#220617) that some dataStreams have event not related to ingested alert. For example it could be event mentioning that an error is happening... This meant that the alert summary UI was showing an incorrect last sync value. The new logic leverage a call using ESQL that works as follow: ``` FROM logs-integrationName.alert-default | WHERE event.kind == "alert" | SORT event.ingested DESC | KEEP event.ingested | LIMIT 1 ``` We fetch the information for each integration. We retrieve the last document with `event.kind == 'alert'` then we sort to get the most recent first, select only the `event.ingested` field/value pair and keep only the first document. The value from the `event.ingested` field is then passed to the integration card and displayed for the last sync. The PR also sets up a 30 second interval, to make sure that the values in the integration card last sync section are updated periodically. This will give the user an indication that the rest of the page below (charts and table) might be out of date. https://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios elastic#220617 (cherry picked from commit 5729233) # Conflicts: # x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_integrations_last_activity.test.ts
PhilippeOberti
added a commit
that referenced
this pull request
Aug 1, 2025
… an integration (#228602) (#229767) # Backport This will backport the following commits from `main` to `8.19`: - [[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)](#228602) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Philippe Oberti","email":"philippe.oberti@elastic.co"},"sourceCommit":{"committedDate":"2025-07-28T22:05:38Z","message":"[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)\n\n## Summary\n\nThis PR changes the logic used in the AI4DSOC alert summary page\nintegration section to display the last sync value for each integration\ncard displayed at the top of the page\n\nIn the previous logic we were fetching all dataStreams and use the\n`last_activity_ms` value to display as the last sync for the\nintegration. We\n[realized](#220617) that some\ndataStreams have event not related to ingested alert. For example it\ncould be event mentioning that an error is happening... This meant that\nthe alert summary UI was showing an incorrect last sync value.\n\nThe new logic leverage a call using ESQL that works as follow:\n```\nFROM logs-integrationName.alert-default\n | WHERE event.kind == \"alert\"\n | SORT event.ingested DESC\n | KEEP event.ingested\n | LIMIT 1\n```\nWe fetch the information for each integration. We retrieve the last\ndocument with `event.kind == 'alert'` then we sort to get the most\nrecent first, select only the `event.ingested` field/value pair and keep\nonly the first document.\nThe value from the `event.ingested` field is then passed to the\nintegration card and displayed for the last sync.\n\nThe PR also sets up a 30 second interval, to make sure that the values\nin the integration card last sync section are updated periodically. This\nwill give the user an indication that the rest of the page below (charts\nand table) might be out of date.\n\n\nhttps://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892\n\n### Checklist\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\nhttps://github.com//issues/220617","sha":"57292335e3cb4011c71e62afa7b3aaea4c5ce034","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Threat Hunting:Investigations","backport:version","v9.1.0","v8.19.0","v9.2.0"],"title":"[AI4DSOC] Change the logic to query the last alert ingested in an integration","number":228602,"url":"https://github.com/elastic/kibana/pull/228602","mergeCommit":{"message":"[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)\n\n## Summary\n\nThis PR changes the logic used in the AI4DSOC alert summary page\nintegration section to display the last sync value for each integration\ncard displayed at the top of the page\n\nIn the previous logic we were fetching all dataStreams and use the\n`last_activity_ms` value to display as the last sync for the\nintegration. We\n[realized](#220617) that some\ndataStreams have event not related to ingested alert. For example it\ncould be event mentioning that an error is happening... This meant that\nthe alert summary UI was showing an incorrect last sync value.\n\nThe new logic leverage a call using ESQL that works as follow:\n```\nFROM logs-integrationName.alert-default\n | WHERE event.kind == \"alert\"\n | SORT event.ingested DESC\n | KEEP event.ingested\n | LIMIT 1\n```\nWe fetch the information for each integration. We retrieve the last\ndocument with `event.kind == 'alert'` then we sort to get the most\nrecent first, select only the `event.ingested` field/value pair and keep\nonly the first document.\nThe value from the `event.ingested` field is then passed to the\nintegration card and displayed for the last sync.\n\nThe PR also sets up a 30 second interval, to make sure that the values\nin the integration card last sync section are updated periodically. This\nwill give the user an indication that the rest of the page below (charts\nand table) might be out of date.\n\n\nhttps://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892\n\n### Checklist\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\nhttps://github.com//issues/220617","sha":"57292335e3cb4011c71e62afa7b3aaea4c5ce034"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/229723","number":229723,"state":"MERGED","mergeCommit":{"sha":"1526f93c70e561b07bcb1a081b3f45e73bea36b6","message":"[9.1] [AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602) (#229723)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.1`:\n- [[AI4DSOC] Change the logic to query the last alert ingested in an\nintegration (#228602)](https://github.com/elastic/kibana/pull/228602)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Philippe Oberti <philippe.oberti@elastic.co>"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/228602","number":228602,"mergeCommit":{"message":"[AI4DSOC] Change the logic to query the last alert ingested in an integration (#228602)\n\n## Summary\n\nThis PR changes the logic used in the AI4DSOC alert summary page\nintegration section to display the last sync value for each integration\ncard displayed at the top of the page\n\nIn the previous logic we were fetching all dataStreams and use the\n`last_activity_ms` value to display as the last sync for the\nintegration. We\n[realized](#220617) that some\ndataStreams have event not related to ingested alert. For example it\ncould be event mentioning that an error is happening... This meant that\nthe alert summary UI was showing an incorrect last sync value.\n\nThe new logic leverage a call using ESQL that works as follow:\n```\nFROM logs-integrationName.alert-default\n | WHERE event.kind == \"alert\"\n | SORT event.ingested DESC\n | KEEP event.ingested\n | LIMIT 1\n```\nWe fetch the information for each integration. We retrieve the last\ndocument with `event.kind == 'alert'` then we sort to get the most\nrecent first, select only the `event.ingested` field/value pair and keep\nonly the first document.\nThe value from the `event.ingested` field is then passed to the\nintegration card and displayed for the last sync.\n\nThe PR also sets up a 30 second interval, to make sure that the values\nin the integration card last sync section are updated periodically. This\nwill give the user an indication that the rest of the page below (charts\nand table) might be out of date.\n\n\nhttps://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892\n\n### Checklist\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\nhttps://github.com//issues/220617","sha":"57292335e3cb4011c71e62afa7b3aaea4c5ce034"}}]}] BACKPORT-->
kibanamachine
added
v8.19.1
and removed
backport missing
delanni
pushed a commit
to delanni/kibana
that referenced
this pull request
Aug 5, 2025
…egration (elastic#228602) ## Summary This PR changes the logic used in the AI4DSOC alert summary page integration section to display the last sync value for each integration card displayed at the top of the page In the previous logic we were fetching all dataStreams and use the `last_activity_ms` value to display as the last sync for the integration. We [realized](elastic#220617) that some dataStreams have event not related to ingested alert. For example it could be event mentioning that an error is happening... This meant that the alert summary UI was showing an incorrect last sync value. The new logic leverage a call using ESQL that works as follow: ``` FROM logs-integrationName.alert-default | WHERE event.kind == "alert" | SORT event.ingested DESC | KEEP event.ingested | LIMIT 1 ``` We fetch the information for each integration. We retrieve the last document with `event.kind == 'alert'` then we sort to get the most recent first, select only the `event.ingested` field/value pair and keep only the first document. The value from the `event.ingested` field is then passed to the integration card and displayed for the last sync. The PR also sets up a 30 second interval, to make sure that the values in the integration card last sync section are updated periodically. This will give the user an indication that the rest of the page below (charts and table) might be out of date. https://github.com/user-attachments/assets/b3524777-f994-40dd-90d1-0c757d1ec892 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios elastic#220617
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR changes the logic used in the AI4DSOC alert summary page integration section to display the last sync value for each integration card displayed at the top of the page
In the previous logic we were fetching all dataStreams and use the
last_activity_msvalue to display as the last sync for the integration. We realized that some dataStreams have event not related to ingested alert. For example it could be event mentioning that an error is happening... This meant that the alert summary UI was showing an incorrect last sync value.The new logic leverage a call using ESQL that works as follow:
We fetch the information for each integration. We retrieve the last document with
event.kind == 'alert'then we sort to get the most recent first, select only theevent.ingestedfield/value pair and keep only the first document.The value from the
event.ingestedfield is then passed to the integration card and displayed for the last sync.The PR also sets up a 30 second interval, to make sure that the values in the integration card last sync section are updated periodically. This will give the user an indication that the rest of the page below (charts and table) might be out of date.
Screen.Recording.2025-07-18.at.11.54.38.AM.mov
Checklist
#220617