[8.12] [Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (#173121)#173310
Merged
kibanamachine merged 1 commit intoelastic:8.12from Dec 13, 2023
Conversation
…s in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (elastic#173121) ## [Security Solution] [Elastic AI Assistant] Include `acknowledged` alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) This PR updates the query used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts elastic#172542](elastic#172542) to include alerts with a `kibana.alert.workflow_status` value of `acknowledged`. The query previously only returned alerts with a status of `open`. This change ensures both `open` and `acknowledged` alerts are provided as context to the LLM. ### Updated Anonymization defaults Three fields, detailed below, were added as anonymization defaults because they improve the quality of responses from the LLM when it answers questions about alerts. For example, the LLM can refer to specific alerts by ID when the `_id` field is provided. This PR makes the following additive changes to the Assistant's `Anonymization` defaults: | Field | Allow by default | Anonymize by default | Value add | |--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs). | | `kibana.alert.risk_score` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query sorts alerts by `kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses. | | `kibana.alert.workflow_status` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query filters alerts by `kibana.alert.workflow_status` to ensure only `open` and `acknowledged` alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses. | - Clicking the `Reset` button shown in the screenshot below will reset the user's `Anonymization` defaults, such that they include the additive changes in the table above:  ### Updated settings text The text in the settings below was also updated:  ### Desk testing To desk test this change: - Enable the `assistantRagOnAlerts` feature flag described in [elastic#172542](elastic#172542) must be enabled, per the following example: ``` xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts'] ``` - The `Alerts` feature must be enabled in the assistant settings, per the screenshot below:  1) Navigate to Security > Alerts 2) Click the `AI Assistant` button to open the assistant 3) Click the `Settings` gear to open the assistant settings 4) Click the `Anonymization` category 5) Click the `Reset` button, shown in the screenshot below  **Expected results** - `65` fields are allowed by default, per the screenshot above - `12` fields are anonymized by default, per the screenshot above - The `_id` field is allowed by default, per the screenshot above - The `_id` field is anonymized by default, per the screenshot above 6) Type `kibana.alert.risk` in the search box **Expected result** - The `kibana.alert.risk_score` field is allowed by default 7) Type `kibana.alert.workflow` in the search box **Expected result** - The `kibana.alert.workflow_status` field is allowed by default 8) Click `Save` 9) Click the `X` button to clear the conversation 10) Close the assistant 11) Add the following two fields as columns to the Alerts page table: - `kibana.alert.workflow_status` - `_id` 12) Sort the table, first by `kibana.alert.risk_score` from high to low, and then by `@timestamp` from new to old, per the screenshot below:  13) Filter the alerts page to only show `open` and `acknowledged` alerts **Expected result** - The alerts page has custom columns, sorting, and filtering, per the screenshot below:  14) Click the `AI Assistant` button to open the assistant 15) Ask the assistant: ``` What is the workflow status of my alerts? ``` **Expected result** - The assistant will report on the workflow status of alerts, per the example response below: ``` The workflow status for your alerts is currently 'open'. This status was observed on alerts related to processes started by Mimikatz, a known tool used in many cyberattacks, and sequences of processes that are often indicative of malicious activity. The severity of most of these alerts is 'high' or 'critical'. You may want to investigate these alerts further to ensure there's no ongoing threat to your system. ```  16) Close the assistant 17) Change the workflow status of an alert in the Alerts table from `open` to `acknowledged` **Expected result** - The alerts table shows the updated alert, per the screenshot below:  18) Once again, open the assistant 19) Once again, ask the (same) question: ``` What is the workflow status of my alerts? ``` **Expected result** - The response from the assistant makes reference to the alert who's workflow status was changed from `open` to `acknowledged`, per the example response below: ``` Based on the latest information, your alerts mainly show 'open' status, indicating that they have not been resolved yet. Some alerts have been acknowledged. Most of these unaddressed alerts have a critical severity rating and are primarily triggered by a Mimikatz process start and an EQL process sequence. You may want to prioritize these if the severity of the threat they pose is truly high or critical. It's also noteworthy that some alerts have a high severity rating. You should review all of these alerts as soon as possible to ensure your systems are secure. ```  20) Ask the assistant for details about the acknowledged alerts: ``` What are the details of the acknowledged alerts? ``` **Expected result** The assistant for details about the acknowledged alert that, for example, includes the `kibana.alert.risk_score`, per the example response below: ``` In response to your previous question, here are the details of the acknowledged alerts: 1. There is a 'mimikatz process started' alert, which is of 'critical' severity and 'acknowledged' status. It has a high risk score of 99. Its threat tactic is 'Command and Control'. The process involved was 'mimikatz.exe' running with arguments '--fo1'. 2. A 'Threshold rule' alert of 'critical' severity and 'open' status has also been detected with a risk score of 99 and threat tactic 'Collection'. 3. Lastly, there are several 'EQL process sequence' alerts of 'high' severity with 'open' status. These alerts involve execution of various processes including 'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these alerts is 73 and the threat tactic involved is 'Execution'. Please, take appropriate action to address these alerts. ```  21) Ask the assistant for the `_id` of the acknowledged alert: ``` What is the id of the acknowledged alert? ``` **Expected results** - The response from the assistant contains the `_id` of the `acknowledged` alert, per the example response below: ``` The id of the acknowledged alert is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'. This alert was for a 'mimikatz process started' event with a severity rating of 'critical' and a risk score of 99. The host name associated with this alert is 'Host-terkvbzvtj'. ```  - The `_id` shown in the assistant is the same `_id` of the acknowledged alert on the alerts page, per the screeenshot below:  22) Click the `Show anonymized` toggle in the assistant **Expected result** - The `_id` shown in the latest result is replaced with the actual anonymized value that was sent to the LLM, per the example screenshot below:  (cherry picked from commit 0d9c261)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Async chunks
To update your PR or re-run it, just comment with: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.12:Questions ?
Please refer to the Backport tool documentation