[Security Assistant] KnowledgeBaseRetrievalTool and AskAboutESQLTool prompt updates#228074
Merged
stephmilovic merged 2 commits intoelastic:mainfrom Jul 15, 2025
Merged
[Security Assistant] KnowledgeBaseRetrievalTool and AskAboutESQLTool prompt updates#228074stephmilovic merged 2 commits intoelastic:mainfrom
KnowledgeBaseRetrievalTool and AskAboutESQLTool prompt updates#228074stephmilovic merged 2 commits intoelastic:mainfrom
Conversation
stephmilovic
added
release_note:skip
stephmilovic
added
backport:version
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
spong
approved these changes
Jul 15, 2025
Member
spong
left a comment
spong
left a comment
There was a problem hiding this comment.
LGTM! 👍 Thanks for pairing on these changes @stephmilovic, and the extra error fix too!
Contributor
|
Starting backport for target branches: 8.19, 9.1 https://github.com/elastic/kibana/actions/runs/16306297315 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jul 15, 2025
…ol` prompt updates (elastic#228074) (cherry picked from commit 9c53a52)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jul 15, 2025
…ol` prompt updates (elastic#228074) (cherry picked from commit 9c53a52)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
1 task
kibanamachine
added a commit
that referenced
this pull request
Jul 16, 2025
…ESQLTool` prompt updates (#228074) (#228112) # Backport This will backport the following commits from `main` to `9.1`: - [[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)](#228074) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Steph Milovic","email":"stephanie.milovic@elastic.co"},"sourceCommit":{"committedDate":"2025-07-15T23:10:39Z","message":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)","sha":"9c53a52b48ac41706e77f3f595f4a2d947f97751","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Security Generative AI","backport:version","v9.1.0","v8.19.0","v9.2.0"],"title":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates","number":228074,"url":"https://github.com/elastic/kibana/pull/228074","mergeCommit":{"message":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)","sha":"9c53a52b48ac41706e77f3f595f4a2d947f97751"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/228074","number":228074,"mergeCommit":{"message":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)","sha":"9c53a52b48ac41706e77f3f595f4a2d947f97751"}}]}] BACKPORT--> Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
kibanamachine
added a commit
that referenced
this pull request
Jul 16, 2025
…tESQLTool` prompt updates (#228074) (#228111) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)](#228074) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Steph Milovic","email":"stephanie.milovic@elastic.co"},"sourceCommit":{"committedDate":"2025-07-15T23:10:39Z","message":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)","sha":"9c53a52b48ac41706e77f3f595f4a2d947f97751","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Security Generative AI","backport:version","v9.1.0","v8.19.0","v9.2.0"],"title":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates","number":228074,"url":"https://github.com/elastic/kibana/pull/228074","mergeCommit":{"message":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)","sha":"9c53a52b48ac41706e77f3f595f4a2d947f97751"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/228074","number":228074,"mergeCommit":{"message":"[Security Assistant] `KnowledgeBaseRetrievalTool` and `AskAboutESQLTool` prompt updates (#228074)","sha":"9c53a52b48ac41706e77f3f595f4a2d947f97751"}}]}] BACKPORT--> --------- Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Bluefinger
pushed a commit
to Bluefinger/kibana
that referenced
this pull request
Jul 22, 2025
…ol` prompt updates (elastic#228074)
kertal
pushed a commit
to kertal/kibana
that referenced
this pull request
Jul 25, 2025
…ol` prompt updates (elastic#228074)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@spong and I iterated on the
KnowledgeBaseRetrievalToolprompt as we noticed poor evals, especially for GPT-4.1. This prompt brings us to 100% correctness rateAdditionally, I noticed the following error:
At first I thought the prompt was missing, but it turns out it was a capitalization mismatch (
AskAboutEsqlToolvsAskAboutESQLTool). The capitalization is corrected in this PR, and here in the integrations repo: elastic/integrations#14555