[UII] Create integrations knowledge tool for security assistant by jen-huang · Pull Request #236197 · elastic/kibana

jen-huang · 2025-09-23T21:51:52Z

Summary

Relates to https://github.com/elastic/ingest-dev/issues/5679.

This PR creates a new IntegrationKnowledgeTool for the Security AI Assistant. This tool reads from the new .integrations_knowledge index for information about installed integrations that have KB content. Much of the tool definition is inspired by SecurityLabsKnowledgeBaseTool which is very similar in scope.

.integrations_knowledge system index is read using an internal ES client. The user's question is queried using semantic search against its content field (mapped as sematic_text). Citations include a link to the corresponding integration details page.

Testing

Upload this sample system-2.3.3-NEXT.zip package via Integrations > Create new integration
- The test package just copies the existing docs/README.md to docs/knowledge_base/README.md so that Fleet ingests it into .integrations_knowledge
Use Security AI Assistant to ask questions related to system integrations, such as:
- How can I collect CPU and memory data for my windows host?
- What OS can I run the system integration on?
- What does the system integration do?
Observe that the assistant returns relevant information that is cited from the system integration.

To-do

Duplicate tool definition/prompts to security-ai-prompt package (will be done after this PR merges, so that prompts are finalized)

Checklist

Unit or functional tests were updated or added to match the most common scenarios
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

jen-huang · 2025-09-23T21:57:20Z

+      async (input) => {
+        try {
+          // Search the .integration_knowledge index using semantic search on the content field
+          const response = await assistantContext.core.elasticsearch.client.asInternalUser.search({


kbDataClient.getKnowledgeBaseDocumentEntries(), which is used by the security labs tool, doesn't work here since that reads from .kibana-elastic-ai-assistant-${resource}

esClient from params doesn't work because that is scoped to current user

but I found that I can access the internal client from assistantContext.core :)

…tegrations-kb-tool

elasticmachine · 2025-09-25T20:22:14Z

Pinging @elastic/fleet (Team:Fleet)

spong · 2025-09-29T19:42:21Z

FYI: Adding the ci:security-genai-run-evals label to execute our evals to ensure the new tool doesn't interfere with other tool selections and result in any regressions. This'll need to run before pushing the change mentioned in this comment to only register if the .integration_knowledge exists. Otherwise we'll need to add some setup code here to initialize the index to ensure the tool is registered. Actually, we might want to do this anyway to ensure the tool is always registered and doesn't interfere.

I went ahead and set up the index in this test suite. the evals passed in the latest CI run that includes this change

That link was only showing the agent for me, but I do see the successful eval run here 👍

Checking the eval results in LangSmith and everything looks good to me (runs prefixed with Eval Automation | Build 345505 | PR 236197). Pretty much matches historical runs from what I can tell. We will need to tune the Assistant Eval: Custom Knowledge data set, but this mostly has to due with KB configuration IIRC, not from the new added tool.

spong · 2025-09-29T19:48:44Z

+  {
+    promptId: 'IntegrationKnowledgeTool',
+    promptGroupId,
+    prompt: {
+      default:
+        'Call this for knowledge from Fleet-installed integrations, which contains information on how to configure and use integrations for data ingestion.',
+    },
+  },


We've discussed this offline, but just commenting for visibility that we'll need to do release of the security_ai_prompts integration as the bot mentioned once this is merged.

While technically I don't think you need to do this (it'll just fall back to this local prompt defined here), it's a nice to have option for tweaking the prompt out of band. cc @stephmilovic

yep! I have that as a to-do item in this PR description, instructions recommend to do the integration update after merge as well

…tegrations-kb-tool

spong

Checked out, tested locally and code reviewed -- LGTM!

Thanks for the collab here @jen-huang, this turned out great! Excited to now be delivering knowledge via integrations! 😀 🙌 🚀

kibanamachine · 2025-09-30T19:12:46Z

🤖 Prompt Changes Detected

Changes have been detected to one or more prompt files in the Elastic Assistant plugin.

Please remember to update the integrations repository with your prompt changes to ensure consistency across all deployments.

Next Steps:

Follow the documentation in x-pack/solutions/security/packages/security-ai-prompts/README.md to update the corresponding prompt files
Make the changes in the integrations repository
Test your changes in the integrations environment
Ensure prompt consistency across all deployments

This is an automated reminder to help maintain prompt consistency across repositories.

elasticmachine · 2025-09-30T20:24:52Z

💚 Build Succeeded

Buildkite Build
Commit: 3c24681

Metrics [docs]

✅ unchanged

History

💛 Build #345505 was flaky dc4dcdb
💔 Build #345462 failed dc4dcdb
💔 Build #345415 failed 329cbb8
💔 Build #345299 failed 9c57991
💚 Build #344506 succeeded ab37467
💔 Build #344016 failed 1a3192c

cc @jen-huang

…tic#236197) ## Summary Relates to elastic/ingest-dev#5679. This PR creates a new `IntegrationKnowledgeTool` for the Security AI Assistant. This tool reads from the new `.integrations_knowledge` index for information about installed integrations that have KB content. Much of the tool definition is inspired by `SecurityLabsKnowledgeBaseTool` which is very similar in scope. `.integrations_knowledge` system index is read using an internal ES client. The user's question is queried using semantic search against its `content` field (mapped as `sematic_text`). Citations include a link to the corresponding integration details page. ## Testing 1. Upload this sample [system-2.3.3-NEXT.zip](https://github.com/user-attachments/files/22546766/system-2.3.3-NEXT.zip) package via Integrations > Create new integration - The test package just copies the existing `docs/README.md` to `docs/knowledge_base/README.md` so that Fleet ingests it into `.integrations_knowledge` 2. Use Security AI Assistant to ask questions related to system integrations, such as: - How can I collect CPU and memory data for my windows host? - What OS can I run the system integration on? - What does the system integration do? 3. Observe that the assistant returns relevant information that is cited from the system integration. <img width="685" height="837" alt="image" src="https://github.com/user-attachments/assets/5b333aaa-7757-483f-9fa2-abc6aa8abd40" /> ## To-do - [ ] Duplicate tool definition/prompts to `security-ai-prompt` package (will be done after this PR merges, so that prompts are finalized) ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Garrett Spong <spong@users.noreply.github.com>

## Summary Adds an integration knowledge tool to Agent Builder that retrieves documentation from Fleet-installed integrations using semantic search on the `.integration_knowledge` index. The tool uses the conditional availability pattern and is only available when the integration knowledge index exists. <p align="center"> <img width="405" src="https://github.com/user-attachments/assets/640d4f54-34cc-47e3-b731-b3913139e84e" /> <img width="395" src="https://github.com/user-attachments/assets/fd66c044-5536-4947-98d8-45e4b168b34c" /> </p> ## Changes * Added `platform.core.integration_knowledge` builtin tool to `agent_builder_platform` that searches Fleet integration documentation * Tool is registered in plugin `setup()` with conditional availability using the `availability` configuration pattern * Availability is checked at runtime via ES search on `.integration_knowledge` index (using `size: 0` query) * Returns structured resource results with package name, version, filename, and content ## Technical Details * Tool registration added to `registerTools()` in plugin `setup()` phase, following the same pattern as `productDocumentationTool` * Uses `availability` configuration with `cacheMode: 'space'` to conditionally show/hide the tool based on index availability * Searches using Elasticsearch semantic search on the `content` field * `esClient.asInternalUser` is used for both handler execution and availability checking (index permissions require internal user) * Results include reference URLs to integration detail pages (`/app/integrations/detail/{package_name}`) ## Considerations * Tool requires Fleet to have indexed integration knowledge into `.integration_knowledge` * Tool availability is checked per-space and cached for performance * No Kibana restart required - tool appears/disappears dynamically based on index availability * This is the onechat/Agent Builder equivalent of the existing `IntegrationKnowledgeTool` in Security Solution's Assistant (#236197) and Observability Solution's Assistant (#237085) added in `9.2`. --- ## Testing > [!NOTE] > You must enable the `xpack.fleet.enableExperimental: ["installIntegrationsKnowledge"]` feature flag until this PR enabling it by default is merged (#245080). 1. Upload this sample [system-2.3.3-NEXT.zip](https://github.com/user-attachments/files/22546766/system-2.3.3-NEXT.zip) package via Integrations > Create new integration - The test package just copies the existing `docs/README.md` to `docs/knowledge_base/README.md` so that Fleet ingests it into `.integrations_knowledge` 2. Create new Agent with the new Integration Knowledge tool and ask questions related to system integrations, such as: - How can I collect CPU and memory data for my windows host? - What OS can I run the system integration on? - What does the system integration do? 3. Observe that the responses returned contain relevant information that is cited from the system integration. _PR developed with Cursor + Opus 4.5_ --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

) ## Summary Adds an integration knowledge tool to Agent Builder that retrieves documentation from Fleet-installed integrations using semantic search on the `.integration_knowledge` index. The tool uses the conditional availability pattern and is only available when the integration knowledge index exists. <p align="center"> <img width="405" src="https://github.com/user-attachments/assets/640d4f54-34cc-47e3-b731-b3913139e84e" /> <img width="395" src="https://github.com/user-attachments/assets/fd66c044-5536-4947-98d8-45e4b168b34c" /> </p> ## Changes * Added `platform.core.integration_knowledge` builtin tool to `agent_builder_platform` that searches Fleet integration documentation * Tool is registered in plugin `setup()` with conditional availability using the `availability` configuration pattern * Availability is checked at runtime via ES search on `.integration_knowledge` index (using `size: 0` query) * Returns structured resource results with package name, version, filename, and content ## Technical Details * Tool registration added to `registerTools()` in plugin `setup()` phase, following the same pattern as `productDocumentationTool` * Uses `availability` configuration with `cacheMode: 'space'` to conditionally show/hide the tool based on index availability * Searches using Elasticsearch semantic search on the `content` field * `esClient.asInternalUser` is used for both handler execution and availability checking (index permissions require internal user) * Results include reference URLs to integration detail pages (`/app/integrations/detail/{package_name}`) ## Considerations * Tool requires Fleet to have indexed integration knowledge into `.integration_knowledge` * Tool availability is checked per-space and cached for performance * No Kibana restart required - tool appears/disappears dynamically based on index availability * This is the onechat/Agent Builder equivalent of the existing `IntegrationKnowledgeTool` in Security Solution's Assistant (elastic#236197) and Observability Solution's Assistant (elastic#237085) added in `9.2`. --- ## Testing > [!NOTE] > You must enable the `xpack.fleet.enableExperimental: ["installIntegrationsKnowledge"]` feature flag until this PR enabling it by default is merged (elastic#245080). 1. Upload this sample [system-2.3.3-NEXT.zip](https://github.com/user-attachments/files/22546766/system-2.3.3-NEXT.zip) package via Integrations > Create new integration - The test package just copies the existing `docs/README.md` to `docs/knowledge_base/README.md` so that Fleet ingests it into `.integrations_knowledge` 2. Create new Agent with the new Integration Knowledge tool and ask questions related to system integrations, such as: - How can I collect CPU and memory data for my windows host? - What OS can I run the system integration on? - What does the system integration do? 3. Observe that the responses returned contain relevant information that is cited from the system integration. _PR developed with Cursor + Opus 4.5_ --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Initial pass at creating integrations knowledge tool

14468c7

jen-huang self-assigned this Sep 23, 2025

jen-huang added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:Fleet Team label for Observability Data Collection Fleet team labels Sep 23, 2025

jen-huang commented Sep 23, 2025

View reviewed changes

jen-huang added 2 commits September 25, 2025 10:52

Merge remote-tracking branch 'upstream/main' into feat/security-ai-in…

7e5203b

…tegrations-kb-tool

Fix citation kibana app url

1a3192c

jen-huang marked this pull request as ready for review September 25, 2025 20:22

jen-huang requested review from a team as code owners September 25, 2025 20:22

jen-huang changed the title ~~[WIP][UII] Create integrations knowledge tool for security assistant~~ [UII] Create integrations knowledge tool for security assistant Sep 25, 2025

jen-huang and others added 2 commits September 26, 2025 11:00

Merge branch 'main' into feat/security-ai-integrations-kb-tool

ab37467

Merge branch 'main' into feat/security-ai-integrations-kb-tool

9c57991

spong reviewed Sep 29, 2025

View reviewed changes

Comment thread ...security_solution/server/assistant/tools/integration_knowledge/integration_knowledge_tool.ts

spong added the ci:security-genai-run-evals label Sep 29, 2025

spong reviewed Sep 29, 2025

View reviewed changes

Comment thread ...security_solution/server/assistant/tools/integration_knowledge/integration_knowledge_tool.ts

jen-huang added 3 commits September 29, 2025 13:54

Add index exists check

6f3a9f8

Set up index during eval tests

7003626

Merge remote-tracking branch 'upstream/main' into feat/security-ai-in…

329cbb8

…tegrations-kb-tool

jen-huang requested a review from spong September 29, 2025 23:58

Merge branch 'main' into feat/security-ai-integrations-kb-tool

dc4dcdb

spong reviewed Sep 30, 2025

View reviewed changes

Comment thread ...security_solution/server/assistant/tools/integration_knowledge/integration_knowledge_tool.ts Outdated

jen-huang added 3 commits September 30, 2025 11:00

Fix index exists query

ccd7f01

Remove no hits message

1796db9

Merge remote-tracking branch 'upstream/main' into feat/security-ai-in…

3c24681

…tegrations-kb-tool

jen-huang requested a review from spong September 30, 2025 18:34

spong approved these changes Sep 30, 2025

View reviewed changes

jen-huang enabled auto-merge (squash) September 30, 2025 19:59

jen-huang mentioned this pull request Sep 30, 2025

[Security AI Prompt] Update prompts for new integrations knowledge tool elastic/integrations#15502

Merged

5 tasks

jen-huang merged commit fdb5f40 into elastic:main Sep 30, 2025
12 checks passed

kibanamachine added the v9.2.0 label Sep 30, 2025

jen-huang deleted the feat/security-ai-integrations-kb-tool branch September 30, 2025 20:32

jen-huang mentioned this pull request Oct 3, 2025

[Obs AI Assistant] aware of new .integration_knowledge* system index #237085

Merged

10 tasks

spong mentioned this pull request Dec 4, 2025

[Agent Builder] Adds Integration Knowledge platform tool #245259

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UII] Create integrations knowledge tool for security assistant#236197

[UII] Create integrations knowledge tool for security assistant#236197
jen-huang merged 12 commits intoelastic:mainfrom
jen-huang:feat/security-ai-integrations-kb-tool

jen-huang commented Sep 23, 2025 •

edited

Loading

Uh oh!

jen-huang Sep 23, 2025 •

edited

Loading

Uh oh!

elasticmachine commented Sep 25, 2025

Uh oh!

Uh oh!

spong Sep 29, 2025

Uh oh!

jen-huang Sep 29, 2025

Uh oh!

spong Sep 30, 2025

Uh oh!

spong Sep 29, 2025

Uh oh!

jen-huang Sep 29, 2025

Uh oh!

Uh oh!

Uh oh!

spong left a comment

Uh oh!

kibanamachine commented Sep 30, 2025

Uh oh!

Uh oh!

elasticmachine commented Sep 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jen-huang commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

To-do

Checklist

Uh oh!

jen-huang Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 25, 2025

Uh oh!

Uh oh!

spong Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

jen-huang Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

spong Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

spong Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

jen-huang Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

spong left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Sep 30, 2025

🤖 Prompt Changes Detected

Next Steps:

Uh oh!

Uh oh!

elasticmachine commented Sep 30, 2025

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jen-huang commented Sep 23, 2025 •

edited

Loading

jen-huang Sep 23, 2025 •

edited

Loading