[Obs AI Assistant] aware of new .integration_knowledge* system index

[arturoliduena]

Closes https://github.com/elastic/obs-ai-assistant-team/issues/357

Summary

This PR adds the awareness of .integration_knowledge* index as another index for recalling. The Obs AI Assistant will retrieve integration knowledge from the index

Value added to the Obs AI Assistant (https://github.com/elastic/obs-ai-assistant-team/issues/357#issuecomment-3303692842):

The assistant will become aware of LLM-facing documentation for any installed integrations in the user's cluster. For example, the Logstash integration might ship documentation that explains how to understand the health reporting metrics collected by the integration and the assistant could answer prompts like "why was my logstash server down yesterday?" using the user's real data.

Manual Testing:
1 -> Follow the instructions from #230107 (comment):

1. If [Fleet] Add integration_knowledge system index for Fleet elasticsearch#132506 has been merged, run yarn es snapshot in Kibana, otherwise, checkout that branch in your local ES and then in kibana run yarn es source in order to use that version of ES which contains the index management, mappings, etc.

2. Install a package with any number of knowledge base docs in the docs/knowledge_base folder. You can use this sample package, or create your own following the guide below:

   • Using elastic-package, create a new package using elastic-package create integration
     - Once created add knowledge_base as a folder inside of the generated docs folder of the integration
     - Add an arbitrary amount of .md files to the knowledge_base folder
     - Run elastic-package build to build the package
     - There are a lot of different options for installing the package in a local kibana instance. I prefer to just take the generated .zip folder from /build in elastic-package and upload it to kibana using the custom integrations feature. You can also expose the package registry, or whatever you see fit.

3. Watch the Kibana logs for errors/debug messages etc

4. Use the new endpoint or just directly check the index using GET /.integration_knowledge/_search to verify that the documents are ingested into the system index of .integration_knowledge

5. Update the package and verify that the KB documents are updated by checking the response again, they should have the updated pkgVersion on the associated docs.

6. Remove the package and then verify (using the endpoint) that the docs are removed from the index

2 - Ask the AI Assistant about information contained in the integration documents

3- check that the documents are listed on the response of executed the function context inside learnings

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[elasticmachine]

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[jen-huang]

while working on the security AI assistant piece for this work (#236197), I found that kibana_system does not have permission to call indices.exists(), nor does it have permission to call indices.get(). both calls would result in an error being thrown from ES

yes... it's weird, apparently read permissions for this index is apparently not sufficient to access these two "simple" index checks

we decided to implement an empty search as a workaround to check against index existence:

kibana/x-pack/solutions/security/plugins/security_solution/server/assistant/tools/integration_knowledge/integration_knowledge_tool.ts

Lines 50 to 57 in 9732c29

	// Check if the .integration_knowledge index exists before registering the tool
	// This has to be done with `.search` since `.exists` and `.get` can't be performed
	// with the internal system user (lack of permissions)
	try {
	const indexExists = await assistantContext.core.elasticsearch.client.asInternalUser.search({
	index: INTEGRATION_KNOWLEDGE_INDEX,
	size: 0,
	});

[arturoliduena]

@jen-huang thanks for pointing that out. I’ve removed the index existence check. The code now handles missing indexes in the try/catch block. If an error occurs (e.g., the index doesn’t exist), it will log the error and return an empty list of retrieved documents.

[2025-10-13T18:15:40.344+02:00][ERROR][plugins.observabilityAIAssistant.service.kb] Error recalling from .integration_knowledge index
[2025-10-13T18:15:40.345+02:00][DEBUG][plugins.observabilityAIAssistant.service.kb] ResponseError: index_not_found_exception
	Root causes:
		index_not_found_exception: no such index [.integration_knowledge]
    at KibanaTransport._request (/Users/arturoliduena/arturoliduena/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:591:17)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at /Users/arturoliduena/arturoliduena/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:697:22
    at KibanaTransport.request (/Users/arturoliduena/arturoliduena/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:694:14)
    

[neptunian]

This looks good. Verified documents are being recalled using Phoenix and logs. Can approve after making @jen-huang's change.

[arturoliduena]

This looks good. Verified documents are being recalled using Phoenix and logs. Can approve after making @jen-huang's change.

thanks @neptunian, I already applied @jen-huang 's suggestion. I removed esClient.asInternalUser.indices.exists in commit ecc74e3, Instead of having two search queries, I kept a single query that handles the case where the index doesn’t exist, it now simply returns an empty list of documents.
I added a log example in the above comment.

[elasticmachine]

⏳ Build in-progress

• Buildkite Build
• Commit: ecc74e3
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-237085-ecc74e329194

History

• 💛 Build #350357 was flaky 8f47c64
• 💔 Build #346240 failed 2e04a35
• 💔 Build #346198 failed c9914d0

[arturoliduena]

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation