[Agent Builder] [AI Infra] Adds product documentation tool and task evals by spong · Pull Request #248370 · elastic/kibana

spong · 2026-01-09T00:32:27Z

Note

Need to iterate on actual baseline evals (they're pretty much the same now), but wanted to check in and get working on CI since we're adding a new package here. Will tune baseline evals for each so that they're somewhat useful, but the intent here is to get something in place to make further feedback cycles quicker/easier.

Summary

This PR adds two complementary evaluation specs for the Product Documentation experience:

Agent Builder tool-behavior evals

Verifies the agent calls only the platform.core.product_documentation tool and follows grounding/insufficiency rules.
File: x-pack/platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/evals/product_documentation/product_documentation.spec.ts

AI Infra retriever-task evals (llm_tasks)

Evaluates the llmTasks.retrieveDocumentation task itself (retriever + token reduction)
File: x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/evals/retrieve_documentation/retrieve_documentation.spec.ts

Key implementation details

New eval suite package for ai-infra tasks: @kbn/evals-suite-llm-tasks
- Path: x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/
New product_documentation eval spec in existing agent-builder/kbn-evals-suite-agent-builder suite

Test Instructions

Start Scout server in another terminal and keep it running:

scripts/scout.js start-server --stateful

Start phoenix in another terminal and keep it running:

node scripts/phoenix

Then run desired suite

Agent Builder: product documentation tool eval

EVALUATION_REPETITIONS=1 \
EVALUATION_CONNECTOR_ID="gemini-3-pro" \
node scripts/playwright test \
--config x-pack/platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/playwright.config.ts \
x-pack/platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/evals/product_documentation/product_documentation.spec.ts \
--project gemini-3-pro

ai-infra: llm_tasks retrieveDocumentation retriever-task eval

EVALUATION_REPETITIONS=1 \
EVALUATION_CONNECTOR_ID="gemini-3-pro" \
node scripts/playwright test \
--config x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/playwright.config.ts \
x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/evals/retrieve_documentation/retrieve_documentation.spec.ts \
--project gemini-3-pro

Note

Replace --project gemini-3-pro with the connector id you want to run against, and EVALUATION_CONNECTOR_ID with the judge connector id.

PR developed with Cursor + GPT 5.2

joemcelroy · 2026-01-09T09:49:41Z

x-pack/platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/src/evaluate.ts

+    async ({ uiSettings, log }, use) => {
+      // Ensure AgentBuilder API is enabled before running the evaluation.
+      // Using Scout's uiSettings fixture is more robust than calling /internal/kibana/settings directly.
+      await uiSettings.set({ ['agentBuilder:enabled']: true });


FYI this feature flag is being removed soon #248050

joemcelroy

LGTM - will be interesting when skills come in and i wonder if the product_documentation.spec will change to be more evaluating the skill vs evaluating an agent with a single tool.

...-infra/kbn-evals-suite-llm-tasks/evals/retrieve_documentation/retrieve_documentation.spec.ts

qn895

LGTM 🎉

...platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/src/evaluate_dataset.ts

elasticmachine · 2026-01-09T20:38:37Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 41df07c

Failed CI Steps

Scout: [ platform / streams_app ] plugin

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/evals`	134	160	+26

Unknown metric groups

API count

id	before	after	diff
`@kbn/evals`	153	179	+26

History

cc @spong

kibanamachine · 2026-01-09T21:54:39Z

Starting backport for target branches: 9.3

https://github.com/elastic/kibana/actions/runs/20866660846

kibanamachine · 2026-01-09T22:02:17Z

💔 All backports failed

Status	Branch	Result
❌	9.3	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 248370

Questions ?

Please refer to the Backport tool documentation

spong · 2026-01-09T22:49:10Z

Will just keep to main/9.4 as the 9.3 backport has a buncha conflicts with the OneChat->AB rename. We won't be running these directly in 9.3, so fine to just have them from here forward.

…vals (elastic#248370) > [!NOTE] > Need to iterate on actual baseline evals (they're pretty much the same now), but wanted to check in and get working on CI since we're adding a new package here. Will tune baseline evals for each so that they're somewhat useful, but the intent here is to get something in place to make further feedback cycles quicker/easier. ## Summary This PR adds two complementary evaluation specs for the Product Documentation experience: ##### Agent Builder tool-behavior evals * Verifies the agent calls only the `platform.core.product_documentation` tool and follows grounding/insufficiency rules. * File: `x-pack/platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/evals/product_documentation/product_documentation.spec.ts` ##### AI Infra retriever-task evals (llm_tasks) * Evaluates the `llmTasks.retrieveDocumentation` task itself (retriever + token reduction) * File: `x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/evals/retrieve_documentation/retrieve_documentation.spec.ts` #### Key implementation details * New eval suite package for ai-infra tasks: `@kbn/evals-suite-llm-tasks` * Path: `x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/` * New `product_documentation` eval spec in existing `agent-builder/kbn-evals-suite-agent-builder` suite #### Test Instructions Start Scout server in another terminal and keep it running: ``` scripts/scout.js start-server --stateful ``` Start phoenix in another terminal and keep it running: ``` node scripts/phoenix ``` Then run desired suite 1) Agent Builder: product documentation tool eval ``` EVALUATION_REPETITIONS=1 \ EVALUATION_CONNECTOR_ID="gemini-3-pro" \ node scripts/playwright test \ --config x-pack/platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/playwright.config.ts \ x-pack/platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/evals/product_documentation/product_documentation.spec.ts \ --project gemini-3-pro ``` <img width="2293" height="958" alt="image" src="https://github.com/user-attachments/assets/d372526e-bdca-4847-b4e2-b0343b7ab390" /> 2) ai-infra: llm_tasks retrieveDocumentation retriever-task eval ``` EVALUATION_REPETITIONS=1 \ EVALUATION_CONNECTOR_ID="gemini-3-pro" \ node scripts/playwright test \ --config x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/playwright.config.ts \ x-pack/platform/packages/shared/ai-infra/kbn-evals-suite-llm-tasks/evals/retrieve_documentation/retrieve_documentation.spec.ts \ --project gemini-3-pro ``` <img width="1146" height="396" alt="image" src="https://github.com/user-attachments/assets/345aa83b-fbf7-407d-87bc-18d2c64879e6" /> > [!NOTE] > Replace `--project gemini-3-pro` with the connector id you want to run against, and EVALUATION_CONNECTOR_ID with the judge connector id. _PR developed with Cursor + GPT 5.2_ --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

* commit 'c4304e27736c62f17af20d145770b2ae9d3fae30': (418 commits) skip failing suite (elastic#89079) [ES|QL] Update grammars (elastic#248600) skip failing test suite (elastic#248579) [ES|QL] Update function metadata (elastic#248601) skip failing test suite (elastic#248554) Fix flaky test runner serverless flag for Search solution (elastic#248559) [Security Solution][Attacks/Alerts][Attacks page][Table section] Remember last selected attack details tab (Summary or Alerts) (elastic#247519) (elastic#247988) Fix ES health check poller (elastic#248496) Fix collector schema ownership (elastic#241292) [api-docs] 2026-01-10 Daily api_docs build (elastic#248574) Update dependency cssstyle to v5.3.5 (main) (elastic#237637) Update dependency @octokit/rest to v22.0.1 (main) (elastic#243102) skip failing test suite (elastic#248504) skip failing test suite (elastic#247685) Remove broken ecommerce_dashboard journeys (elastic#248162) [Obs AI] Hide AI Insight component when there are no connectors (elastic#248542) skip failing suite (elastic#248433) [Security Solution][Attacks/Alerts][Attacks page][Table section] Hide tabs for generic attack groups (elastic#248444) [Agent Builder] [AI Infra] Adds product documentation tool and task evals (elastic#248370) [Controls Anywhere] Keep controls focused when creating + editing other panels (elastic#248021) ...

spong added 2 commits January 8, 2026 17:12

Add product doc tool and task evals

18b7358

Merge branch 'main' of github.com:elastic/kibana into product-doc-evals

bbe00c8

spong requested a review from a team January 9, 2026 00:32

spong self-assigned this Jan 9, 2026

spong requested a review from a team as a code owner January 9, 2026 00:32

spong added release_note:skip Skip the PR/issue when compiling release notes backport:version Backport to applied version labels v9.3.0 v9.4.0 labels Jan 9, 2026

kibanamachine and others added 5 commits January 9, 2026 00:49

Changes from yarn openapi:bundle

07074dd

Changes from node scripts/lint_ts_projects --fix

65ebc55

Changes from node scripts/generate codeowners

59f67c6

Changes from node scripts/regenerate_moon_projects.js --update

043f7de

Merge branch 'main' into product-doc-evals

b96fcd7

joemcelroy reviewed Jan 9, 2026

View reviewed changes

joemcelroy approved these changes Jan 9, 2026

View reviewed changes

qn895 reviewed Jan 9, 2026

View reviewed changes

...-infra/kbn-evals-suite-llm-tasks/evals/retrieve_documentation/retrieve_documentation.spec.ts Outdated Show resolved Hide resolved

qn895 approved these changes Jan 9, 2026

View reviewed changes

abhi-elastic reviewed Jan 9, 2026

View reviewed changes

...platform/packages/shared/agent-builder/kbn-evals-suite-agent-builder/src/evaluate_dataset.ts Outdated Show resolved Hide resolved

abhi-elastic approved these changes Jan 9, 2026

View reviewed changes

spong and others added 3 commits January 9, 2026 09:22

Merge branch 'main' into product-doc-evals

108bb94

Feedback from review

893d1fa

Fix jest config rootDir

41df07c

spong merged commit 16e3505 into elastic:main Jan 9, 2026
15 checks passed

spong deleted the product-doc-evals branch January 9, 2026 21:54

spong removed the backport:version Backport to applied version labels label Jan 9, 2026

spong removed the v9.3.0 label Jan 9, 2026

kibanamachine added the backport:skip This PR does not require backporting label Jan 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Agent Builder] [AI Infra] Adds product documentation tool and task evals#248370

[Agent Builder] [AI Infra] Adds product documentation tool and task evals#248370
spong merged 10 commits intoelastic:mainfrom
spong:product-doc-evals

spong commented Jan 9, 2026 •

edited by kibanamachine

Loading

Uh oh!

joemcelroy Jan 9, 2026

Uh oh!

joemcelroy left a comment

Uh oh!

Uh oh!

qn895 left a comment

Uh oh!

Uh oh!

elasticmachine commented Jan 9, 2026

API count

Uh oh!

Uh oh!

kibanamachine commented Jan 9, 2026

Uh oh!

kibanamachine commented Jan 9, 2026

Uh oh!

spong commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

spong commented Jan 9, 2026 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Agent Builder tool-behavior evals

AI Infra retriever-task evals (llm_tasks)

Key implementation details

Test Instructions

Uh oh!

joemcelroy Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

joemcelroy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qn895 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Jan 9, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Public APIs missing comments

API count

History

Uh oh!

Uh oh!

kibanamachine commented Jan 9, 2026

Uh oh!

kibanamachine commented Jan 9, 2026

💔 All backports failed

Manual backport

Questions ?

Uh oh!

spong commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

spong commented Jan 9, 2026 •

edited by kibanamachine

Loading