[Security Solution][Elastic AI Assistant] Adds Model Evaluation Tooling

[spong]

Summary

This PR introduces a new internal/elastic_assistant/evaluate route and Evaluation Advanced Setting within the Assistant for benchmarking and testing models, agents, and other aspects of the Assistant configuration.

Enable via the assistantModelEvaluation experimental feature in your kibana.dev.yml (and better add discoverInTimeline for good measure as well! :)

xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'discoverInTimeline']

Then access from within the Advanced Settings modal in the Assistant. To use, first select your Connectors/Models, then corresponding Agent configurations, then what model you would like to use for final evaluation, the evaluation type, and if custom, you can specify the evaluation prompt that is sent off to the evaluator model. Finally, specify the dataset, and output index that the results should be written to, then click Perform evaluation.

Sample datasets can be found in x-pack/plugins/elastic_assistant/server/lib/model_evaluator/datasets, and include:

• esql_dataset.json
• query_dataset.json
• security_labs.json
• security_questions_dataset.json

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

[andrew-goldstein]

this is a placeholder per an offline discussion

[spong]

++, this is for utilizing the yarn evaluate-models CLI tooling. Ended up going the UI route first for flexibility/ease of use, but this is where we'll plumb through the CLI/test tooling.

[andrew-goldstein]

Thanks @spong for providing this capability to test at scale 🙏
✅ Desk tested locally
LGTM 🚀

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 8429801

Failed CI Steps

• FTR Configs #5

Test Failures

• [job] [logs] FTR Configs #5 / serverless observability UI navigation navigate observability sidenav & breadcrumbs

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	4557	4560	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	12.8MB	12.8MB	+53.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	62.7KB	62.7KB	+28.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
elasticAssistant	10	13	+3

Total ESLint disabled count

id	before	after	diff
elasticAssistant	10	13	+3

History

• 💔 Build #163801 failed 8657581
• 💔 Build #163768 failed 62c8cf0
• 💔 Build #163761 failed 84c20cc
• 💔 Build #163714 failed e4ece02
• 💛 Build #163639 was flaky d900c7b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong