[Security Solution] Security AI Assistant persistent storage.

[YulNaumenko]

Summary

This PR including both new APIs and client side changes to use data stream for Security Solution AI Assistant conversations persistence storage.
Issue https://github.com/elastic/security-team/issues/7810

Extended description

elastic-assistant plugin

All API changes are introduced in elastic-assistant plugin server

• RequestContextFactory - this class helps to provide the needed context for each API request for routes handler context.

• AIAssistantService - This service instance is created on the plugin setup and included to the request context factory. It is responsible for the needed conversations storage resources initialization and installation. It uses DataStreamAdapter from packages/kbn-data-stream-adapter. Conversations fieldMap definition here

• AIAssistantConversationsDataClient - data client which has a set of methods to interact with conversation storage on behalf of the current user and space.

• ConversationDataWriter - is a helper class which implements a bulk method to interact with esClient

• Added new routes using versioned router and openAPI code generator schemas:
  createConversationRoute
readConversationRoute
updateConversationRoute
deleteConversationRoute
appendMessagesRoute
findUserConversationsRoute
bulkActionsRoute

• Migrated existing knowledge_base, evaluate and post_actions_connector_execute routes to versioned routing and openAPI code generator schemas.

kbn-elastic-assistant package

• removed local storage persistency logic for assistantConversations.
• added API requests definition to communicate to server side.

kbn-elastic-assistant-common package

• Changed transformsRowData function to use async add replacements API.
• Exposed routing URLs with constants file to be available for server and client.

security_solution plugin

• Added migrateConversationsFromLocalStorage for existing conversations in the local storage. This migration happening only for the first time when user doesn't have any conversations persisted in the current space. After mirgation complete, the old local storage key securitySolution.assistantConversation will be removed.
• Passing security related baseConversation as a property to ElasticAssistantProvider
• Changed useAssistantTelemetry to fetch information about the conversation from the conversations API
• Modified useConversationStore to fetch the data from the conversations API /api/elastic_assistant/conversations/current_user/_find and merge with security predefined baseConversations if they are not used(persisted) yet.
• Extracted AssistantTab to a separate lazy loaded file to avoid unnecessary rendering/requests till this tab will be shown in Timeline.

[xcrzx]

The Rules Area changes are pretty straightforward, LGTM 👍

[andrew-goldstein]

Thanks @YulNaumenko for realizing persistence of conversations, and for all the foundational APIs and abstractions created in this PR! 🙏
✅ Desk tested conversation migration, ES|QL query generation, RAG on alerts use cases, and local context (all with anonymization)
LGTM 🚀

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: c9926e7
• Cloud Deployment

Failed CI Steps

• Jest Integration Tests #7

Test Failures

• [job] [logs] Jest Integration Tests #7 / getOutdatedDocumentsQuery creates a query returning the expected documents
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #bulkGet returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #bulkResolve returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #get returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #resolve returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Task priority checks detects tasks with priority definitions
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: false should fail the task run when setting allow_reading_invalid_state:false and reading an invalid state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should debug log by default when reading an invalid task state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should drop unknown fields from the task state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should fail to update the task if the task runner returns an unknown property in the state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should migrate the task state
• [job] [logs] Jest Integration Tests #7 / ZDT upgrades - introducing a new SO type should support adding the bar type
• [job] [logs] Jest Integration Tests #7 / ZDT upgrades - switching from v2 algorithm when switching from a compatible version it able to re-use a cluster state from the v2 algorithm
• [job] [logs] Jest Integration Tests #7 / ZDT upgrades - switching from v2 algorithm when switching from an incompatible version fails and throws an explicit error
• [job] [logs] Jest Integration Tests #7 / ZDT with v2 compat - recovering from partially migrated state migrates the documents

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5019	5030	+11

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	84	132	+48
@kbn/elastic-assistant-common	57	207	+150
elasticAssistant	27	31	+4
total			+202

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	12.8MB	15.5MB	⚠️ +2.7MB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	6	8	+2
stackConnectors	1	0	-1
total			+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	72.0KB	72.0KB	-9.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant	104	154	+50
@kbn/elastic-assistant-common	59	220	+161
elasticAssistant	41	45	+4
total			+215

ESLint disabled line counts

id	before	after	diff
@kbn/elastic-assistant	19	20	+1
elasticAssistant	13	29	+16
securitySolution	482	485	+3
total			+20

Total ESLint disabled count

id	before	after	diff
@kbn/elastic-assistant	20	21	+1
elasticAssistant	13	29	+16
securitySolution	557	560	+3
total			+20

History

• 💔 Build #197564 failed 0c3b0b7
• 💔 Build #197535 failed ee7e269
• 💔 Build #197497 failed d2bcda5
• 💛 Build #197467 was flaky d25950e
• 💔 Build #197335 failed acbf3b7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @YulNaumenko

[stephmilovic]

I'm wondering if we should have not deleted this? @andrew-goldstein

I see an error in our telemetry logs like:

Failed to parse. Text: "```json { "action": "Final Answer", "action_input": "Here are a few more example ES|QL queries for different data exfiltration scenarios:\n\n1. Detect large volumes of network traffic by user and process:\n```\nFROM logs-*\n| WHERE event.category == \"network\" AND agent.type == \"windows\"\n| STATS bytes = sum(network.bytes) BY process.name, user.name, source.ip, destination.ip\n| EVAL mb = bytes / (1024 * 1024)\n| WHERE mb > <network_threshold_value>\n| SORT mb DESC\n| KEEP process.name, user.name, source.ip, destination.ip, mb\n```\n2. Detect unusual file creation activity:\n```\nFROM logs-*\n| WHERE event.category == \"file\" AND event.action == \"create\" AND agent.type == \"windows\"\n| STATS count() BY process.name, user.name, host.name, file.extension\n| WHERE count() > <file_creation_threshold_value>\n| SORT count() DESC\n| KEEP process.name, user.name, host.name, file.extension, count()\n```\n3. Detect suspicious email attachments:\n```\nFROM logs-*\n| WHERE event.category == \"email\" AND event.action == \"send\" AND agent.type == \"windows\"\n| WHERE file.extension IN (\\".exe\", \\".bat\", \\".com\", \\".dll\", \\".docm\", \\".jar\", \\".js\", \\".jse\", \\".lnk\", \\".msi\", \\".ps\", \\".ps1\", \\".scr\", \\".vbs\", \\".wsf\")\n| STATS count() BY process.name, user.name, source.email, destination.email, file.name\n| WHERE count() > <email_attachment_threshold_value>\n| SORT count() DESC\n| KEEP process.name, user.name, source.email, destination.email, file.name, count()\n```\nRemember to replace `<network_threshold_value>`, `<file_creation_threshold_value>`, and `<email_attachment_threshold_value>` with values appropriate for your environment. Adjust the queries as needed based on your specific logging setup and fields." } ```". Error: SyntaxError: Expected ',' or '}' after property value in JSON at position 1101

Could this be coming from us?

[jamesspi]

@stephmilovic , that was from my cluster.

[stephmilovic]

but it is an 8.13 cluster, before this code was deleted.