[Spike][Security Solution] AI SOC: Default Agent skills, tools, and workflow playbooks#259559
Draft
patrykkopycinski wants to merge 9 commits into
Draft
[Spike][Security Solution] AI SOC: Default Agent skills, tools, and workflow playbooks#259559patrykkopycinski wants to merge 9 commits into
patrykkopycinski wants to merge 9 commits into
Conversation
Contributor
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
Contributor
Author
|
/ci |
patrykkopycinski
changed the title
…aybooks Implements the Elastic AI SOC feature as a spike to validate the end-to-end value chain of Agent Builder agents orchestrated through One Workflow playbooks for automated security operations. All new registrations are gated behind the `aiSocAgents` experimental feature flag. **7 new Agent Builder tools:** - `response_actions` — endpoint isolation/release/kill/suspend via Response Actions API - `mitre_mapping` — LLM-powered MITRE ATT&CK technique mapping with confidence scoring - `threat_intel_enrich` — IOC enrichment against TI indicator indices - `timeline_create` — investigation timeline creation via Timeline API service - `report_generate` — structured incident report generation (markdown/JSON) - `case_manage` — full Cases API integration (create/update/comment/attach/status) - `entity_store_query` — Entity Store v2 unified entity profile queries **5 new Agent Builder skills:** - Alert Triage, Investigation, MITRE Coverage Analysis, Incident Reporting, Response Recommendation — each with comprehensive methodology guides **6 new Agent Builder agents:** - Triage, Investigator, Correlator, Responder, Reporter, MITRE Analyst — each with focused tool assignments and specialized system prompts **4 pre-built One Workflow playbooks (YAML):** - Incident Response (alert-triggered: triage → investigate → respond → report) - Full Investigation (manual: investigate → correlate → MITRE map → report) - Proactive Threat Hunt (weekly scheduled: hunt → correlate → create rules) - Detection Coverage Audit (monthly scheduled: audit → generate rules) - All playbooks use structured output schemas on ai.agent steps for deterministic condition routing instead of fragile string matching **Testing:** - 98 unit tests across 6 tool test files - 6 @kbn/evals agent evaluation suites (30 test cases total) - Integration test suite for tool handlers - Workflow trigger definition for `security.alertCreated` events Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The tool was rewritten to use `createTimeline()` + `savePinnedEvents()` from the Timeline API service, but the test file still mocked raw saved objects. Updated tests to mock the Timeline service functions and verify the correct API calls are made. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
patrykkopycinski
force-pushed
the
elastic-ai-soc-plan
branch
from
143cff4 to
2d05593
Compare
patrykkopycinski
changed the title
patrykkopycinski
changed the title
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Elastic promotes the Default Elastic Agent approach — no custom agents. All SOC capabilities now live as enhanced skills that activate contextually within the default agent based on message content. Changes: - Removed 6 custom agent ID constants from common/constants.ts - Reverted agents/index.ts to only register the Threat Hunting Agent - Enhanced 5 skills with full tool assignments (getRegistryTools) from agents - Added `experimental: true` to all SOC skills - Merged agent system prompt content (escalation, case creation, rollback procedures) into skill content - Updated workflow playbooks to use default agent (removed agent-id configs) - Playbook messages now explicitly reference skill names for activation Note: The 6 emptied agent files still need to be deleted manually. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
patrykkopycinski
changed the title
Completes the migration to the Default Elastic Agent + Skills pattern by converting the last custom agent (Threat Hunting) to a skill. Changes: - Created threat_hunting skill with all platform + security tools - Removed registerAgents() call from plugin.ts (no more agent registrations) - Updated public hooks (use_agent_builder_attachment, use_agent_builder_stream) to use the default agent instead of THREAT_HUNTING_AGENT_ID - Removed THREAT_HUNTING_AGENT_ID constant and internalNamespaces import - Removed last agent-id reference from workflow playbooks - agents/index.ts now contains only a migration comment Note: threat_hunting_agent.ts file needs manual deletion (hook blocks it) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- entity_store_query: Use space-specific index pattern in availability check - entity_store_query: Pass time_range parameter to query functions - response_actions: Fix PID 0 validation (use === undefined instead of falsy) - mitre_mapping: Handle content block arrays from LLM responses - integration test: Add missing ES search mock for attach_alerts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/ci |
The attach_alerts action now queries ES for alert rule metadata before attaching. Tests were missing the esClient.asCurrentUser.search mock, causing them to fail on .hits.hits access. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/ci |
Comment on lines
+122
to
+124
| const providers = ( | ||
| result.results[0].data.matches as Array<Record<string, unknown>> | ||
| ).map((m) => m.provider); |
Contributor
There was a problem hiding this comment.
🟢 Low __integration__/soc_tools.integration.test.ts:122
The assertion on line 124 reads m.provider but the match objects are spread from Elasticsearch _source where provider is nested at threat.indicator.provider. Since the tool spreads _source directly into matches, m.provider returns undefined for every item, causing the toContain assertions to incorrectly fail even when the data is present.
- const providers = (
- result.results[0].data.matches as Array<Record<string, unknown>>
- ).map((m) => m.provider);
+ const providers = (
+ result.results[0].data.matches as Array<Record<string, unknown>>
+ ).map((m) => (m.threat as Record<string, unknown>)?.indicator?.provider);🤖 Copy this AI Prompt to have your agent fix this:
In file x-pack/solutions/security/plugins/security_solution/server/agent_builder/tools/__integration__/soc_tools.integration.test.ts around lines 122-124:
The assertion on line 124 reads `m.provider` but the match objects are spread from Elasticsearch `_source` where `provider` is nested at `threat.indicator.provider`. Since the tool spreads `_source` directly into matches, `m.provider` returns `undefined` for every item, causing the `toContain` assertions to incorrectly fail even when the data is present.
Evidence trail:
x-pack/solutions/security/plugins/security_solution/server/agent_builder/tools/threat_intel_enrich_tool.ts lines 138-144 (matches spread `_source` directly)
x-pack/solutions/security/plugins/security_solution/server/agent_builder/tools/__integration__/soc_tools.integration.test.ts lines 50-95 (mock _source has provider at threat.indicator.provider)
x-pack/solutions/security/plugins/security_solution/server/agent_builder/tools/__integration__/soc_tools.integration.test.ts lines 122-127 (test asserts on m.provider which is undefined)
Contributor
Author
|
/ci |
Comment on lines
+59
to
+143
| async converse({ messages, conversationId, agentId }: ConverseParams): Promise<ConverseResponse> { | ||
| const callConverseApi = async (): Promise<ConverseResponse> => { | ||
| const response = await this.fetch('/api/agent_builder/converse', { | ||
| method: 'POST', | ||
| version: '2023-10-31', | ||
| body: JSON.stringify({ | ||
| agent_id: agentId, | ||
| connector_id: this.connectorId, | ||
| conversation_id: conversationId, | ||
| input: messages[messages.length - 1].message, | ||
| }), | ||
| }); | ||
|
|
||
| const chatResponse = response as { | ||
| conversation_id: string; | ||
| trace_id?: string; | ||
| steps: Step[]; | ||
| response: { message: string }; | ||
| model_usage?: ModelUsageStats; | ||
| }; | ||
|
|
||
| const { | ||
| conversation_id: conversationIdFromResponse, | ||
| response: latestResponse, | ||
| steps, | ||
| trace_id: traceId, | ||
| model_usage: modelUsage, | ||
| } = chatResponse; | ||
|
|
||
| return { | ||
| conversationId: conversationIdFromResponse, | ||
| messages: [...messages, latestResponse], | ||
| steps, | ||
| traceId, | ||
| modelUsage, | ||
| errors: [], | ||
| }; | ||
| }; | ||
|
|
||
| try { | ||
| return await pRetry(callConverseApi, { | ||
| retries: RETRIES, | ||
| minTimeout: MIN_TIMEOUT, | ||
| onFailedAttempt: (error) => { | ||
| const isLastAttempt = error.retriesLeft === 0; | ||
|
|
||
| if (isLastAttempt) { | ||
| this.log.error( | ||
| new Error(`Failed to call converse API after ${error.attemptNumber} attempts`, { | ||
| cause: error, | ||
| }) | ||
| ); | ||
| } else { | ||
| this.log.warning( | ||
| new Error(`Converse API call failed on attempt ${error.attemptNumber}; retrying...`, { | ||
| cause: error, | ||
| }) | ||
| ); | ||
| } | ||
| }, | ||
| }); | ||
| } catch (error) { | ||
| this.log.error('Error occurred while calling converse API'); | ||
| return { | ||
| conversationId, | ||
| steps: [], | ||
| messages: [ | ||
| ...messages, | ||
| { | ||
| message: | ||
| 'This question could not be answered as an internal error occurred. Please try again.', | ||
| }, | ||
| ], | ||
| errors: [ | ||
| { | ||
| error: { | ||
| message: error instanceof Error ? error.message : 'Unknown error', | ||
| stack: error instanceof Error ? error.stack : undefined, | ||
| }, | ||
| type: 'error', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🟢 Low evals/chat_client.ts:59
Accessing messages[messages.length - 1].message throws TypeError: Cannot read property 'message' of undefined when messages is an empty array. The Messages type permits empty arrays, and there's no validation before accessing the last element. Consider adding a check for empty messages before the API call, or document the precondition if callers are expected to always provide at least one message.
async converse({ messages, conversationId, agentId }: ConverseParams): Promise<ConverseResponse> {
+ if (messages.length === 0) {
+ throw new Error('messages array must not be empty');
+ }
+
const callConverseApi = async (): Promise<ConverseResponse> => {🤖 Copy this AI Prompt to have your agent fix this:
In file x-pack/solutions/security/plugins/security_solution/server/agent_builder/evals/chat_client.ts around lines 59-143:
Accessing `messages[messages.length - 1].message` throws `TypeError: Cannot read property 'message' of undefined` when `messages` is an empty array. The `Messages` type permits empty arrays, and there's no validation before accessing the last element. Consider adding a check for empty `messages` before the API call, or document the precondition if callers are expected to always provide at least one message.
Evidence trail:
x-pack/solutions/security/plugins/security_solution/server/agent_builder/evals/chat_client.ts at REVIEWED_COMMIT:
- Line 16: `export type Messages = { message: string }[];` (permits empty arrays)
- Line 56: `async converse({ messages, conversationId, agentId }: ConverseParams)` (no validation)
- Line 65: `input: messages[messages.length - 1].message` (unsafe access)
Contributor
💔 Build Failed
Failed CI Steps
Metrics [docs]
History
|
Closed
20 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spike implementing the Elastic AI SOC feature using the Default Elastic Agent + Skills pattern — zero custom agents. All SOC capabilities are registered as enhanced Agent Builder skills that activate contextually within the default agent, orchestrated through One Workflow playbooks.
All new registrations are gated behind the
aiSocAgentsexperimental feature flag +experimental: trueon each skill.Architecture: Default Agent + Skills (No Custom Agents)
This PR also migrates the existing Threat Hunting Agent to a skill, completing the transition to zero custom agents in Security Solution. The
registerAgents()call is removed from plugin.ts entirely.Why Skills Instead of Custom Agents
agent-id: security.triageexperimental: trueon skill + feature flagNew Components
7 Agent Builder tools:
response_actionsEndpointAppContextService.getInternalResponseActionsClient()mitre_mappingthreat_intel_enrichlogs-ti_*indicestimeline_createcreateTimeline()+savePinnedEvents()Timeline APIreport_generatecase_managecases.getCasesClientWithRequest()plugin contractentity_store_query.entities.v1.latest.*indices6 Agent Builder skills (all with
experimental: trueexcept threat-hunting):4 One Workflow playbooks (YAML with structured output schemas):
Workflow trigger:
security.alertCreatedregistered withworkflowsExtensionsExisting Code Changes (Threat Hunting Agent Migration)
use_agent_builder_attachment.ts— removedagentId: THREAT_HUNTING_AGENT_ID(uses default agent)use_agent_builder_stream.ts— removedagent_id: THREAT_HUNTING_AGENT_ID(uses default agent)common/constants.ts— removedTHREAT_HUNTING_AGENT_IDconstant and unusedinternalNamespacesimportserver/plugin.ts— removedregisterAgents()call entirelyagents/index.ts— replaced with migration comment (file to be deleted)Key Design Decisions
ai.agentworkflow steps for deterministic condition routingdata.mapsteps between workflow steps for structured field extractionEndpointAppContextService, Timelines via Timeline API, Cases via plugin contractproductDocumentation) in skillsTesting
@kbn/evalssuitesHandoff Items
security.alertCreatedtriggergetInternalResponseActionsClientusagecreateTimeline+savePinnedEventspatternthreat_hunting_agent.ts+ 6 empty SOC agent files need manual deletionTest plan
xpack.securitySolution.enableExperimental: ['aiSocAgents']yarn test:jeston new test files@kbn/evalssuites against configured LLM connector🤖 Generated with Claude Code
Production-Readiness Checklist — Agent Skills Ecosystem
Generated against [Epic] Creation of the Agent Skills Ecosystem for Elastic Security.
Narrative role: Flagship end-to-end proof of the epic's "Default Elastic Agent + Skills (no custom agents)" architecture. The single PR most directly executing on the vision's positioning statement.
Must-do before this can ship
registerAgents()is architecturally correct but must be aligned with Observability (#255706) — land a cross-solution Decision Record firstresponse_actions,case_manage(full lifecycle),timeline_create,threat_intel_enrich(if external calls). Each needsrequiredPrivilegesgating + an explicit user confirmation pathentity_store_queryshould become a standalone Entity Analytics skill primitive (vision's "horizontal enrichment layer") — design the shared contract before baking it into this PR's toolaiSocAgents+ per-skillexperimental: true(already in scope — verify end-to-end)Follow-ups (post-merge)