[Security Solution] Migrate Threat Hunting Agent to modular Agent Builder skills

[patrykkopycinski]

Summary

Decomposes the monolithic security.agent (Threat Hunting Agent) into two focused, modular skills for the Elastic AI Agent. This improves discoverability, reduces token overhead via on-demand loading, and enables independent evolution of each security capability.

New Skills

Skill	Purpose	Registry Tools	Inline Tools
threat-hunting	Hypothesis-driven hunting, IOC search, anomaly detection, case creation	6 (generateEsql, executeEsql, search, listIndices, getIndexMapping, cases)	—
alert-analysis	Alert triage, entity correlation, disposition workflow	3 (security.alerts, security_labs_search, entity_risk_score)	1 (get-related-alerts)

Key Changes

• Created 2 SkillDefinition registrations with curated tool sets and rich skill content
• Added inline get-related-alerts tool using space-scoped detection engine alert search for entity-based alert correlation (with terms queries for multi-valued ECS fields)
• Enriched threat hunting agent instructions to skill-quality content
• Added platformCoreTools.cases to threat-hunting skill so confirmed findings can be escalated into investigation cases
• Added cross-skill reference: alert-analysis references entity-analytics for deeper entity profiling and asset criticality
• Added comprehensive unit tests for all skills, tools, and the inline handler
• Added eval suite with ExpectedSkillInvocation evaluator for dynamic skill activation validation

Security Hardening

• ES|QL injection prevention: alertIds in attack_discovery_search_tool are validated against SAFE_ID_PATTERN (/^[a-zA-Z0-9_\-.:]+$/) before interpolation into ES|QL queries. Invalid IDs are silently filtered; all-invalid returns an error result.
• spaceId validation: Validated before interpolation into ES|QL index patterns (defense-in-depth).

Architecture

• Skills use getRegistryTools() to reference shared platform/security tools
• Inline tools only access ToolHandlerContext (esClient, spaceId, request) — no plugin service dependencies
• Threat hunting agent conditionally hides agentId when skills are enabled (agent availability gated on AGENT_BUILDER_EXPERIMENTAL_FEATURES_SETTING_ID)
• Agent description and instructions softened for exception/tuning language to match actual tool capabilities

Eval Suite

The eval suite includes a metadata-driven ExpectedSkillInvocation evaluator that dynamically validates skill activation based on expectedSkill and shouldNotActivateSkill metadata on each example. It queries OTEL traces for filestore.read tool calls matching the skill's SKILL.md path, producing pass/fail scores per example. This covers:

• Threat hunting queries → threat-hunting skill activation
• Alert investigation queries → alert-analysis skill activation
• Negative cases ensuring non-security queries do not activate security skills

Run locally: node scripts/evals start --suite agent-builder
Trigger in CI: add evals:agent-builder label to PR

Test Coverage

Category	Count	Details
Skill definition tests	20	Metadata, tool counts, cases tool presence, content assertions, cross-skill references, referenced content
attack_discovery_search tool tests	12	Schema, ES
alert_analysis inline tool tests	11	Entity extraction (single/multi-type/multi-valued), space-scoped index, error paths
use_agent_builder_attachment hook tests	7	Conditional agentId, skills-enabled behavior, attachment callbacks
Eval suite examples	12	Skill activation for threat hunting, alert analysis queries; cross-skill routing; distractor queries
Total	62	Across 5 test files

Test Plan

Automated (CI)

• Unit tests pass for both skills — metadata, tool counts, content/description lengths, cross-skill references, cases tool, referenced content (20 tests)
• Unit tests pass for attack_discovery_search tool — injection guards, spaceId validation (12 tests)
• Unit tests pass for alert_analysis inline tool handler — entity extraction, correlation query, error paths (11 tests)
• Unit tests pass for use_agent_builder_attachment hook — conditional agentId when skills enabled (7 tests)
• Eval suite examples cover skill activation routing for both skills plus distractor queries (12 examples)
• ExpectedSkillInvocation evaluator validates expectedSkill / shouldNotActivateSkill metadata against OTEL traces
• ESLint passes on all changed files
• TypeScript type check passes for security_solution and kbn-evals-suite-agent-builder projects
• node scripts/check_changes.ts passes
• Buildkite CI passes

Manual verification

• Skills appear in Agent Builder skill listing when AGENT_BUILDER_EXPERIMENTAL_FEATURES_SETTING_ID is enabled
• Threat hunting queries activate the threat-hunting skill (e.g. "Hunt for lateral movement in my environment")
• Alert investigation queries activate the alert-analysis skill (e.g. "Help me triage this critical LSASS alert")
• threat-hunting skill can create cases via platform.core.cases when escalating confirmed findings
• alert-analysis skill's get-related-alerts inline tool returns related alerts using space-scoped .alerts-security.alerts-<spaceId> index
• alert-analysis skill references entity-analytics for deeper entity profiling (visible in skill content)
• Threat hunting agent hides agentId when skills are enabled (attachment hook respects feature flag)
• Non-security queries (e.g. "Show me Kibana dashboards") do not activate security skills

Follow-up PRs

• Detection Engineering skill + manage_rule_exceptions tool — net-new skill for rule authoring, coverage gaps, and exception management (tracked separately, not part of this decomposition)
• Scout API integration tests — [Security Solution] Add Scout API integration tests for Agent Builder security skills #259755

Closes https://github.com/elastic/security-team/issues/15697

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[stephmilovic]

Suggested change

	if ((await uiSettings.get(AGENT_BUILDER_EXPERIMENTAL_FEATURES_SETTING_ID)) === true) {
	return {
	status: 'unavailable',
	reason:
	'Skills are enabled, which takes precedence over threat hunting agent availability',
	};
	}
	return getAgentBuilderResourceAvailability({ core, request, logger });

If skills are enabled, the threat hunting agent should not be available.

[stephmilovic]

We also invoke attachments with the threat hunting agent, and should add a condition to use_agent_builder_attachment.ts that if skills are enabled, do not pass an agentId argument

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

Both items from @stephmilovic's review addressed:

1. Agent availability gating: When AGENT_BUILDER_EXPERIMENTAL_FEATURES_SETTING_ID is true, the agent returns { status: 'unavailable' } — see threat_hunting_agent.ts line 93.

2. Conditional agentId in attachments: use_agent_builder_attachment.ts reads the setting and conditionally omits agentId: ...(skillsEnabled ? {} : { agentId: THREAT_HUNTING_AGENT_ID }) — see line 79.

Additionally, the sample alertAnalysisSampleSkill (skills/alert_analysis_skill.ts) has been deleted since the registered alertAnalysisSkill (skills/alert_analysis/) supersedes it.

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[jonwalstedt]

Nice work Patryk! 🚢

Skill vs. Agent: Comparison Summary

Setup: Two agent configurations tested on the same prompts in the same Elastic Security environment. The "skill" version pre-loads SKILL.md files with ES|QL templates from a filestore before acting. The "agent" (threat hunting agent) uses Elastic Security tools directly without pre-loading.

---

Results by Task

Task	Winner	Key differentiator
Lateral movement hunt (7d)	Skill	Found 4-user PowerShell anomaly via aggregation query; agent never ran it
C2 beaconing	Skill	Pivoted to process-frequency analysis when network logs absent; surfaced 5 IPs, counts, paths. Agent stayed surface-level
Brute force login	Tie	Both hit dead end (no auth indices). Skill got there in 1 wildcard query vs. 3 targeted searches; gave credential-spraying query too
Rare process execution	Ambiguous	Different query filters → different results. Agent found 3 processes cheaply; skill found 1 rare + full landscape including mimikatz (11×)
Summarize malware alert	Split	Skill: 20-alert scope table, 5 C2 IPs, MITRE T1003, explicit "True Positive". Agent: found "Detect Malware Only" policy gap (critical config finding skill missed)
Write certutil ES|QL rule	Agent slight edge	Agent covered both -// flag prefixes, ftp://, MITRE+schedule guidance. Skill had better index targeting
Find lateral movement from host	Tie	Same conclusion, same IPs. Skill broader investigation + data gap disclosure; agent tighter table output
How many critical alerts?	Agent (marginal)	Skill loaded full SKILL.md unnecessarily; agent answered in one sentence
Complex flow (Alert→VT→On-call→Slack)	Skill	Used get-related-alerts for 20 structured alerts + Security Labs enrichment. Agent did flat summary only

---

Overall Pattern

Skill wins on depth (5-6/9 tasks), especially when:

• Data is absent and investigation requires pivoting (C2 beaconing is the clearest example)
• Alert triage requires scope awareness across related alerts
• Template queries surface patterns the agent would never formulate independently

Agent wins on efficiency — consistently 40–60% fewer tokens and faster to first token. Also occasionally surfaces unique signals: the "Detect Malware Only" policy gap, ML job recommendations, and risk engine status check are things the skill never exercised.

---

Core Tradeoff

Dimension	Skill	Agent
Avg. token cost	Higher (sometimes 4×)	Lower
Template queries	Yes — ready-to-run patterns, catches edge cases (credential spraying, proxy beaconing)	No
Alert correlation	get-related-alerts → full scope	Flat alert list only
Elastic ecosystem tools	Never uses ML jobs, risk engine, entity analytics	Uses them (even when not enabled)
Error recovery	Investigates failures, adapts queries	Moves on faster
Simple factual queries	Wastes tokens loading SKILL.md	Answers directly
Security Labs / product docs	Available in this env → valuable	Not installed → dead end

---

Bottom Line

The skill approach is the better threat hunter — it finds more signals and handles data gaps more gracefully. The agent is cheaper and occasionally finds high-value operational findings (config gaps, ML job recommendations) that the skill ignores entirely. For the tasks in this eval, they are complementary rather than substitutes, which is exactly the architecture the PR implements: decompose by specialization and let the router decide which skill activates.

[jonwalstedt]

This change result.success flips from true to false for empty arrays. This is correct behavior, but any existing callers passing [] will now get a schema error instead of an empty result. Might be worth checking downstream how callers handle this

[patrykkopycinski]

/ci

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: adfbee0

Failed CI Steps

• FTR Configs #51
• Build API Docs

Test Failures

• [job] [logs] FTR Configs #51 / task_manager migrations 8.5.0 migrates active tasks to set enabled to true

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	9241	9242	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	11.5MB	11.5MB	+603.0B

History

• 💔 Build #419380 failed e40496b
• 💛 Build #417613 was flaky a25fe92
• 💛 Build #417556 was flaky 56db000
• 💔 Build #417482 failed 9bb0fd7
• 💛 Build #417039 was flaky 250eeee
• 💔 Build #416910 failed 71c79d8

[denar50]

Code review only. LGTM for files owned by the Detection Engine.

[jaredburgettelastic]

Entity Analytics changes LGTM 👍

Thank you!

[patrykkopycinski]

/ci

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

• 🛠️ Update Documentation: Commit on current branch
• 🛠️ Update Documentation: Create PR

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}