feat(security): Incremental Attack Discovery - Delta + Progressive Modes

[patrykkopycinski]

Summary

Implements Incremental Attack Discovery with dual-mode support (delta + progressive) enabling efficient processing of large alert volumes through bounded round processing.

What This PR Does

Core: Round-based processing with graph bypass

• Fetches and anonymizes all alerts once upfront
• Splits into rounds (auto-tuned batch size based on 32K context budget)
• Each round passes pre-fetched alerts directly to the AD graph, bypassing the ES re-fetch (leverages the entry edge bypass when anonymizedDocuments is pre-populated)
• Merges insights across rounds with Jaccard + meaningful-word dedup

Two modes:

• Progressive (ad-hoc): Process large alert sets in bounded rounds. Auto-enabled in UI when alert count exceeds model-aware threshold (50 for OSS, 200 for frontier, 100 default).
• Delta (scheduled): Process only NEW alerts since last run. Backend-only, tracks processed alerts in a namespaced ES index with ILM (30d retention). TODO: wire into schedule creation UI.

Quality optimizations (eval-validated):

• Alert clustering by host/rule before splitting into rounds
• Adaptive batch sizing: shrinks batch when model produces ≤1 insight
• Robust JSON parsing fallback for OSS models that don't support tool calling

---

Real LLM Eval Results

All results from @kbn/evals framework with real LLM calls, real attack discovery data (load_attack_discovery_data), and automated evaluators.

Progressive Mode — Cross-Model (115 alerts, 25-50/round)

Metric	GPT 5.2 (Azure)	Qwen 2.5 7B (MLX local)
Rounds completed	3	3-4
Insights produced	6-11 per run	5-9 per run
Max tokens/round	13,323	14,260
Total tokens	~26K-31K	~28K-31K
Context budget (<32K)	PASS	PASS

Delta Mode — Qwen 2.5 7B

Scenario	Alerts Processed	Tokens	Latency	Efficiency
Initial (115 new)	115 (3 rounds)	27,640	159s	-
Incremental (23 new/115 total)	23 (1 round)	6,757	31s	80% skipped

Quality Comparison — Single-Pass vs Incremental (100 alerts, Qwen 7B)

	Single-Pass	Incremental Progressive
InsightCount	3	5
AvgAlertIds/Insight	8.67	13.40
TotalTokens	22,503	30,637

Quality Improvements A/B Results (100 alerts, 25/round, Qwen 7B)

Improvement	InsightCount	Decision
Baseline (25/round)	9	Default
+Alert clustering	8 (better coherence)	KEPT
+Adaptive batch	7+ (unblocks stuck model)	KEPT
+Synthesis pass	6 (worse — collapsed)	Dropped
+Progressive context	4 (worse — anchoring bias)	Dropped

---

Key Implementation Details

Files Changed

Core incremental logic:

• server/lib/attack_discovery/incremental/index.ts — Orchestrator with computeAlertsPerRound (32K budget → ~50 alerts/round)
• server/lib/attack_discovery/incremental/round_processor.ts — Alert clustering + adaptive batch sizing
• server/lib/attack_discovery/incremental/insight_merger.ts — Dedup by alert ID overlap (30%+), title similarity, MITRE tactics merge
• server/lib/attack_discovery/incremental/state_tracker.ts — Namespaced ES index with ILM, lazy creation, space-aware
• server/lib/attack_discovery/incremental/feature_flags.ts — enabled: false by default, configurable
• server/lib/attack_discovery/incremental/types.ts — Config with contextBudget, MergeStrategy

Integration (graph bypass):

• invoke_attack_discovery_graph/index.tsx — Added anonymizedDocuments? param, passes to graph.invoke() initial state
• invoke_incremental_attack_discovery.ts — Fetches + anonymizes once via getAnonymizedAlerts, passes round subsets

API:

• common_attributes.gen.ts — incrementalMode, sessionId, incrementalConfig (all optional, backward compatible)
• generate_discoveries.ts — Feature flag check + branching to incremental or standard mode

UI:

• use_attack_discovery/index.tsx — Model-aware threshold: OSS=50, frontier=200, default=100

Eval suite:

• kbn-evals-suite-attack-discovery/ — Progressive, delta, quality comparison, A/B experiment specs
• Robust JSON parser for OSS models, qualityOptions for A/B testing

Backward Compatible

All new fields are optional. Existing clients continue to work unchanged. Feature flag enabled: false by default.

---

Testing

Run evals locally

# Start Scout servers
node scripts/scout start-server --arch stateful --domain classic

# Load attack discovery data
node x-pack/solutions/security/plugins/security_solution/scripts/load_attack_discovery_data.js \
  --kibanaUrl http://localhost:5620 --elasticsearchUrl http://localhost:9220

# Run incremental evals with local model
CONNECTORS=$(echo '{"lmstudio":{"name":"LM Studio","actionTypeId":".gen-ai","config":{"apiProvider":"Other","apiUrl":"http://localhost:1234/v1/chat/completions","defaultModel":"your-model"},"secrets":{"apiKey":"not-needed"}}}' | base64)

TEST_RUN_ID="test-$(date +%s)" \
KBN_EVALS_SKIP_PREFLIGHT_EXPORT=true \
EVALUATION_CONNECTOR_ID=lmstudio \
KIBANA_TESTING_AI_CONNECTORS="$CONNECTORS" \
node scripts/evals run --suite attack-discovery --grep "Incremental" --project lmstudio

Type check

yarn test:type_check --project x-pack/solutions/security/plugins/elastic_assistant/tsconfig.json

---

Risk

Low — Feature flag enabled: false by default. All incremental processing is opt-in. Standard AD mode is completely unchanged. Backward compatible API.

🤖 Generated with Claude Code

Production-Readiness Checklist — Agent Skills Ecosystem

Generated against [Epic] Creation of the Agent Skills Ecosystem for Elastic Security.

Narrative role: Attack Discovery skill capability. Directly referenced by the end-to-end Alert Investigation Pipeline (#257957) and Alert Deduplication (#254356).

Must-do before this can ship

• Clean the diff. 393k additions / 6944 files is almost certainly yarn.lock, build output, or a bad merge. Rebase and confirm the real change-set is a few thousand lines
• Fix the 2 failing CI checks after cleanup
• Replace the hard-coded 32K context budget with a model-capability read from connector.capabilities — frontier models (200K) shouldn't be artificially capped
• Ship the delta-mode schedule-creation UI — today it's documented as TODO. Without it the feature is backend-only and cannot be adopted
• Document and get review on the processed-alerts ES index: retention (30d ILM stated), namespace, PII, space awareness
• Promote the per-model eval table (GPT 5.2, Qwen 2.5 7B, 115/100 alerts) into the Monitoring tab from #261057 — this is the vision's "measurable impact" KPI and shouldn't live only in the PR body
• Emit a kill-switch-able telemetry event for every round so we can spot context-budget regressions in production

Follow-ups (post-merge)

• Publish Attack Discoveries as durable Case attachments (unified v2) instead of a bespoke attachment type (coordinate with #254356, #260544, #257708)
• Pipe dedup output (#254356) as the input cluster — vision's upstream-noise-reduction pattern

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!
• Click to trigger kibana-entity-store-performance-from-pr for this PR!
• Click to trigger kibana-storybooks-from-pr for this PR!

[patrykkopycinski]

Cross-Model Eval Results (Additional OSS Models)

Ran incremental progressive mode against 4 additional models to validate breadth of OSS support.

Results Summary

Model	Params	Hosting	Progressive Insights	Single-Pass Insights	Status
GPT 5.2 (baseline)	Frontier	Azure	11 per run	-	✅ PASS
GPT-OSS 20B	20B	Bedrock	6 (R1), R2 failed (500)	5	⚠️ PARTIAL
Qwen 2.5 Coder 7B	7B	Local MLX	5-9 per run	3	✅ PASS
Llama 3.1 8B	8B	Local MLX	0 (JSON parse fail)	-	❌ FAIL
DeepSeek-R1 7B	7B	Local MLX	0 (CoT, no JSON)	-	❌ FAIL
Mistral Small 24B	24B	Local MLX	0 (timeout 1108s)	-	❌ FAIL

Key Findings

1. GPT-OSS 20B is the best OSS option — 6 insights/round with proper tool calling, but needs cloud hosting (Bedrock). R2 failed with Bedrock 500 error (likely rate limiting).

2. Qwen 2.5 7B is the only local 7B that works — produces parseable JSON via our fallback extraction. Llama 3.1 and DeepSeek-R1 return text formats our parser can't handle.

3. DeepSeek-R1's chain-of-thought is incompatible — wraps output in <think> tags, doesn't produce structured JSON. Would need a model-specific prompt.

4. 24B models timeout locally — Mistral Small 24B took 18+ min per round, exceeding Kibana's inference proxy timeout. Works fine cloud-hosted.

5. Recommended models for incremental AD: GPT-OSS 20B (Bedrock), Qwen 2.5 7B+ (local), or any frontier model. Llama 3.1 and DeepSeek-R1 need parser improvements for their output formats.

Test Environment

• LM Studio with MLX 4-bit quantized models on Apple Silicon (M-series)
• Scout servers (ES 9.4.0-SNAPSHOT + Kibana dev)
• 115 real attack discovery alerts from load_attack_discovery_data
• @kbn/evals framework with automated evaluators