Add markdown output, CSS-verbatim, and GitHub sharing

[davethegut]

Summary

• Dual output format: Skill now infers HTML vs markdown from prompt context. "Post on the PR" → markdown. "Reference doc" → HTML. Default is HTML with markdown offered at save step. No extra questions asked.
• CSS-verbatim instruction: Explicitly tells the LLM to copy the <style> block unchanged from template.html — prevents CSS drift, reduces token waste, and keeps output consistent across generations.
• GitHub-native sharing: Markdown output renders natively in PRs, issues, and repo files. Step 5 offers gh pr comment --body-file or commit-to-branch. Always offers cross-format conversion.

Test plan

• Run /deep-dive with no sharing signal → should default to HTML
• Run with "deep dive into X and post it on the PR" → should produce markdown
• Run with "create a reference page for X" → should produce HTML
• Verify HTML output copies CSS verbatim from template (diff the style block)
• Verify markdown output renders correctly when pasted into a GitHub PR comment

🤖 Generated with Claude Code

[davethegut]

This is a test for the new .md abilities in this skill - Info not applicable to this PR

Automated Alert-to-Investigation Pipeline — Deep Dive

A comprehensive reference for Epic #16339 — the end-to-end pipeline connecting alert generation to investigation in Elastic Security. Generated 2026-03-19 from elastic/security-team.

Table of Contents

• Overview
• The Problem
• Architecture
• Pipeline Stages
• Work Streams & Dependencies
• Key Components & Types
• Engineering Momentum
• Success Criteria
• Related Epics & Cross-Team Impact
• Further Reading

Overview

The Automated Alert-to-Investigation Pipeline is an epic (#16339) that connects Elastic Security's three most powerful but currently disconnected features — Alerts, Cases, and Attack Discovery — into a single automated workflow. Instead of analysts manually triaging each alert, searching for related cases, and running Attack Discovery from scratch, the pipeline deduplicates alerts, extracts entities, matches them to open cases, and triggers incremental Attack Discovery automatically.

The initiative is owned by the Security AI team (PM: @davethegut, Eng: @patrykkopycinski) and targets the EASE (Elastic AI SOC Engine) subscription tier.

The Problem

SOCs running Elastic Security at scale face a compounding triage burden:

• Alert volume compounds: A single lateral movement campaign across 10 hosts generates hundreds of near-identical alerts across 30+ detection rules. In production environments, 60-80% of alerts describe incidents already represented by other alerts.

• Manual glue work: Analysts are the integration layer between features. For every alert: review it, search for a related case, attach it (or create a new one), navigate to Attack Discovery, select alerts, run generation, read results, go back to the case. Repeat for every alert, every shift.

• Attack Discovery operates in isolation: AD regenerates from scratch every run. A case with 500 already-analyzed alerts that receives 10 new ones must re-analyze all 510 — discarding previous insights and consuming full LLM budget each time.

• 100-alert ceiling: Current AD can only process 100 alerts per run, far below production-scale volumes.

Before (manual):
  Alert → Analyst reviews → Analyst searches for case → Analyst attaches →
  Analyst opens AD → AD regenerates ALL alerts → Analyst reads results →
  Analyst returns to case → Repeat per alert

After (automated):
  Alert → Dedup → Entity extraction → Case matching → Auto-attach →
  Incremental AD (delta only) → Case comment with findings →
  Analyst opens case with complete context

Architecture

The pipeline is composed of four subsystems that flow into each other:

graph TD
  subgraph dedup ["1. Alert Deduplication"]
    Vectorize["Vectorization Engine<br/>608-dim feature vectors"]
    Cluster["Hybrid Clustering<br/>Vector + LLM comparison"]
    LeaderState["Leader State Persistence<br/>Cross-run incremental"]
    DedupTags["Dedup Tagging<br/>kibana.alert.dedup.*"]

    Vectorize -->|vectors| Cluster
    Cluster -->|leaders| LeaderState
    Cluster -->|tags| DedupTags
  end

  subgraph grouping ["2. Alert Grouping Pipeline"]
    EntityExtract["Entity Extraction<br/>13 observable types from ECS"]
    CaseMatch["Case Matching Engine<br/>4 strategies with scoring"]
    GroupWorkflow["Scheduled Grouping Workflow<br/>12-step Task Manager pipeline"]

    EntityExtract -->|entities| CaseMatch
    CaseMatch -->|scores| GroupWorkflow
  end

  subgraph ad ["3. Attack Discovery Evolution"]
    BatchedAD["Batched AD<br/>Removes 100-alert ceiling"]
    IncrementalAD["Incremental AD<br/>Delta-only processing"]
    CaseScopedAD["Case-Scoped AD<br/>Filtered to case alerts"]
    AutoTrigger["Auto Trigger<br/>On alert attachment"]

    BatchedAD -->|batch + merge| IncrementalAD
    IncrementalAD -->|delta| CaseScopedAD
    CaseScopedAD --> AutoTrigger
  end

  subgraph enrichment ["4. Case Enrichment"]
    AutoObs["Observable Auto-Extraction"]
    EventTriggers["Event Trigger System"]
    EntitySearch["Entity-Based Case Search"]
  end

  RawAlerts["Raw Alerts"] --> dedup
  DedupTags -->|leader alerts only| grouping
  GroupWorkflow -->|attach to cases| AutoTrigger
  GroupWorkflow -->|new observables| AutoObs
  AutoTrigger -->|AD results| CaseComment["Case Comments"]
  AutoObs -->|enriched observables| CaseMatch
  EventTriggers -->|triggers| IncrementalAD
  EntitySearch -->|find related| CaseMatch

Loading

Pipeline Stages

Stage 1: Alert Deduplication (60-80% noise reduction)

A hybrid vector + LLM pipeline identifies duplicate alerts before they enter the triage queue.

Component	What It Does
Vectorization Engine (#16173)	Generates 608-dimensional feature vectors from alert fields via n-gram hashing
Hybrid Clustering (#16174)	Two-stage online clustering: fast vector similarity screening, then selective LLM comparison for borderline cases
Leader State Persistence (#16175)	Elasticsearch index-backed state for cross-run incremental dedup with configurable eviction policies
Workflow Steps (#16176)	Registered as composable Elastic Workflow steps via workflows_extensions API
Dedup Tagging (#16177)	Structured kibana.alert.dedup.* tags that downstream pipelines can filter on

Key insight: Dedup runs upstream of everything else. By reducing 1,000 alerts to ~200-400 unique leaders, every downstream component (entity extraction, case matching, AD) operates on cleaner signal at lower cost.

Stage 2: Entity Extraction & Case Matching

Extracted entities become the scoring currency for matching alerts to investigations.

Component	What It Does
Entity Extraction Engine (#16179)	Extracts 13 observable types (hosts, users, IPs, domains, file hashes, etc.) from ECS fields with configurable mappings and exclusion filters
Case Matching Engine (#16180)	Scores alerts against open cases using entity overlap. 4 strategies: strict (exact match), relaxed (partial), weighted (configurable per entity type), temporal (time-windowed)

Stage 3: Automated Grouping & AD

The orchestration layer that turns building blocks into a hands-off pipeline.

Component	What It Does
Scheduled Grouping Workflow (#16181)	12-step Task Manager pipeline running on configurable intervals (default 15 min). Includes dry-run mode, per-space configuration, and execution history
Batched AD (#16182)	Removes the 100-alert ceiling. Processes alerts in batches with hierarchical merge, auto-discovering optimal batch sizes per LLM connector
Incremental AD (#16183)	Tracks processed_alert_ids per case. Only sends new (delta) alerts + existing narrative to the LLM. Merges new findings with previous discoveries
Case-Scoped AD (#16186)	case_id parameter on AD API. Filters alerts to those attached to a specific case. Discoveries queryable by case
Auto AD Trigger (#16187)	Hook in the alert attachment flow. Fires async AD scoped to the case. Posts results as a case comment

Stage 4: Case Enrichment

Component	What It Does
Observable Auto-Extraction (#16184)	Automatically extracts observables when alerts are attached to a case
Event Trigger System	Debounced triggers that fire when case state changes (e.g., "new alerts arrived → run AD")
Entity-Based Case Search	Query: "which open cases involve host.name: web-server-04?" Returns scored results in < 1 second across 1,000+ cases

Work Streams & Dependencies

Four work streams with clear sequencing:

graph LR
  WS1["WS1: Foundation<br/>Entity Extraction + Case-Scoped AD"]
  WS2["WS2: Intelligence<br/>Case Matching + Batched AD"]
  WS3["WS3: Automation<br/>Incremental AD + Grouping Workflow + Auto Trigger"]
  WS4["WS4: Enrichment & Dedup<br/>(parallel track)"]

  WS1 --> WS2
  WS2 --> WS3
  WS4 -.->|cleaner input| WS2
  WS4 -.->|enriched observables| WS2

  style WS1 fill:#0B64DD22,stroke:#0B64DD
  style WS2 fill:#00BFB322,stroke:#00BFB3
  style WS3 fill:#731DCF22,stroke:#731DCF
  style WS4 fill:#F04E9822,stroke:#F04E98

Loading

Work Stream	Issues	Dependency
WS1: Foundation	#16179 Entity Extraction, #16186 Case-Scoped AD	Start here — no dependencies
WS2: Intelligence	#16180 Case Matching, #16182 Batched AD	After WS1
WS3: Automation	#16183 Incremental AD, #16181 Grouping Workflow, #16187 Auto Trigger	After WS1+2
WS4: Enrichment & Dedup	#16184 Case Observables, #16178 Alert Dedup (5 sub-issues)	Parallel — independent track

Key Components & Types

Entity Extraction — 13 Observable Types

Extracted from standard ECS fields with configurable mappings:

Observable Type	Example ECS Fields
Host	host.name, host.hostname, host.id
User	user.name, user.id, user.domain
IP Address	source.ip, destination.ip, host.ip
Domain	dns.question.name, url.domain
File Hash	file.hash.sha256, file.hash.md5
Process	process.name, process.executable
URL	url.full, url.original
Email	email.from.address, email.to.address
Registry	registry.key, registry.path
File Path	file.path, file.name
Port	destination.port, source.port
MAC Address	host.mac, source.mac
Certificate	tls.server.hash.sha256

Case Matching — 4 Scoring Strategies

Strategy	How It Scores	Best For
Strict	Exact entity match required	High-confidence grouping, low false positives
Relaxed	Partial entity overlap accepted	Broader grouping, catching related activity
Weighted	Configurable weights per entity type (e.g., host.name = 0.8, IP = 0.3)	Tuned for specific environments
Temporal	Entity overlap + time-window proximity	Time-bounded campaigns

Incremental AD — Delta Processing

Existing case: 500 alerts already analyzed → narrative exists
10 new alerts arrive

Without incremental: LLM processes 510 alerts (full cost)
With incremental:    LLM processes 10 alerts + existing narrative (~80% cost reduction)

The merge step uses the LLM to intelligently combine new findings with the existing attack narrative — updating, extending, or creating new attack chains as warranted.

Engineering Momentum

Active PRs (in development)

PR	What It Builds	Status
#257832	Case-scoped Attack Discovery runs	Open
#257831	Batched AD with hierarchical merge	Open
#254356	Alert deduping, grouping, incremental AD & attachment type	Open
#257957	Automated Alert-to-Investigation Pipeline — E2E spike	Open

Recently Merged (foundation in place)

PR	What It Delivered
#256200	Telemetry for Attacks and Alerts Alignment
#255442	Attack transform functions refactor

Success Criteria

Metric	Target	Validation Approach
Alert dedup rate	60-80% duplicates identified	Run against production-scale alert sets from attack emulation (Atomic Red Team)
Incremental AD cost	~80% reduction vs full regen	Benchmark token usage: incremental vs full on same case across multiple LLM connectors
AD alert ceiling	50,000+ alerts processable	Load test with synthetic alerts at 10K, 25K, 50K+
Triage time	Hours → minutes at scale	Time-to-first-investigation with pipeline enabled vs disabled at 1,500+ alerts
Case-AD auto-link	Zero manual steps	E2E test: alert fires → appears in case with AD comment, no analyst interaction
Entity search	< 1 second	P95 response time against 1K+ open cases with populated observables

Related Epics & Cross-Team Impact

Direct Child Epics

Epic	Role
Alert Dedup #16178	Upstream signal — reduces noise before grouping
Alert Grouping Workflow #16185	Core pipeline — entity extraction, matching, grouping, AD, enrichment
Case-AD Integration #16188	Foundation — simplest viable integration (case-scoped AD + auto-trigger)
CI Agent #16194	Meta-level — monitors and improves LLM call quality across pipeline

Cross-Team Acceleration

This pipeline provides shared primitives that benefit 8+ in-flight epics:

• Alert Triage Workflow (#15370) — gets cleaner input + shared entity extraction
• EASE Agent Builder Tools (#14620) — gets concrete automation skills
• AD 2.0 + Workflows Integration (#14619) — gets its foundation layer
• Agent Skills Ecosystem (#15972) — gets composable Workflow steps
• Security Tools for Agent Builder (#15695) — gets entity search + matching as tools
• Workflow Template Library (#15748) — gets a flagship template
• NL Threat Hunting (#12672) — gets structured entity data
• Attacks/Alerts Telemetry (#256200) — already shipped, provides measurement layer

graph TD
  Pipeline["Alert-to-Investigation Pipeline<br/>#16339"]

  Pipeline --> Triage["Alert Triage Workflow<br/>#15370"]
  Pipeline --> EASE["EASE Agent Builder<br/>#14620"]
  Pipeline --> AD2["AD 2.0 + Workflows<br/>#14619"]
  Pipeline --> Skills["Agent Skills Ecosystem<br/>#15972"]
  Pipeline --> Tools["Security Tools for AB<br/>#15695"]
  Pipeline --> Templates["Workflow Templates<br/>#15748"]
  Pipeline --> Hunting["NL Threat Hunting<br/>#12672"]
  Pipeline --> Telemetry["Attacks/Alerts Telemetry<br/>#256200 ✅"]

  style Pipeline fill:#0B64DD,stroke:#0B64DD,color:#fff
  style Telemetry fill:#00BFB322,stroke:#00BFB3

Loading

Further Reading

• Epic #16339 — Automated Alert-to-Investigation Pipeline — Full epic with all sub-issues
• PR #257957 — E2E Spike — End-to-end implementation spike
• PR #257832 — Case-Scoped AD — Foundation PR for case-scoped Attack Discovery
• PR #257831 — Batched AD — Removes the 100-alert ceiling
• PR #254356 — Alert Dedup + Grouping — Core pipeline components

---

Deep-dive reference for Automated Alert-to-Investigation Pipeline in elastic/security-team — Generated 2026-03-19