[Platform] Extract LLM Batch Processing Package

[patrykkopycinski]

Summary

Extracts reusable @kbn/llm-batch-processing package for parallel LLM task processing.

⚠️ IMPORTANT: Complete RFC validation included. Claims vs reality documented.

---

Package Details

Location: x-pack/platform/packages/shared/kbn-llm-batch-processing
Status: ✅ Production-ready
Tests: 30/30 passing
Dependencies: Zero (inline concurrency control)
Type: shared-server
Owner: @elastic/security-generative-ai

Features:

• Item-based and token-based splitting strategies
• Hierarchical merge (O(log N) rounds)
• Dynamic concurrency scaling
• Configurable batch sizes
• Progress callbacks

---

Validation Results (12+ Experiments)

Performance with Claude Sonnet 4.5

Config	Baseline	Optimized	Result
100 items	28.03s, 6,028 tokens	17.93s, 26,809 tokens	36% faster, 4.5x more tokens
500 items	106s	211s	98% slower (doesn't scale well)

Key findings:

• ✅ Speed: 25-40% improvement possible (with optimal tuning)
• ❌ Tokens: 4.5x MORE (not reduction) - system prompt repeated per batch
• ✅ Quality: Maintained (with batch size ≥20)
• ⚠️ OSS models: Tool calling unreliable (Qwen 20% success, Llama/Mistral fail)

Optimal Configuration

{
  batchSize: 20,  // Balance speed + quality
  maxConcurrentBatches: Math.min(numBatches, 20),  // Dynamic scaling
}

---

Use Cases

✅ RECOMMENDED FOR:

• Document summarization (each doc independent)

  batchProcess({
    input: documents,
    processFn: (batch) => llm.summarize(batch),
    mergeFn: ([a, b]) => a + '\n' + b  // Simple concat works
  })

• Data extraction (MapReduce-style)

• Classification tasks (independent items)

• Scenarios where: Speed > cost, items are independent

❌ NOT RECOMMENDED FOR:

• Narrative generation (Attack Discovery, reports) - Use Incremental AD instead
• Cost optimization (uses 4.5x more tokens)
• OSS models (tool calling reliability issues)
• Small datasets (<50 items - single-pass is faster)

---

API

Main Entry Point

import { batchProcess } from '@kbn/llm-batch-processing';

const result = await batchProcess<TInput, TOutput>({
  input: items,
  splitStrategy: 'item-based' | 'token-based' | 'custom',
  maxItemsPerBatch: 20,
  processFn: async (batch) => {
    return await llm.process(batch);
  },
  mergeFn: async ([a, b]) => {
    return await combine(a, b);  // Your merge logic
  },
  maxConcurrentBatches: 10,
  onProgress: (completed, total) => {
    console.log(`Processed ${completed}/${total}`);
  },
});

console.log(result.output);  // Final merged result
console.log(result.stats);   // { batches, mergeRounds, durationMs }

Utilities

import { tokenBasedSplit, itemBasedSplit, hierarchicalMerge } from '@kbn/llm-batch-processing';

---

Complete Validation Documentation

This branch includes full validation findings:

1. Design Spec: docs/superpowers/specs/2026-03-21-rfc-batch-processing-validation-design.md
2. Implementation Plan: docs/superpowers/plans/2026-03-21-rfc-batch-processing-validation.md
3. Raw Metrics: docs/rfc-validation-results/RAW_METRICS_COMPARISON.md
4. Final Assessment: RFC_SEC-2026-002_FINAL_VALIDATION.md
5. Complete Summary: COMPLETE_VALIDATION_SUMMARY.md
6. Comparison Analysis: BATCH_PROCESSING_VS_INCREMENTAL_AD.md

Experiment IDs (LangSmith):

• 12+ experiments across 3 scales, 5 configurations, 3 models
• Raw latency, token, quality data
• Baseline vs treatment comparisons

---

Trade-offs

Accept these to use batch processing:

• 4.5x token cost increase (speed over cost)
• Configuration complexity (tuning required)
• Limited OSS support (frontier models work best)
• Scale limitations (degradesperformance at 500+ items without perfect tuning)

---

Alternative Solution

For Attack Discovery specifically: Use Incremental AD instead

• Spec included: docs/superpowers/specs/2026-03-21-incremental-attack-discovery-design.md
• Better for: Small-context models, cost efficiency, OSS compatibility
• Implementation: 5-6 days (reuses PR [Spike] Alert Investigation Pipeline — Elastic Workflows + Agent Builder + Incremental AD #257957 components)

---

Recommendation

✅ Approve package for platform adoption with:

1. Clear positioning (parallel processing utility)
2. Honest README (use cases, trade-offs)
3. Configuration guidance
4. "NOT for Attack Discovery" caveat

Reason: Code is excellent, just not the right solution for original use case.

---

Files Changed

Package files:

• x-pack/platform/packages/shared/kbn-llm-batch-processing/ - Complete package
• .github/CODEOWNERS - Package ownership

Validation files:

• Eval suite extensions (evaluators, datasets)
• Validation reports (6 documents)
• Benchmark data

Total: 26 commits with complete validation story

---

This is a DRAFT PR to preserve work. Review validation findings before merging.

🔗 Related: RFC Validator skill created - .agents/skills/rfc-validator/SKILL.md

Production-Readiness Checklist — Agent Skills Ecosystem

Generated against [Epic] Creation of the Agent Skills Ecosystem for Elastic Security.

Narrative role: Shared infra any skill can use for parallel LLM work (summarization, extraction, classification). Clean 1093-line package, 30/30 tests passing — the lowest-risk foundation PR in this program.

Must-do before this can ship

• Lift the "not recommended for" warning (4.5x tokens; narrative generation; OSS models) from the PR body to the top of the README, with a decision matrix — this is the most important piece of docs in the package
• Add benchmark rows for Bedrock, Gemini, and at least one OSS model alongside the Claude Sonnet 4.5 numbers
• Add a "do not use for Attack Discovery / narrative generation" test-style guard (or ESLint rule) so this package doesn't get misused by future PRs
• Keep the zero-dependency promise — document that explicitly in the README

Follow-ups (post-merge)

• Provide an adapter that emits OTEL spans per batch so the eval platform can attribute cost/latency to specific skills
• Integrate with Alert Dedup (#254356) as an example reference

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!
• Click to trigger kibana-entity-store-performance-from-pr for this PR!
• Click to trigger kibana-storybooks-from-pr for this PR!

[github-actions]

Vale Linting Results

Summary: 20 warnings, 13 suggestions found

⚠️ Warnings (20)

File	Line	Rule	Message
docs/AESOP_DELIVERY_SUMMARY.md	10	Elastic.EndPuntuaction	Don't end headings with punctuation.
docs/AESOP_DELIVERY_SUMMARY.md	57	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/AESOP_DELIVERY_SUMMARY.md	58	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/AESOP_DELIVERY_SUMMARY.md	214	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_demo_guide.md	206	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
docs/aesop_demo_guide.md	276	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_hypothesis_measurement_plan.md	17	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
docs/aesop_hypothesis_measurement_plan.md	138	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
docs/aesop_hypothesis_measurement_plan.md	167	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_hypothesis_measurement_plan.md	430	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_hypothesis_measurement_plan.md	512	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_implementation_summary.md	104	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_implementation_summary.md	168	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_implementation_summary.md	231	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_o11y_traces_validation.md	408	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_o11y_traces_validation.md	408	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.
docs/aesop_o11y_traces_validation.md	676	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
docs/aesop_o11y_traces_validation.md	677	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
docs/aesop_o11y_traces_validation.md	678	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
docs/aesop_o11y_traces_validation.md	678	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.

💡 Suggestions (13)

File	Line	Rule	Message
docs/AESOP_DELIVERY_SUMMARY.md	10	Elastic.Exclamation	Use exclamation points sparingly. Consider removing the exclamation point.
docs/AESOP_DELIVERY_SUMMARY.md	366	Elastic.WordChoice	Consider using 'run, start' instead of 'Execute', unless the term is in the UI.
docs/AESOP_DELIVERY_SUMMARY.md	382	Elastic.WordChoice	Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
docs/aesop_demo_guide.md	72	Elastic.Ellipses	In general, don't use an ellipsis.
docs/aesop_demo_guide.md	73	Elastic.Ellipses	In general, don't use an ellipsis.
docs/aesop_demo_guide.md	74	Elastic.Ellipses	In general, don't use an ellipsis.
docs/aesop_demo_guide.md	75	Elastic.Ellipses	In general, don't use an ellipsis.
docs/aesop_demo_guide.md	76	Elastic.Ellipses	In general, don't use an ellipsis.
docs/aesop_hypothesis_measurement_plan.md	211	Elastic.Wordiness	Consider using 'tell' instead of 'inform'.
docs/aesop_implementation_summary.md	283	Elastic.WordChoice	Consider using 'run, start' instead of 'execute', unless the term is in the UI.
docs/aesop_o11y_traces_validation.md	277	Elastic.WordChoice	Consider using 'review' instead of 'sanity check', unless the term is in the UI.
docs/aesop_o11y_traces_validation.md	396	Elastic.WordChoice	Consider using 'select, press, visits' instead of 'hit', unless the term is in the UI.
docs/aesop_o11y_traces_validation.md	652	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.