feat(evals): autonomous skill discovery and generation system

[patrykkopycinski]

AESOP Spike: Autonomous Skill Discovery for Agent Builder

Summary

AESOP (Autonomous Exploration of Security Operations Patterns) is a spike exploring how an autonomous agent can discover, validate, and deploy Agent Builder skills by analyzing live Elasticsearch data — without requiring human authoring from scratch.

Problem: Agent Builder skills are manually authored today. Security teams must identify patterns, write ES|QL queries, define investigation workflows, and iterate. This is slow and requires deep platform expertise.

Solution: AESOP autonomously explores the cluster's security data, discovers cross-index relationships and behavioral patterns, synthesizes Agent Builder skills via LLM, then puts a human in the loop for validation and approval before deployment.

Stats: 105 files changed, ~31K lines added, 39 test files

---

Architecture

5-Phase Exploration Workflow

1. Schema Discovery — getMapping on all security-relevant indices to understand field structure
2. Data Profiling — terms aggregations on keyword fields + date_histogram for temporal patterns
3. Relationship Analysis — Cross-index field overlap via value set intersection
4. Pattern Mining + Conversation Analysis — Alert rule frequency, event patterns, log sources, plus Agent Builder conversation extraction (tool usage, ES|QL patterns, recurring investigation flows, failure modes)
5. Skill Synthesis — LLM generates skills from all discovery context, analyzes existing Agent Builder skills for improvements, deduplicates against previous runs

Data Sources

Uses indices.resolveIndex to discover all data streams and indices:

• Security alerts (.internal.alerts-security.alerts-*)
• Endpoint events (logs-endpoint.events.{process,file,network,dns}-*)
• System auth (logs-system.auth-*)
• Agent Builder conversations (.kibana-agent-builder-conversations) — extracts tool usage, query patterns, failure modes
• Metrics, traces, and any other non-system indices

Human-in-the-Loop Review Pipeline

Discovery → Proposed Skills → LLM Validation ←→ Auto-Improve (convergence loop)
                                                       ↓
                                              Human Review → Agent Builder
                                                  |
                                        Accept / Reject / Edit
                                        (with cross-evaluation)

---

Key Features

Convergence Validation Loop

• Iterates improve→validate automatically until score >= 0.85 or plateau detected
• Stops on: pass, plateau (2 consecutive low-delta comparisons), max iterations (5), error
• "Validate & Auto-Improve" one-click button — typical flow: 56% → 78% → 96% in 3 iterations
• Full iteration history shown as score badges in the UI

LLM-Powered Skill Validation

• 5 criteria: relevance, completeness, accuracy, specificity, safety
• Per-criteria scoring with detailed feedback, strengths, weaknesses, suggestions
• Connector picker — works with OpenAI, Bedrock, Gemini, DeepSeek
• Handles provider-specific response formats (DeepSeek <think> tags, etc.)

Agent Builder Conversation Analysis

• Extracts tool usage frequency from real analyst conversations
• Identifies common ES|QL query patterns used in practice
• Detects recurring investigation flows (tool call sequences)
• Surfaces failure modes (tool errors) to avoid in generated skills
• All insights fed into LLM context during skill synthesis

Cross-Evaluation on Rejection

• When a skill is rejected, LLM evaluates all remaining pending skills
• High severity → auto-rejected; medium/low → flagged with warning badge

Skill Improvement Proposals

• Analyzes existing Agent Builder skills against discovered data
• Built-in skills → "Create as New Skill" (customized variant)
• User skills → "Update Existing" or "Create as New" for comparison

Skill Deduplication

• Jaccard similarity on tokenized names + source index overlap
• Deduplicates within batch AND against previous exploration runs
• Keeps higher-confidence skill when duplicates detected

Persistent Rate Limiting

• ES-backed rate limiter (.aesop-rate-limits index) replacing in-memory Map
• Optimistic concurrency control (if_seq_no/if_primary_term) for atomic increments
• Fail-open on ES errors, retry-on-conflict up to 3 attempts
• Cached index creation with failure recovery (clears promise on error)

Agent Builder Integration (via Plugin Contract)

• Uses agentBuilder.skills.getRegistry({ request }) — proper plugin contract, not HTTP fetch
• Registry eagerly created while request is active (avoids stale auth in background tasks)
• Deploy order: ES update first → Agent Builder create → rollback on failure
• Re-deploy button for accidentally deleted skills

Source Tracking & Filtering

• Each skill stores which specific indices contributed (source_indices)
• derived_from field: patterns, relationships, conversations, llm, skill_improvement
• Filterable in UI via badge toggles

Index Lifecycle Management

• ILM policy (aesop-lifecycle) applied to all .aesop-* indices at creation
• 90-day retention for most indices, 7-day for rate limits, 180-day for proposed skills
• Policy created at plugin start via internal ES client

---

Production Quality

Type Safety

• ProposedSkillDocument interface defines all ES document fields
• Replaces as any casts across 6 route handlers
• Proper null guards on skill.name, confidence: 0, and extractLLMText(undefined)

Security

• Input sanitization via sanitizeSkillMarkdown() before Agent Builder deployment
• ES|QL string escaping for bucket.key values in generated queries
• Rate limiting with persistent storage and atomic operations

Accessibility

• 22+ aria-label attributes across all AESOP components
• Emoji status indicators replaced with screen-reader-friendly text
• AesopErrorBoundary wrapping all 3 AESOP routes with recovery

Testing

• 39 test files with 100+ tests
• Route handler tests for list, approve, history endpoints
• Unit tests for convergence loop (9), conversation analyzer (12), deduplicator (30), rate limiter (10)
• Skill content validation, error types, sampling strategy tests

Error Handling

• No silent .catch(() => {}) blocks — all background errors logged
• Convergence loop onError callback with skill_id context
• Deploy rollback on Agent Builder failure
• Fail-open rate limiting with retry recovery

---

Spike Findings

What Worked Well

1. LLM skill synthesis produces dramatically better skills than templates — discovery context grounds the LLM
2. Convergence loop eliminates manual improve→validate cycles — one click gets to 95%+
3. indices.resolveIndex cleanly separates data streams from backing indices
4. Cross-evaluation catches related quality issues efficiently
5. Conversation analysis surfaces real analyst patterns that pure data mining misses
6. Skill improvement proposals customize existing skills to the user's specific data

Production Considerations

1. Connector abstraction — needs adapter per connector type for messages format
2. Convergence tuning — threshold, delta, and max iterations may need per-deployment tuning
3. Scale testing — conversation analysis on large conversation volumes needs pagination
4. Deduplication — current word-level Jaccard is simple; production may need embedding similarity

---

Routes

Route	Method	Description
/internal/aesop/exploration/run	POST	Trigger exploration with LLM connector
/internal/aesop/exploration/history	GET	List past explorations with index details
/internal/aesop/exploration/{id}/progress	GET	Real-time progress polling
/internal/aesop/skills/proposed	GET	List skills (filterable by status, derived_from)
/internal/aesop/skills/{skillId}	GET	Get single skill detail
/internal/aesop/skills/{skillId}	PUT	Edit skill content
/internal/aesop/skills/{skillId}/validate	POST	LLM validation (supports auto_converge)
/internal/aesop/skills/{skillId}/improve	POST	Apply LLM suggestions + auto-revalidate
/internal/aesop/skills/{skillId}/approve	POST	Deploy via SkillRegistry (supports update_existing)
/internal/aesop/skills/{skillId}/reject	POST	Reject + cross-evaluate siblings
/internal/aesop/skills/{skillId}/unreject	POST	Restore to pending review
/internal/aesop/skills/{skillId}/redeploy	POST	Re-create in Agent Builder

Production-Readiness Checklist — Agent Skills Ecosystem

Generated against [Epic] Creation of the Agent Skills Ecosystem for Elastic Security.

Narrative role: Embodies the "self-improving detection system" from the epic's feedback-loop pattern. Depends on the rest of the eval platform + skill CLI to be safely deployable.

Must-do before this can ship

• Split this PR. +34k / 127 files → (a) 5 discovery phases, (b) LLM synthesis + dedup, (c) HITL review UI, (d) storage layer
• Querying .kibana-agent-builder-conversations to mine user chats is sensitive — add explicit consent + data-classification before any LLM prompt sees a conversation
• Every synthesized skill must pass a sandbox run against the cluster that produced it before it enters human review (otherwise you offer broken skills to analysts)
• Route every generated ES|QL through the same AST validator the #264378 PCI skill uses — no string interpolation of FROM or time ranges
• Dedup against existing skills must compare tool-graph + data-scope, not text similarity — today AESOP risks proposing near-duplicates of prebuilt skills
• Depend on #255890 for skill CLI and #261057 for eval platform — do not duplicate either
• Ship disabled-by-default and behind an experimental flag, with a kill switch on the synthesis step

Follow-ups (post-merge)

• Close the loop back to Detection Engineering (the vision's Triage → DE feedback): synthesized rule improvements must flow through the same apply_triage_feedback tool
• Add rate limiting on synthesis runs (cost budget)

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!
• Click to trigger kibana-entity-store-performance-from-pr for this PR!
• Click to trigger kibana-storybooks-from-pr for this PR!

[macroscopeapp]

🟢 Low aesop/reject_skill.ts:117

When skill.deployment.deployed is true but skill.deployment.agent_builder_skill_id is undefined, the code passes undefined to SkillAlreadyDeployedError, which expects a string. This produces a confusing error message like "Skill 'x' is already deployed to Agent Builder (ID: undefined)". Consider either making the parameter optional in SkillAlreadyDeployedError or providing a fallback value when agent_builder_skill_id is missing.

-          // 2. Validate skill is not already deployed
-          if (skill.deployment?.deployed) {
-            throw new SkillAlreadyDeployedError(
-              skillId,
-              skill.deployment.agent_builder_skill_id
-            );
-          }

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/routes/aesop/reject_skill.ts around lines 117-122:

When `skill.deployment.deployed` is true but `skill.deployment.agent_builder_skill_id` is undefined, the code passes `undefined` to `SkillAlreadyDeployedError`, which expects a `string`. This produces a confusing error message like `"Skill 'x' is already deployed to Agent Builder (ID: undefined)"`. Consider either making the parameter optional in `SkillAlreadyDeployedError` or providing a fallback value when `agent_builder_skill_id` is missing.

Evidence trail:
x-pack/platform/plugins/shared/evals/server/routes/aesop/reject_skill.ts lines 117-121 (REVIEWED_COMMIT): Shows `skill.deployment.agent_builder_skill_id` passed to SkillAlreadyDeployedError.

x-pack/platform/plugins/shared/evals/server/lib/aesop/types.ts lines 66-73 (REVIEWED_COMMIT): `deployment.agent_builder_skill_id?: string;` is optional.

x-pack/platform/plugins/shared/evals/server/lib/aesop/errors/aesop_errors.ts lines 115-126 (REVIEWED_COMMIT): `constructor(skillId: string, agentBuilderSkillId: string)` expects non-optional string.

[macroscopeapp]

🟡 Medium aesop/run_skill_validation.ts:161

When loop.run() throws, the .catch handler only logs the error and never updates the skill status. The skill remains stuck in 'validating' state with no recovery path. Consider updating the skill status to 'failed' in the catch handler, matching the error handling pattern used in runLLMValidation.

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/routes/aesop/run_skill_validation.ts around line 161:

When `loop.run()` throws, the `.catch` handler only logs the error and never updates the skill status. The skill remains stuck in `'validating'` state with no recovery path. Consider updating the skill status to `'failed'` in the catch handler, matching the error handling pattern used in `runLLMValidation`.

Evidence trail:
x-pack/platform/plugins/shared/evals/server/routes/aesop/run_skill_validation.ts lines 161-166 (convergence loop catch handler - only logs error), lines 73-78 (sets status to 'validating'), lines 318-350 (runLLMValidation catch block - updates status to 'failed'). Verified at commit REVIEWED_COMMIT.

[macroscopeapp]

🟡 Medium aesop/improve_skill.ts:52

When the skill document doesn't exist in Elasticsearch, esClient.get() throws a 404 ResponseError that is caught by the generic catch block at line 177, causing the endpoint to return 500 instead of 404. The check at line 59-61 only handles the case where _source is disabled on an existing document, not a missing document. Consider catching the specific Elasticsearch 404 error before the generic handler and returning response.notFound().

          const skillDoc = await esClient.get({
            index: '.aesop-proposed-skills',
            id: skillId,
          });

          const skill = skillDoc._source as ProposedSkillDocument | undefined;
          if (!skill) {
            return response.notFound({ body: { message: `Skill ${skillId} not found or source unavailable` } });
          }

          if (!skill.validation?.llm_feedback) {

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/routes/aesop/improve_skill.ts around lines 52-67:

When the skill document doesn't exist in Elasticsearch, `esClient.get()` throws a 404 `ResponseError` that is caught by the generic catch block at line 177, causing the endpoint to return 500 instead of 404. The check at line 59-61 only handles the case where `_source` is disabled on an existing document, not a missing document. Consider catching the specific Elasticsearch 404 error before the generic handler and returning `response.notFound()`.

Evidence trail:
- x-pack/platform/plugins/shared/evals/server/routes/aesop/improve_skill.ts lines 52-55: esClient.get() call
- x-pack/platform/plugins/shared/evals/server/routes/aesop/improve_skill.ts lines 59-61: check for !skill (only handles undefined _source)
- x-pack/platform/plugins/shared/evals/server/routes/aesop/improve_skill.ts lines 177-185: generic catch block returning 500
- x-pack/platform/plugins/shared/evals/server/routes/aesop/get_skill_detail.ts lines 58-64: correct pattern showing 404 handling for ES errors
- git_grep results showing 404 handling pattern used in reject_skill.ts:104, get_execution_detail.ts:193, workflow_state_tracker.ts:391, etc.

[macroscopeapp]

🟢 Low workflows/exploration_workflow_executor.ts:1033

The greedy regex /\[\s\S]*\]/ on line 1033 matches from the first [ to the last ] in the entire response. If the LLM response contains text with brackets after the JSON array (e.g., [{"valid": 1}] Note: see [docs]), the regex captures [{"valid": 1}] Note: see [docs], causing JSON.parse to throw and parseSkillImprovements to silently return an empty array, discarding valid skill improvements.

-        const match = cleaned.match(/\[\s\S]*\]/);
+        const match = cleaned.match(/\[\s\S]*?\]/);

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/lib/aesop/workflows/exploration_workflow_executor.ts around line 1033:

The greedy regex `/\[\s\S]*\]/` on line 1033 matches from the first `[` to the **last** `]` in the entire response. If the LLM response contains text with brackets after the JSON array (e.g., `[{"valid": 1}] Note: see [docs]`), the regex captures `[{"valid": 1}] Note: see [docs]`, causing `JSON.parse` to throw and `parseSkillImprovements` to silently return an empty array, discarding valid skill improvements.

Evidence trail:
x-pack/platform/plugins/shared/evals/server/lib/aesop/workflows/exploration_workflow_executor.ts lines 1020-1082 at REVIEWED_COMMIT. Line 1032 contains the greedy regex: `const match = cleaned.match(/\[[\s\S]*\]/);`. Lines 1076-1082 show the catch block that logs error and returns empty array: `return [];`

[macroscopeapp]

🟡 Medium agents/agent_orchestrator.ts:46

When a valid response is emitted but the stream doesn't complete within timeoutMs, the timeout error triggers catchError to emit of(''). This causes lastValueFrom to return the empty string instead of the previously emitted valid response. For example: a conversationUpdate with content "Hello" passes all filters, but if the stream doesn't complete within 120 seconds, timeout fires, catchError returns of(''), and the function returns '' rather than "Hello". Consider using take(1) before timeout to capture the first valid response and complete immediately, or use timeout({ first: this.timeoutMs }) to only timeout on the first emission.

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/lib/aesop/agents/agent_orchestrator.ts around line 46:

When a valid response is emitted but the stream doesn't complete within `timeoutMs`, the `timeout` error triggers `catchError` to emit `of('')`. This causes `lastValueFrom` to return the empty string instead of the previously emitted valid response. For example: a `conversationUpdate` with content "Hello" passes all filters, but if the stream doesn't complete within 120 seconds, `timeout` fires, `catchError` returns `of('')`, and the function returns `''` rather than "Hello". Consider using `take(1)` before `timeout` to capture the first valid response and complete immediately, or use `timeout({ first: this.timeoutMs })` to only timeout on the first emission.

Evidence trail:
x-pack/platform/plugins/shared/evals/server/lib/aesop/agents/agent_orchestrator.ts lines 45-67 - The RxJS pipeline uses `lastValueFrom` with `timeout(this.timeoutMs)` followed by `catchError((err) => of(''))`. Per RxJS semantics, when timeout fires after a value was already emitted, catchError's `of('')` becomes the last emitted value that `lastValueFrom` returns, discarding the valid response.

[macroscopeapp]

🟢 Low workflows/exploration_workflow_executor.ts:1483

In parseLLMSkills, the expression Number(item.confidence) || 0.8 treats a valid confidence of 0 as falsy and replaces it with 0.8. When the LLM explicitly returns confidence: 0 to indicate high uncertainty, the code ignores this and inflates it to the default threshold. This prevents any LLM-generated skill from having a confidence score below 0.8 even when the LLM signals low confidence. Compare to parseSkillImprovements at line 1107 which correctly handles this with item.confidence != null && !isNaN(Number(item.confidence)).

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/lib/aesop/workflows/exploration_workflow_executor.ts around line 1483:

In `parseLLMSkills`, the expression `Number(item.confidence) || 0.8` treats a valid confidence of `0` as falsy and replaces it with `0.8`. When the LLM explicitly returns `confidence: 0` to indicate high uncertainty, the code ignores this and inflates it to the default threshold. This prevents any LLM-generated skill from having a confidence score below 0.8 even when the LLM signals low confidence. Compare to `parseSkillImprovements` at line 1107 which correctly handles this with `item.confidence != null && !isNaN(Number(item.confidence))`.

Evidence trail:
x-pack/platform/plugins/shared/evals/server/lib/aesop/workflows/exploration_workflow_executor.ts lines 1483 (showing `Number(item.confidence) || 0.8`) and lines 1107 (showing `item.confidence != null && !isNaN(Number(item.confidence)) ? Math.max(0, Math.min(1, Number(item.confidence))) : 0.7`). JavaScript semantics confirm that `0 || 0.8` evaluates to `0.8` because `0` is falsy.

[macroscopeapp]

🟡 Medium agents/agent_orchestrator.ts:157

The regex /\[[\s\S]*\]/ in parseSkillsFromResponse greedily matches from the first [ to the last ] in the string. When the agent response contains multiple bracket pairs like [{"skill":1}] See also [docs], the regex captures the entire span including [docs], causing JSON.parse to throw a syntax error and parseSkillsFromResponse to return [] even though valid JSON was present. The same pattern affects parseJsonFromResponse at line 171.

-      const match = cleaned.match(/\[[\s\S]*\]/);
+      const match = cleaned.match(/\[[\s\S]*?\]/);

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/lib/aesop/agents/agent_orchestrator.ts around line 157:

The regex `/\[[\s\S]*\]/` in `parseSkillsFromResponse` greedily matches from the first `[` to the last `]` in the string. When the agent response contains multiple bracket pairs like `[{"skill":1}] See also [docs]`, the regex captures the entire span including `[docs]`, causing `JSON.parse` to throw a syntax error and `parseSkillsFromResponse` to return `[]` even though valid JSON was present. The same pattern affects `parseJsonFromResponse` at line 171.

Evidence trail:
x-pack/platform/plugins/shared/evals/server/lib/aesop/agents/agent_orchestrator.ts lines 152-163 and 165-179 at REVIEWED_COMMIT. Line 157 contains `const match = cleaned.match(/\[\[\s\S\]*\]/);` - greedy regex. Line 171 contains `const match = cleaned.match(/\{[\s\S]*\}/);` - same greedy pattern for objects. Both use greedy `*` quantifier which matches from first bracket to last bracket in string, causing JSON.parse failure when multiple bracket pairs exist.

[macroscopeapp]

🟡 Medium aesop/run_skill_validation.ts:56

When esClient.get() is called for a non-existent skillId, Elasticsearch throws a ResponseError with status 404. This exception is caught by the outer catch block at line 279, which returns a 500 error. The if (!skill) check at lines 62-64 is never reached for missing documents because the exception is thrown before _source can be accessed. Users requesting a non-existent skill receive a 500 error instead of the intended 404.

🤖 Copy this AI Prompt to have your agent fix this:

In file x-pack/platform/plugins/shared/evals/server/routes/aesop/run_skill_validation.ts around line 56:

When `esClient.get()` is called for a non-existent `skillId`, Elasticsearch throws a `ResponseError` with status 404. This exception is caught by the outer `catch` block at line 279, which returns a 500 error. The `if (!skill)` check at lines 62-64 is never reached for missing documents because the exception is thrown before `_source` can be accessed. Users requesting a non-existent skill receive a 500 error instead of the intended 404.

Evidence trail:
x-pack/platform/plugins/shared/evals/server/routes/aesop/run_skill_validation.ts lines 56-64 (esClient.get without ignore option, followed by unreachable if (!skill) check), lines 262-282 (catch block returning 500). Elasticsearch JavaScript client documentation at https://www.elastic.co/docs/reference/elasticsearch/clients/javascript/ignore_examples confirms that 404 errors are thrown unless `ignore: [404]` is specified. Elasticsearch API documentation at https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-exists confirms 'If the document doesn't exist, the API returns 404 - Not Found'.