feat(evals): AESOP spike + eval platform for Agent Builder skills#261057

Draft

patrykkopycinski wants to merge 84 commits into

elastic:mainfrom

patrykkopycinski:worktree-skill-eval-platform

Commits on Mar 31, 2026

feat: create @kbn/llm-batch-processing package structure

patrykkopycinski
and
claude
committed
feat(aesop): add remaining aesop implementation files

patrykkopycinski
and
claude
committed
feat(evals): integrate autonomous skill discovery routes and agents

patrykkopycinski
and
claude
committed
feat(llm-batch): restore hierarchical batch processing package

patrykkopycinski
and
claude
committed
feat(evals): add skill review workflow UI with execution details

patrykkopycinski
and
claude
committed
feat(evals): implement real-time workflow progress tracking

patrykkopycinski
and
claude
committed
feat(evals): add incremental exploration with intelligent change detection

patrykkopycinski
and
claude
committed
test(evals): add comprehensive validation and benchmarking test suites

patrykkopycinski
and
claude
committed
docs(evals): add production implementation planning and architecture decisions

patrykkopycinski
and
claude
committed
feat(evals): implement active feedback learning loop for continuous improvement

patrykkopycinski
and
claude
committed
feat(evals): add performance monitoring dashboard with 8 operational panels

patrykkopycinski
and
claude
committed
feat(evals): implement security hardening with 4-layer defense and rate limiting

patrykkopycinski
and
claude
committed
test(evals): expand test coverage to 90%+ with UI and integration tests

patrykkopycinski
and
claude
committed
test(evals): add Scout E2E tests and workflow error recovery system

patrykkopycinski
and
claude
committed
feat(evals): add APM instrumentation and production alerting

patrykkopycinski
and
claude
committed
docs(evals): add comprehensive production documentation and operational guides

patrykkopycinski
and
claude
committed
docs(evals): mark production implementation 100% complete

patrykkopycinski
and
claude
committed
feat(evals): add Docker and dev container setup for hypothesis validation

patrykkopycinski
and
claude
committed
feat(evals): add fully automated hypothesis validation with complete test data

patrykkopycinski
and
claude
committed
feat(evals): AESOP spike - autonomous skill discovery, LLM validation, and Agent Builder deployment

patrykkopycinski
and
claude
committed
chore: remove generated spike documentation files

patrykkopycinski
and
claude
committed
fix(evals): address code quality issues in AESOP spike

patrykkopycinski
and
claude
committed
feat(evals): add skill improvement proposals, source indices, and derived-from filtering

patrykkopycinski
and
claude
committed
fix(evals): address PR feedback and clean up non-AESOP files

patrykkopycinski
and
claude
committed
feat(evals): add convergence validation loop for AESOP skill auto-improvement

patrykkopycinski
and
claude
committed
feat(evals): add persistent rate limiter backed by Elasticsearch

patrykkopycinski
and
claude
committed
feat(evals): add deep Agent Builder conversation analysis for AESOP

patrykkopycinski
and
claude
committed
feat(evals): add skill deduplication for AESOP exploration workflow

patrykkopycinski
and
claude
committed
fix(evals): production readiness fixes for AESOP spike

patrykkopycinski
and
claude
committed
fix(evals): add accessibility aria-labels and error boundary to AESOP UI

patrykkopycinski
and
claude
committed
fix(evals): add ProposedSkillDocument type and replace `as any` casts in AESOP route handlers

patrykkopycinski
and
claude
committed
test(evals): replace stub E2E tests with real route handler unit tests for AESOP

patrykkopycinski
and
claude
committed
fix(evals): address macroscope review findings in AESOP spike

patrykkopycinski
and
claude
committed
fix(evals): address 6 macroscope review comments in AESOP spike

patrykkopycinski
and
claude
committed
feat(evals): define AESOP agent configurations for Agent Builder orchestration

patrykkopycinski
and
claude
committed
feat(evals): add agent lifecycle manager for AESOP Agent Builder agents

patrykkopycinski
and
claude
committed
feat(evals): add Agent Builder orchestration for AESOP with backwards-compatible fallback

patrykkopycinski
and
claude
committed
fix(evals): fix duration_ms bug and add agent-based improvement route

patrykkopycinski
and
claude
committed
fix(evals): add agent progress tracking, conversation persistence, fix re-serialization

patrykkopycinski
and
claude
committed

Commits on Apr 2, 2026

feat(evals): AESOP spike + eval platform with suites, evaluators, monitoring, comparison

patrykkopycinski
and
claude
committed

Commits on Apr 3, 2026

fix(evals): address all macroscope review comments

patrykkopycinski
and
claude
committed

Commits on Apr 5, 2026

Commits on Apr 6, 2026

Commits on Apr 21, 2026