Migrate infosec agentic alert triage workflows to Agent Builder skills#258397
Closed
patrykkopycinski wants to merge 4 commits into
Closed
Migrate infosec agentic alert triage workflows to Agent Builder skills#258397patrykkopycinski wants to merge 4 commits into
patrykkopycinski wants to merge 4 commits into
Conversation
Migrates the agentic-alert-triage workflows from elastic/infosec into user-created Agent Builder skills, with framework fixes to unblock @kbn/evals execution. ## Migration artifacts - 14 skill definitions (L1 Investigation, L1 Triage, Threshold Context, 9 L2 domain skills, L3 Review, Orchestrator) as JSON files - Feature-flagged workflow YAML with `use_orchestrator` toggle - Patched workflow YAML integrating the orchestrator - Import script, checklist, and gap RFC ## @kbn/evals framework fixes - `.text` file require hook for Playwright workers — packages like @kbn/evals import `.text` files (LLM prompt templates) that crash workers without a hook. Added `dot_text_setup.ts` in @kbn/scout mirroring the peggy_setup pattern. - `NO_COLOR`/`FORCE_COLOR` warning ignore — Playwright sets FORCE_COLOR while NO_COLOR may be present, triggering exit_on_warning to kill the worker. Added to the ignore list. ## Eval suite - 5 test cases for L1 Investigation, L1 Triage (benign/suspicious), and Orchestrator (full flow/benign) using @kbn/evals + EDOT tracing
Contributor
Author
|
/ci |
Contributor
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
… spec - Update 10 L2/L3 agent IDs in flagged YAML to match migrated skill IDs (e.g. l2_macos_forensics_agent → agentic-alert-triage-l2-macos) - Replace hardcoded /tmp response file in import_skills.sh with mktemp + trap cleanup - Fix shell injection risk: pass filename via sys.argv instead of string interpolation - Extract requireAgentId() helper in triage.spec.ts to reduce repeated guard blocks - Wrap agent creation in try/catch with explicit error logging - Update import_checklist.md to reflect that flagged YAML now uses migrated skill IDs
Contributor
Author
|
/ci |
Contributor
💔 Build Failed
Failed CI StepsMetrics [docs]
History
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the agentic-alert-triage workflows from
elastic/infosecinto user-created Agent Builder skills, with framework fixes to unblock@kbn/evalsexecution.Migration artifacts
use_orchestratortoggle for gradual rolloutimport_skills.sh) for automated skill import/update@kbn/evalsframework fixesTwo bugs prevented Playwright workers from executing
@kbn/evalstest suites:.textfile imports crash workers —@kbn/evalsevaluators import.textfiles (LLM prompt templates) that need a require hook. Addeddot_text_setup.tsin@kbn/scout(mirroring the existingpeggy_setup.tspattern).NO_COLORenv warning kills workers — Playwright setsFORCE_COLORwhileNO_COLORmay also be present. Node emits a warning, andexit_on_warning.jsterminates the process. Added to the ignore list.Eval suite
5 test cases using
@kbn/evalswith EDOT tracing:Test plan
@kbn/evalssuites no longer crash with.textSyntaxError orNO_COLORwarningnode scripts/evals run --suite agent-builder --grep triageruns all 5 testsbash import_skills.shagainst a running Kibana with Agent Builder enabled/api/agent_builder/converseproduces expected outputMade with Cursor
Production-Readiness Checklist — Agent Skills Ecosystem
Generated against [Epic] Creation of the Agent Skills Ecosystem for Elastic Security.
Narrative role: Concrete proof the Agent Builder skill contract can carry real SOC workflows. Migrates the internal agentic-alert-triage suite, so it's also the reference for everyone else migrating from bespoke pipelines.
Must-do before this can ship
@kbn/evalsframework fixes (.textrequire hook +NO_COLORworker crash) into a tiny standalone PR and merge it this week — every eval-using PR in the program depends on itimport_skills.shwith the #255890 CLI so the same skill can be shipped out-of-bandcases.addAlerts,cases.findCases,timeline.create,response_actions, etc.) and link them to this epicFollow-ups (post-merge)
use_orchestratortoggle as a Workflows template (out-of-band delivery) instead of YAML in the repo