Add prebuilt skills, osquery skill, and improve agent tool error handling#254611
Closed
patrykkopycinski wants to merge 70 commits into
Closed
Add prebuilt skills, osquery skill, and improve agent tool error handling#254611patrykkopycinski wants to merge 70 commits into
patrykkopycinski wants to merge 70 commits into
Conversation
…execution, and demo skills Implements planning mode for the Agent Builder, allowing agents to create structured execution plans before acting. Includes a new agent mode selector, plan panel sidebar with real-time progress tracking, mode suggestion banner, planning tools (create/update plan, list tools, suggest mode), plan execution instructions for the default agent, and four stubbed security skills for demo purposes. Fixes several UX issues: stale plan state across conversations, draft-to-ready status promotion, approve-and-execute flow with correct agent mode, local execution for real-time streaming during plan execution, and improved JSON readability in tool response flyouts.
…unner - Remove unused conversationTimestamp variable in run_planning_agent.ts - Fix BuiltinToolDefinition array type to use Array<BuiltinToolDefinition<any>> to avoid Zod schema variance issues with specific tool parameter types
The button now shows for draft plans (pre-approval) from planning source, not for ready plans (already approved). Added test for all-completed case.
Adds support for user-created skills in the Agent Builder plugin: - Server: CRUD API routes for skills (create, read, update, delete) with versioned endpoints, composite skill registry (built-in + persisted), Elasticsearch-backed persistence, and skill selection on agents. - Client: Skills management UI with list page, create/edit forms, context menus, delete confirmation, and skills tab on the agent configuration form. - Common: Shared types, validation schemas, and skill selection utilities. - Built-in: Platform-level "data-exploration" skill registered at startup. - Tests: Scout API and UI e2e tests covering CRUD flows, built-in skill protection, and skills list/form interactions.
- Add tsconfig.json for Scout API and UI test directories - Fix TypeScript errors: add missing `id` property on mock providers, remove unused variable, add missing `getRegistry` mock - Update OAS validation baseline for new skills API routes - Remove openspec workflow artifacts from git tracking
- Refactor API tests to use apiClient fixture instead of kbnClient.request per scout_require_api_client_in_api_test ESLint rule - Fix hook ordering in UI test (beforeEach before afterAll) - Fix TS1109 parsing error in skills_on_agent_form.spec.ts
- Add 5-tool limit validation on skill create/update - Gate all skill CRUD routes behind experimentalFeatures flag - Update runtime to resolve user-created skills via CompositeSkillRegistry - Fix Scout API tests: add SAML auth, required headers (kbn-xsrf, x-elastic-internal-origin), and enable experimental features flag - Fix Scout UI tests: enable experimental features flag for API routes - Add unit tests for 5-tool limit validation
…ed layer - Fold CompositeSkillRegistry, BuiltinSkillStore, and SkillProvider into a single SkillRegistry within skill_service.ts - Remove deprecated getSkillDefinition/listSkills from SkillServiceStart - Inline persisted provider logic into skill_service.ts and flatten persisted/client/ to skills/client/ - Remove unnecessary section divider comments across changed files - Add comprehensive tests for skill_service.ts covering both SkillService and SkillRegistry
…6 demo) Brings over endpoint tooling from feature/skills-all: - GCP Fleet VM provisioner with Tailscale connectivity - REF7707 Caldera lab emulation (DNS, download, execution, persistence) - RSA 2026 demo provisioner (browser history, detection rules, workflows) - Caldera MITRE rule validation framework - Agent skills demo runner - Supporting utilities (enable browsers, remote access, install browsers) - Common endpoint services modifications for GCP VM support
Brings over skill infrastructure from feature/skills-all: - Security solution prebuilt skills (SkillDefinition type migration) - Forensics analytics skill for REF7707 demo - Alert triage, entity analytics, security labs search skills - Security agent builder tools (alerts, cases, detection rules, etc.) - Osquery live query service extraction with tests - Observability skills (alerts, APM, logs, metrics, SLO, synthetics) - Platform skills (ESQL, cases, data views, workflows, etc.) - Agent builder tracing infrastructure - Inference tracing (baggage, elasticsearch exporter, span processors) - kbn-langchain tracer updates - kbn-evals tool usage evaluator
…s, evals) Brings over supporting infrastructure from feature/skills-all: - Elastic assistant: alert grouping service, batch/incremental attack discovery, workflow steps, alert deduplication, route updates - Cases: attack discovery tab, user actions list updates - Data sources: GitHub issue aggregator - Security solution evals: chat client, evaluate dataset, HTML reporter - ai-infra-common package (index settings utilities) - Security solution experimental features - Deep agent middleware type definitions - Root package.json, tsconfig.base.json, and yarn.lock updates
Remove inference tracing, langchain tracer, and agent builder tracer changes that were brought over from feature/skills-all. These are not needed for the prebuilt skills demo.
…ilt-skills # Conflicts: # .buildkite/scout_ci_config.yml # src/platform/packages/private/kbn-validate-oas/src/oas_error_baseline.json # x-pack/platform/packages/shared/agent-builder/agent-builder-common/agents/definition.ts # x-pack/platform/packages/shared/kbn-evals/index.ts # x-pack/platform/packages/shared/kbn-evals/src/utils/score_repository.ts # x-pack/platform/plugins/shared/agent_builder/public/application/components/agents/edit/agent_form_validation.ts # x-pack/platform/plugins/shared/agent_builder/public/application/components/conversations/conversation_header/more_actions_button.tsx # x-pack/platform/plugins/shared/agent_builder/public/application/hooks/agents/use_agent_edit.ts # x-pack/platform/plugins/shared/agent_builder/public/plugin.tsx # x-pack/platform/plugins/shared/agent_builder/server/routes/agents.ts # x-pack/platform/plugins/shared/agent_builder/server/services/agents/persisted/client/converters.ts # x-pack/platform/plugins/shared/agent_builder/server/services/agents/persisted/client/storage.ts # x-pack/platform/plugins/shared/agent_builder/server/services/runner/store/utils/load_skill.ts # x-pack/platform/plugins/shared/agent_builder/server/services/skills/skill_registry.test.ts # x-pack/platform/plugins/shared/agent_builder/server/services/skills/skill_registry.ts # x-pack/platform/plugins/shared/agent_builder/server/services/skills/types.ts # x-pack/platform/plugins/shared/agent_builder/server/test_utils/runner.ts # x-pack/solutions/security/plugins/security_solution/common/experimental_features.ts # x-pack/solutions/security/plugins/security_solution/scripts/endpoint/common/vm_services.ts # x-pack/solutions/security/plugins/security_solution/server/agent_builder/skills/register_skills.ts # x-pack/solutions/security/plugins/security_solution/server/plugin.ts
…ling - Add security osquery skill with live query, agents, status, saved queries, packs, and results tools - Add osquery onechat skills (status, packs, saved queries, live query) - Add dashboard agent, fleet, and ML agent builder skills - Register new security tools in allow list (detection_rules, cases, exception_lists, timelines) - Gracefully handle tool-not-found and invalid-params errors for agent calls - Fix async skills list call in select_tools
Contributor
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
patrykkopycinski
added
ci:cloud-deploy
Contributor
Author
|
/ci |
…dling - Add get_table_schema tool to discover columns via PRAGMA before querying custom tables - Fix get_results polling to detect errors immediately instead of waiting for timeout - Update skill instructions to require schema discovery for custom/Elastic tables
Contributor
Author
|
/ci |
Contributor
Author
|
/ci |
…tion_tests/ci_checks
…cinski/kibana into agent-builder-prebuilt-skills
Contributor
Author
|
/ci |
Contributor
Author
|
/ci |
Contributor
Author
|
/ci |
1 similar comment
Contributor
Author
|
/ci |
…ills - Normalize MCP tool schemas to ZodObject to prevent runtime errors with non-object schemas - Add resolveSkillSelection and listSkillDefinitions to SkillRegistry interface - Remove unused skillMiddleware and skill_discovery utilities - Revamp alert_triage_skill with comprehensive triage workflow including VirusTotal enrichment and forensic escalation - Enhance forensics_analytics_skill with VirusTotal correlation, REF7707 reference, and cross-endpoint browser history sweeps - Add trigger-alert step to RSA 2026 demo provisioner for single-endpoint headless Chrome traffic generation
Contributor
Author
|
/ci |
1 similar comment
Contributor
Author
|
/ci |
…o-result fetching - Add table listing/search to get_table_schema (search by name, filter by platform) so the LLM discovers correct table names instead of guessing - Add agentAll and agentPolicyIds params to run_live_query for simpler cross-endpoint sweeps - Increase poll timeout to 10min and schema timeout to 2min for large fleet queries - Add "Never Pause" instructions so the LLM fetches results immediately without asking - Update alert triage and forensics skills to use agentAll: true for sweeps - Remove unused skill_aware.ts and skill_aware_graph.ts (dead code)
Contributor
Author
|
/ci |
Contributor
Author
|
/ci |
… and agent count - Query all response indexes (data stream, component template, fleet legacy) to fix agents_responded always showing 0 - Use parent action_id for response tracking (not per-query action_id) - Extract agent.id from ECS nested format instead of flat agent_id - Add unique_agents_with_results to surface affected host count - Cap online_agent_count to never exceed targeted agent_count - Fix non-null assertion eslint error
Contributor
Author
|
/ci |
…ilt-skills # Conflicts: # x-pack/platform/plugins/shared/agent_builder/public/plugin.tsx # x-pack/platform/plugins/shared/agent_builder/server/services/runner/run_tool.ts # x-pack/platform/plugins/shared/agent_builder/server/services/runner/store/utils/load_skill.ts
Contributor
Author
|
/ci |
Contributor
⏳ Build in-progress, with failures
Failed CI Steps
Test Failures
History
|
Add endpoint response actions tool/skill, workflow CRUD tools, detection rules patch operation, bulk criticality and risk engine management inline tools to achieve 1:1 coverage with SOC agent skills.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
askUser: 'once'settingProduction-Readiness Checklist — Agent Skills Ecosystem
Generated against [Epic] Creation of the Agent Skills Ecosystem for Elastic Security.
Narrative role: Specialization pillar ("focused skills for discrete tasks") + the composable skill library the vision describes.
Must-do before this can ship
agent.callgraceful-error change@kbn/evalssuite (today there is only one PR-wide test plan)aiSocAgents). Today osquery tools are added toallow_lists.tsungated — that violates the dark-launch / out-of-band delivery requirementFollow-ups (post-merge)