diff --git a/docs/about-nemo-relay/concepts/events.mdx b/docs/about-nemo-relay/concepts/events.mdx index dddf27cd0..c0bb16261 100644 --- a/docs/about-nemo-relay/concepts/events.mdx +++ b/docs/about-nemo-relay/concepts/events.mdx @@ -109,6 +109,71 @@ streams are blocked, or how exporter-specific semantic projection works. Those decisions stay in the session, codec, plugin, guardrail, or exporter layer that owns them. +### Automatic Skill-Load Marks + +NeMo Relay automatically records an on-demand skill-load attempt when an +instrumented tool explicitly names a skill or requests a complete `SKILL.md` +read. The following payload excerpt shows the stable skill-load fields. +Standard fields from the [shared envelope](#shared-envelope), such as +`atof_version`, `uuid`, `parent_uuid`, and `timestamp`, are omitted. + +```json +{ + "kind": "mark", + "name": "skill.load", + "data": {"skill_name": "review"}, + "metadata": { + "skill_load_source": "structured_read", + "tool_name": "read_file" + } +} +``` + +Relay uses these `skill_load_source` values: + +| Source | Detected Request | +| --- | --- | +| `skill_tool` | A first-class skill tool, such as `Skill` or `skill_view` | +| `structured_read` | A structured request for a whole-file read | +| `shell_read` | A standalone, complete `cat`, `bat`, `batcat`, or PowerShell `Get-Content` command | + +For reader detections, the path must end in `SKILL.md`; the skill name is its +immediate parent directory. A nonzero structured offset, any structured limit +or range control, shell pipelines, redirections, compound commands, and +range-limited shell readers do not count. An explicit structured `offset` of +`0` still represents a complete read. Relay omits the full path and shell +command from the mark. + +The mark has these lifecycle semantics: + +- Relay emits it immediately after the tool-start event and before execution, + with the tool span as its parent. +- It records an eager load attempt and remains present if the tool later fails. +- A blocked call that never starts does not emit the mark. +- Relay emits each skill once per tool call. A later call that attempts the + same load emits another mark. + +Claude Code `Skill`, Codex complete-reader shell calls, Hermes `skill_view`, and +tool calls made through the Rust, Python, Node.js, Go/FFI, LangChain, LangGraph, +and Deep Agents paths use this observed contract. The existing Deep Agents +`DeepAgents Skills Configured` mark remains a separate configuration summary; +it does not mean that a skill body was loaded. + +Claude Code exposes slash-command expansion without saying whether the command +came from a skill or a legacy command. For a nonempty slash-command expansion, +Relay emits `skill.load.inferred` with `data.skill_name`, +`metadata.skill_load_source = "prompt_expansion"`, and +`metadata.inferred = true`. Relay also includes `metadata.command_source` when +Claude Code provides it. Keep inferred and observed counts separate. + +Exporters handle the mark as follows: + +- ATOF preserves the canonical mark event. +- ATIF omits marks because it models trajectory steps rather than independent + point-in-time events. +- OpenTelemetry and OpenInference apply their configured `mark_projection` to + parented marks. + ## How Events Are Produced Scope APIs emit `scope` start and end events and can also emit named `mark` diff --git a/docs/nemo-relay-cli/basic-usage.mdx b/docs/nemo-relay-cli/basic-usage.mdx index 674ebed8b..24c7d8c7b 100644 --- a/docs/nemo-relay-cli/basic-usage.mdx +++ b/docs/nemo-relay-cli/basic-usage.mdx @@ -264,9 +264,10 @@ calling NeMo Relay APIs. that scope when it is still active. - Tool pre-use starts a NeMo Relay tool span. Tool post-use, denial, or failure closes it. -- Prompt, response, agent-thought, and legacy Hermes `pre_llm_call` / - `post_llm_call` hooks are retained as private correlation hints. They are not - emitted as NeMo Relay events. +- Generated `UserPromptSubmit`, `Stop`, and Hermes `pre_llm_call` / + `post_llm_call` hooks are retained as private correlation hints. The adapters + do the same when a custom or older integration delivers a response or + agent-thought hook. These hints are not emitted as NeMo Relay events. - Compaction, notification, and unknown hook events become mark events under the active session scope. - Gateway requests emit NeMo Relay LLM start and end events under the active @@ -304,10 +305,10 @@ also add `llm_correlation_source`, `llm_correlation_subagent_id`, Generated hook bundles subscribe to the events needed for that mapping: -| Agent | LLM lifecycle and correlation hooks | Scope, tool, and mark hooks | +| Agent | LLM Lifecycle and Correlation Hooks | Scope, Tool, and Mark Hooks | | --- | --- | --- | -| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` | -| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` | +| Claude Code | `UserPromptSubmit`, `Stop` | `SessionStart`, `SessionEnd`, `UserPromptExpansion`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `PermissionRequest`, `Notification`, `PreCompact`, `PostCompact` | +| Codex | `UserPromptSubmit`, `Stop` | `SessionStart`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PermissionRequest`, `PreCompact`, `PostCompact` | | Hermes | `pre_api_request`, `post_api_request`, `api_request_error`, `pre_llm_call`, `post_llm_call` | `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `subagent_start`, `subagent_stop`, `pre_tool_call`, `post_tool_call` | Hermes `pre_api_request`, `post_api_request`, and `api_request_error` hooks diff --git a/docs/nemo-relay-cli/claude-code.mdx b/docs/nemo-relay-cli/claude-code.mdx index abc80726a..a8cc0b108 100644 --- a/docs/nemo-relay-cli/claude-code.mdx +++ b/docs/nemo-relay-cli/claude-code.mdx @@ -121,11 +121,19 @@ process. ## Captured Events Generated Claude Code hooks include `SessionStart`, `SessionEnd`, -`SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, -`PostToolUseFailure`, `Notification`, and `PreCompact` for scope, tool, and -mark events. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and -`Stop` are retained as private LLM correlation hints and are not emitted as -standalone NeMo Relay events. +`UserPromptSubmit`, `UserPromptExpansion`, `PreToolUse`, `PostToolUse`, +`PostToolUseFailure`, `PermissionRequest`, `SubagentStart`, `SubagentStop`, +`Notification`, `Stop`, `PreCompact`, and `PostCompact`. Both compaction hooks +emit canonical `compaction` marks. Relay normalizes the other hooks as scope, +prompt, tool, notification, subagent, or private LLM correlation events +according to the hook payload. `UserPromptExpansion` records nonempty +slash-command expansions as `skill.load.inferred`; Claude Code does not +identify whether an expansion came from a skill or a legacy custom command. +The normal `Skill` pre-tool hook emits an observed `skill.load` mark. + +The wrapper requires Claude Code 2.1.121 or newer. Earlier versions do not +support every required hook event, and `nemo-relay doctor` reports the version +mismatch. Tool hooks preserve canonical fields such as `tool_use_id`, `tool_name`, `tool_input`, `error`, `duration_ms`, and `is_interrupt`. Subagent hooks use @@ -207,7 +215,7 @@ plugin, such as through `nemo-relay claude` or `nemo-relay run --agent claude`. The standalone gateway can still observe Anthropic LLM traffic, but it cannot invent missing tool, prompt, compaction, notification, or subagent hooks. -`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are -used as private correlation and turn-boundary hints. They are not exported as -standalone user-visible mark events unless they also produce a scoped turn, -tool, LLM, or lifecycle observation. +`UserPromptSubmit` contributes prompt and LLM correlation data. `Stop` is a +private correlation and turn-boundary hint rather than a standalone +user-visible mark. The other generated hooks normalize to their corresponding +scope, tool, mark, notification, compaction, or subagent semantics. diff --git a/docs/nemo-relay-cli/codex.mdx b/docs/nemo-relay-cli/codex.mdx index 2446e1ff4..6f7b713a6 100644 --- a/docs/nemo-relay-cli/codex.mdx +++ b/docs/nemo-relay-cli/codex.mdx @@ -179,12 +179,17 @@ traffic to pass through the gateway. ## Captured Events -Generated Codex hooks include `SessionStart`, `SessionEnd`, `SubagentStart`, -`SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, -`Notification`, and `PreCompact` for scope, tool, and mark events. -`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are -retained as private LLM correlation hints and are not emitted as standalone -NeMo Relay events. +The generated hook set includes `SessionStart`, `UserPromptSubmit`, +`PreToolUse`, `PostToolUse`, `PermissionRequest`, `SubagentStart`, +`SubagentStop`, `Stop`, `PreCompact`, and `PostCompact`. Both compaction hooks +emit canonical `compaction` marks. Relay requires the Codex app-server to +discover exactly one enabled, trusted handler for every generated event. +`PostToolUseFailure`, `Notification`, and `SessionEnd` are not in the Codex 0.143 +plugin hook schema, so Relay does not generate undiscoverable handlers for them. +Relay maps delivered events to agent, turn, subagent, tool, and mark lifecycle +events; prompt and stop payloads also provide LLM correlation context. Complete +standalone `cat`, `bat`, `batcat`, or PowerShell `Get-Content` reads of +`SKILL.md` emit observed `skill.load` marks. The transparent wrapper passes hook entries as Codex CLI config overrides and sets `features.hooks=true` for that launched process. Persistent install writes diff --git a/docs/supported-integrations/about.mdx b/docs/supported-integrations/about.mdx index d71e9981b..3d193f12a 100644 --- a/docs/supported-integrations/about.mdx +++ b/docs/supported-integrations/about.mdx @@ -40,5 +40,9 @@ instructions. Agents extra and capturing Deep Agents-specific marks, skills, subagents, and human-in-the-loop lifecycle events. +Across LangChain, LangGraph, and Deep Agents, instrumented tool lifecycles also +inherit Relay's automatic `skill.load` detection when a tool requests a +complete `SKILL.md` read. + If you are building a new public-API framework integration, use [Integrate into Frameworks](/integrate-into-frameworks/about) instead. diff --git a/docs/supported-integrations/deepagents.mdx b/docs/supported-integrations/deepagents.mdx index 39235c802..1dba41efe 100644 --- a/docs/supported-integrations/deepagents.mdx +++ b/docs/supported-integrations/deepagents.mdx @@ -109,6 +109,8 @@ It captures: - LangGraph run scopes through callbacks. - Human-in-the-loop interrupt and resume marks. - Configured skills and subagent summaries at agent-run start. +- Automatic `skill.load` marks when a Deep Agents tool requests a complete + `SKILL.md` read; this is distinct from the configured-skills summary. - In-process dictionary-style subagents with the same NeMo Relay middleware, so their model and tool calls are captured when Deep Agents invokes them. diff --git a/integrations/coding-agents/claude-code/README.md b/integrations/coding-agents/claude-code/README.md index 1fef347de..d293a27c6 100644 --- a/integrations/coding-agents/claude-code/README.md +++ b/integrations/coding-agents/claude-code/README.md @@ -26,10 +26,11 @@ The bundle forwards `SessionStart`, `SessionEnd`, `UserPromptSubmit`, `PreCompact`, and `PostCompact` as scope, tool, mark, or private LLM correlation events. -The bundle requires Claude Code 2.1.116 or newer. Older versions do not have -`UserPromptExpansion` in their hook-event whitelist and reject the entire -plugin hook configuration, so no relay hooks load. `nemo-relay doctor` reports -this condition. +The bundle requires Claude Code 2.1.121 or newer. These versions provides the +`alwaysLoad` MCP startup barrier used to make Relay ready before session hooks +and accepts the complete generated hook schema. Installation fails before +changing host state when the version is too old, and `nemo-relay doctor` +reports the required upgrade. Claude Code observability is turn-oriented. A multi-turn session can produce one root `claude-code-turn` span or ATIF trajectory per user turn. That is expected diff --git a/skills/nemo-relay-plugin-observability/SKILL.md b/skills/nemo-relay-plugin-observability/SKILL.md index 9a32019fa..c8fe82523 100644 --- a/skills/nemo-relay-plugin-observability/SKILL.md +++ b/skills/nemo-relay-plugin-observability/SKILL.md @@ -56,6 +56,12 @@ Use this model when explaining how capture and export relate: - Event fields include semantic input/output through the ATOF `data` field, typed profile data such as `model_name` and `tool_call_id`, and codec-provided annotated LLM request/response data for in-process subscribers and exporters. +- First-class skill tools and the requests to read a complete `SKILL.md` + automatically emit `skill.load` marks under the tool span. The payload + contains only `skill_name`; metadata records the load source and tool name. + Partial reads do not count, and ambiguous slash-command expansions use the + separate `skill.load.inferred` name. The eager mark remains present if tool + execution later fails. ## Shared Lifecycle