Skip to content
Merged
65 changes: 65 additions & 0 deletions docs/about-nemo-relay/concepts/events.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,71 @@ streams are blocked, or how exporter-specific semantic projection works. Those
decisions stay in the session, codec, plugin, guardrail, or exporter layer that
owns them.

### Automatic Skill-Load Marks

NeMo Relay automatically records an on-demand skill-load attempt when an
instrumented tool explicitly names a skill or requests a complete `SKILL.md`
read. The following payload excerpt shows the stable skill-load fields.
Standard fields from the [shared envelope](#shared-envelope), such as
`atof_version`, `uuid`, `parent_uuid`, and `timestamp`, are omitted.

```json
{
"kind": "mark",
"name": "skill.load",
"data": {"skill_name": "review"},
"metadata": {
"skill_load_source": "structured_read",
"tool_name": "read_file"
}
}
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Relay uses these `skill_load_source` values:

| Source | Detected Request |
| --- | --- |
| `skill_tool` | A first-class skill tool, such as `Skill` or `skill_view` |
| `structured_read` | A structured request for a whole-file read |
| `shell_read` | A standalone, complete `cat`, `bat`, `batcat`, or PowerShell `Get-Content` command |

For reader detections, the path must end in `SKILL.md`; the skill name is its
immediate parent directory. A nonzero structured offset, any structured limit
or range control, shell pipelines, redirections, compound commands, and
range-limited shell readers do not count. An explicit structured `offset` of
`0` still represents a complete read. Relay omits the full path and shell
command from the mark.

The mark has these lifecycle semantics:

- Relay emits it immediately after the tool-start event and before execution,
with the tool span as its parent.
- It records an eager load attempt and remains present if the tool later fails.
- A blocked call that never starts does not emit the mark.
- Relay emits each skill once per tool call. A later call that attempts the
same load emits another mark.

Claude Code `Skill`, Codex complete-reader shell calls, Hermes `skill_view`, and
tool calls made through the Rust, Python, Node.js, Go/FFI, LangChain, LangGraph,
and Deep Agents paths use this observed contract. The existing Deep Agents
`DeepAgents Skills Configured` mark remains a separate configuration summary;
it does not mean that a skill body was loaded.

Claude Code exposes slash-command expansion without saying whether the command
came from a skill or a legacy command. For a nonempty slash-command expansion,
Relay emits `skill.load.inferred` with `data.skill_name`,
`metadata.skill_load_source = "prompt_expansion"`, and
`metadata.inferred = true`. Relay also includes `metadata.command_source` when
Claude Code provides it. Keep inferred and observed counts separate.

Exporters handle the mark as follows:

- ATOF preserves the canonical mark event.
- ATIF omits marks because it models trajectory steps rather than independent
point-in-time events.
- OpenTelemetry and OpenInference apply their configured `mark_projection` to
parented marks.

## How Events Are Produced

Scope APIs emit `scope` start and end events and can also emit named `mark`
Expand Down
13 changes: 7 additions & 6 deletions docs/nemo-relay-cli/basic-usage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -264,9 +264,10 @@ calling NeMo Relay APIs.
that scope when it is still active.
- Tool pre-use starts a NeMo Relay tool span. Tool post-use, denial, or failure
closes it.
- Prompt, response, agent-thought, and legacy Hermes `pre_llm_call` /
`post_llm_call` hooks are retained as private correlation hints. They are not
emitted as NeMo Relay events.
- Generated `UserPromptSubmit`, `Stop`, and Hermes `pre_llm_call` /
`post_llm_call` hooks are retained as private correlation hints. The adapters
do the same when a custom or older integration delivers a response or
agent-thought hook. These hints are not emitted as NeMo Relay events.
- Compaction, notification, and unknown hook events become mark events under
the active session scope.
- Gateway requests emit NeMo Relay LLM start and end events under the active
Expand Down Expand Up @@ -304,10 +305,10 @@ also add `llm_correlation_source`, `llm_correlation_subagent_id`,

Generated hook bundles subscribe to the events needed for that mapping:

| Agent | LLM lifecycle and correlation hooks | Scope, tool, and mark hooks |
| Agent | LLM Lifecycle and Correlation Hooks | Scope, Tool, and Mark Hooks |
| --- | --- | --- |
| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Claude Code | `UserPromptSubmit`, `Stop` | `SessionStart`, `SessionEnd`, `UserPromptExpansion`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `PermissionRequest`, `Notification`, `PreCompact`, `PostCompact` |
| Codex | `UserPromptSubmit`, `Stop` | `SessionStart`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PermissionRequest`, `PreCompact`, `PostCompact` |
Comment thread
willkill07 marked this conversation as resolved.
| Hermes | `pre_api_request`, `post_api_request`, `api_request_error`, `pre_llm_call`, `post_llm_call` | `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `subagent_start`, `subagent_stop`, `pre_tool_call`, `post_tool_call` |

Hermes `pre_api_request`, `post_api_request`, and `api_request_error` hooks
Expand Down
26 changes: 17 additions & 9 deletions docs/nemo-relay-cli/claude-code.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -121,11 +121,19 @@ process.
## Captured Events

Generated Claude Code hooks include `SessionStart`, `SessionEnd`,
`SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`,
`PostToolUseFailure`, `Notification`, and `PreCompact` for scope, tool, and
mark events. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and
`Stop` are retained as private LLM correlation hints and are not emitted as
standalone NeMo Relay events.
`UserPromptSubmit`, `UserPromptExpansion`, `PreToolUse`, `PostToolUse`,
`PostToolUseFailure`, `PermissionRequest`, `SubagentStart`, `SubagentStop`,
`Notification`, `Stop`, `PreCompact`, and `PostCompact`. Both compaction hooks
emit canonical `compaction` marks. Relay normalizes the other hooks as scope,
prompt, tool, notification, subagent, or private LLM correlation events
according to the hook payload. `UserPromptExpansion` records nonempty
slash-command expansions as `skill.load.inferred`; Claude Code does not
identify whether an expansion came from a skill or a legacy custom command.
The normal `Skill` pre-tool hook emits an observed `skill.load` mark.

The wrapper requires Claude Code 2.1.121 or newer. Earlier versions do not
support every required hook event, and `nemo-relay doctor` reports the version
mismatch.

Tool hooks preserve canonical fields such as `tool_use_id`, `tool_name`,
`tool_input`, `error`, `duration_ms`, and `is_interrupt`. Subagent hooks use
Expand Down Expand Up @@ -207,7 +215,7 @@ plugin, such as through `nemo-relay claude` or `nemo-relay run --agent claude`.
The standalone gateway can still observe Anthropic LLM traffic, but it cannot
invent missing tool, prompt, compaction, notification, or subagent hooks.

`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are
used as private correlation and turn-boundary hints. They are not exported as
standalone user-visible mark events unless they also produce a scoped turn,
tool, LLM, or lifecycle observation.
`UserPromptSubmit` contributes prompt and LLM correlation data. `Stop` is a
private correlation and turn-boundary hint rather than a standalone
user-visible mark. The other generated hooks normalize to their corresponding
scope, tool, mark, notification, compaction, or subagent semantics.
17 changes: 11 additions & 6 deletions docs/nemo-relay-cli/codex.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -179,12 +179,17 @@ traffic to pass through the gateway.

## Captured Events

Generated Codex hooks include `SessionStart`, `SessionEnd`, `SubagentStart`,
`SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`,
`Notification`, and `PreCompact` for scope, tool, and mark events.
`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are
retained as private LLM correlation hints and are not emitted as standalone
NeMo Relay events.
The generated hook set includes `SessionStart`, `UserPromptSubmit`,
`PreToolUse`, `PostToolUse`, `PermissionRequest`, `SubagentStart`,
`SubagentStop`, `Stop`, `PreCompact`, and `PostCompact`. Both compaction hooks
emit canonical `compaction` marks. Relay requires the Codex app-server to
discover exactly one enabled, trusted handler for every generated event.
`PostToolUseFailure`, `Notification`, and `SessionEnd` are not in the Codex 0.143
plugin hook schema, so Relay does not generate undiscoverable handlers for them.
Relay maps delivered events to agent, turn, subagent, tool, and mark lifecycle
events; prompt and stop payloads also provide LLM correlation context. Complete
standalone `cat`, `bat`, `batcat`, or PowerShell `Get-Content` reads of
`SKILL.md` emit observed `skill.load` marks.

The transparent wrapper passes hook entries as Codex CLI config overrides and
sets `features.hooks=true` for that launched process. Persistent install writes
Expand Down
4 changes: 4 additions & 0 deletions docs/supported-integrations/about.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,9 @@ instructions.
Agents extra and capturing Deep Agents-specific marks, skills, subagents, and
human-in-the-loop lifecycle events.

Across LangChain, LangGraph, and Deep Agents, instrumented tool lifecycles also
inherit Relay's automatic `skill.load` detection when a tool requests a
complete `SKILL.md` read.

If you are building a new public-API framework integration,
use [Integrate into Frameworks](/integrate-into-frameworks/about) instead.
2 changes: 2 additions & 0 deletions docs/supported-integrations/deepagents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ It captures:
- LangGraph run scopes through callbacks.
- Human-in-the-loop interrupt and resume marks.
- Configured skills and subagent summaries at agent-run start.
- Automatic `skill.load` marks when a Deep Agents tool requests a complete
`SKILL.md` read; this is distinct from the configured-skills summary.
- In-process dictionary-style subagents with the same NeMo Relay middleware, so
their model and tool calls are captured when Deep Agents invokes them.

Expand Down
9 changes: 5 additions & 4 deletions integrations/coding-agents/claude-code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,11 @@ The bundle forwards `SessionStart`, `SessionEnd`, `UserPromptSubmit`,
`PreCompact`, and `PostCompact` as scope, tool, mark, or private LLM
correlation events.

The bundle requires Claude Code 2.1.116 or newer. Older versions do not have
`UserPromptExpansion` in their hook-event whitelist and reject the entire
plugin hook configuration, so no relay hooks load. `nemo-relay doctor` reports
this condition.
The bundle requires Claude Code 2.1.121 or newer. These versions provides the
`alwaysLoad` MCP startup barrier used to make Relay ready before session hooks
and accepts the complete generated hook schema. Installation fails before
changing host state when the version is too old, and `nemo-relay doctor`
reports the required upgrade.

Claude Code observability is turn-oriented. A multi-turn session can produce one
root `claude-code-turn` span or ATIF trajectory per user turn. That is expected
Expand Down
6 changes: 6 additions & 0 deletions skills/nemo-relay-plugin-observability/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ Use this model when explaining how capture and export relate:
- Event fields include semantic input/output through the ATOF `data` field,
typed profile data such as `model_name` and `tool_call_id`, and codec-provided
annotated LLM request/response data for in-process subscribers and exporters.
- First-class skill tools and the requests to read a complete `SKILL.md`
automatically emit `skill.load` marks under the tool span. The payload
contains only `skill_name`; metadata records the load source and tool name.
Partial reads do not count, and ambiguous slash-command expansions use the
separate `skill.load.inferred` name. The eager mark remains present if tool
execution later fails.

## Shared Lifecycle

Expand Down
Loading