Skip to content
Merged
65 changes: 65 additions & 0 deletions docs/about-nemo-relay/concepts/events.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,71 @@ streams are blocked, or how exporter-specific semantic projection works. Those
decisions stay in the session, codec, plugin, guardrail, or exporter layer that
owns them.

### Automatic Skill-Load Marks

NeMo Relay automatically records an on-demand skill-load attempt when an
instrumented tool explicitly names a skill or requests a complete `SKILL.md`
read. The following payload excerpt shows the stable skill-load fields.
Standard fields from the [shared envelope](#shared-envelope), such as
`atof_version`, `uuid`, `parent_uuid`, and `timestamp`, are omitted.

```json
{
"kind": "mark",
"name": "skill.load",
"data": {"skill_name": "review"},
"metadata": {
"skill_load_source": "structured_read",
"tool_name": "read_file"
}
}
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Relay uses these `skill_load_source` values:

| Source | Detected Request |
| --- | --- |
| `skill_tool` | A first-class skill tool, such as `Skill` or `skill_view` |
| `structured_read` | A structured request for a whole-file read |
| `shell_read` | A standalone, complete `cat`, `bat`, `batcat`, or PowerShell `Get-Content` command |

For reader detections, the path must end in `SKILL.md`; the skill name is its
immediate parent directory. A nonzero structured offset, any structured limit
or range control, shell pipelines, redirections, compound commands, and
range-limited shell readers do not count. An explicit structured `offset` of
`0` still represents a complete read. Relay omits the full path and shell
command from the mark.

The mark has these lifecycle semantics:

- Relay emits it immediately after the tool-start event and before execution,
with the tool span as its parent.
- It records an eager load attempt and remains present if the tool later fails.
- A blocked call that never starts does not emit the mark.
- Relay emits each skill once per tool call. A later call that attempts the
same load emits another mark.

Claude Code `Skill`, Codex complete-reader shell calls, Hermes `skill_view`, and
tool calls made through the Rust, Python, Node.js, Go/FFI, LangChain, LangGraph,
and Deep Agents paths use this observed contract. The existing Deep Agents
`DeepAgents Skills Configured` mark remains a separate configuration summary;
it does not mean that a skill body was loaded.

Claude Code exposes slash-command expansion without saying whether the command
came from a skill or a legacy command. For a nonempty slash-command expansion,
Relay emits `skill.load.inferred` with `data.skill_name`,
`metadata.skill_load_source = "prompt_expansion"`, and
`metadata.inferred = true`. Relay also includes `metadata.command_source` when
Claude Code provides it. Keep inferred and observed counts separate.

Exporters handle the mark as follows:

- ATOF preserves the canonical mark event.
- ATIF omits marks because it models trajectory steps rather than independent
point-in-time events.
- OpenTelemetry and OpenInference apply their configured `mark_projection` to
parented marks.

## How Events Are Produced

Scope APIs emit `scope` start and end events and can also emit named `mark`
Expand Down
6 changes: 3 additions & 3 deletions docs/nemo-relay-cli/basic-usage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -304,10 +304,10 @@ also add `llm_correlation_source`, `llm_correlation_subagent_id`,

Generated hook bundles subscribe to the events needed for that mapping:

| Agent | LLM lifecycle and correlation hooks | Scope, tool, and mark hooks |
| Agent | LLM Lifecycle and Correlation Hooks | Scope, Tool, and Mark Hooks |
| --- | --- | --- |
| Claude Code | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Codex | `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Notification`, `PreCompact` |
| Claude Code | `UserPromptSubmit`, `Stop` | `SessionStart`, `SessionEnd`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `PermissionRequest`, `Notification`, `PreCompact`, `PostCompact`, `UserPromptExpansion` |
| Codex | `UserPromptSubmit`, `Stop` | `SessionStart`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PermissionRequest`, `PreCompact`, `PostCompact` |
Comment thread
willkill07 marked this conversation as resolved.
| Hermes | `pre_api_request`, `post_api_request`, `api_request_error`, `pre_llm_call`, `post_llm_call` | `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `subagent_start`, `subagent_stop`, `pre_tool_call`, `post_tool_call` |

Hermes `pre_api_request`, `post_api_request`, and `api_request_error` hooks
Expand Down
15 changes: 11 additions & 4 deletions docs/nemo-relay-cli/claude-code.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,17 @@ process.

Generated Claude Code hooks include `SessionStart`, `SessionEnd`,
`SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`,
`PostToolUseFailure`, `Notification`, and `PreCompact` for scope, tool, and
mark events. `UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and
`Stop` are retained as private LLM correlation hints and are not emitted as
standalone NeMo Relay events.
`PostToolUseFailure`, `PermissionRequest`, `Notification`, `PreCompact`, and
`PostCompact` for scope, tool, and mark events. `UserPromptExpansion` records
nonempty slash-command expansions as `skill.load.inferred`; Claude Code does
not identify whether an expansion came from a skill or a legacy custom command.
The normal `Skill` pre-tool hook emits an observed `skill.load` mark.
`UserPromptSubmit` and `Stop` are retained as private LLM correlation hints and
are not emitted as standalone NeMo Relay events.

The wrapper requires Claude Code 2.1.121 or newer. Earlier versions do not
support every required hook event, and `nemo-relay doctor` reports the version
mismatch.

Tool hooks preserve canonical fields such as `tool_use_id`, `tool_name`,
`tool_input`, `error`, `duration_ms`, and `is_interrupt`. Subagent hooks use
Expand Down
12 changes: 6 additions & 6 deletions docs/nemo-relay-cli/codex.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -179,12 +179,12 @@ traffic to pass through the gateway.

## Captured Events

Generated Codex hooks include `SessionStart`, `SessionEnd`, `SubagentStart`,
`SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`,
`Notification`, and `PreCompact` for scope, tool, and mark events.
`UserPromptSubmit`, `AfterAgentResponse`, `AfterAgentThought`, and `Stop` are
retained as private LLM correlation hints and are not emitted as standalone
NeMo Relay events.
Generated Codex hooks include `SessionStart`, `SubagentStart`, `SubagentStop`,
`PreToolUse`, `PostToolUse`, `PermissionRequest`, `PreCompact`, and `PostCompact`
for scope, tool, and mark events. Complete standalone `cat`,
`bat`, `batcat`, or PowerShell `Get-Content` reads of `SKILL.md` emit observed
`skill.load` marks. `UserPromptSubmit` and `Stop` are retained as private LLM
correlation hints and are not emitted as standalone NeMo Relay events.

The transparent wrapper passes hook entries as Codex CLI config overrides and
sets `features.hooks=true` for that launched process. Persistent install writes
Expand Down
4 changes: 4 additions & 0 deletions docs/nemo-relay-cli/hermes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,10 @@ The generated Hermes hooks cover `on_session_start`, `on_session_end`,
`pre_api_request`, `post_api_request`, `api_request_error`, `pre_tool_call`,
`post_tool_call`, `subagent_start`, and `subagent_stop`.

Hermes `skill_view` pre-tool calls emit observed `skill.load` marks. The mark
stores the skill name in `data.skill_name` and the source and tool name in
metadata. It does not retain skill paths.

The API hooks are the main Hermes LLM lifecycle path for NeMo Relay. The legacy
LLM hooks remain installed because they can still provide useful private hints,
but they are not treated as equal peers to the API hooks in the observability
Expand Down
4 changes: 4 additions & 0 deletions docs/supported-integrations/about.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,9 @@ instructions.
Agents extra and capturing Deep Agents-specific marks, skills, subagents, and
human-in-the-loop lifecycle events.

Across LangChain, LangGraph, and Deep Agents, instrumented tool lifecycles also
inherit Relay's automatic `skill.load` detection when a tool requests a
complete `SKILL.md` read.

If you are building a new public-API framework integration,
use [Integrate into Frameworks](/integrate-into-frameworks/about) instead.
2 changes: 2 additions & 0 deletions docs/supported-integrations/deepagents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ It captures:
- LangGraph run scopes through callbacks.
- Human-in-the-loop interrupt and resume marks.
- Configured skills and subagent summaries at agent-run start.
- Automatic `skill.load` marks when a Deep Agents tool requests a complete
`SKILL.md` read; this is distinct from the configured-skills summary.
- In-process dictionary-style subagents with the same NeMo Relay middleware, so
their model and tool calls are captured when Deep Agents invokes them.

Expand Down
7 changes: 3 additions & 4 deletions integrations/coding-agents/claude-code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,9 @@ The bundle forwards `SessionStart`, `SessionEnd`, `UserPromptSubmit`,
`PreCompact`, and `PostCompact` as scope, tool, mark, or private LLM
correlation events.

The bundle requires Claude Code 2.1.116 or newer. Older versions do not have
`UserPromptExpansion` in their hook-event whitelist and reject the entire
plugin hook configuration, so no relay hooks load. `nemo-relay doctor` reports
this condition.
The bundle requires Claude Code 2.1.121 or newer. Older versions do not support
every required hook event and can reject the entire plugin hook configuration,
so no relay hooks load. `nemo-relay doctor` reports this condition.

Claude Code observability is turn-oriented. A multi-turn session can produce one
root `claude-code-turn` span or ATIF trajectory per user turn. That is expected
Expand Down
6 changes: 6 additions & 0 deletions skills/nemo-relay-plugin-observability/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ Use this model when explaining how capture and export relate:
- Event fields include semantic input/output through the ATOF `data` field,
typed profile data such as `model_name` and `tool_call_id`, and codec-provided
annotated LLM request/response data for in-process subscribers and exporters.
- First-class skill tools and requests to read a complete `SKILL.md`
Comment thread
willkill07 marked this conversation as resolved.
Outdated
automatically emit `skill.load` marks under the tool span. The payload
contains only `skill_name`; metadata records the load source and tool name.
Partial reads do not count, and ambiguous slash-command expansions use the
separate `skill.load.inferred` name. The eager mark remains present if tool
execution later fails.

## Shared Lifecycle

Expand Down
Loading