[rollout-observability][2/7] Add Claude Code rollout observations by Glorf · Pull Request #2153 · NVIDIA-NeMo/Gym

Glorf · 2026-07-27T18:47:11Z

Summary

Adds rollout observations to the existing Claude Code Agent integration. Claude Code transcripts are read before temporary configuration cleanup and returned as ng_agent_observations; rollout collection then joins them with Model Server capture.

Data flow

Claude transcript + CLI events ──► agent observations ──┐
Model Server capture ────────────────────────────────────┼──► rollout evidence
Unknown or ambiguous evidence ──► explicit gaps ─────────┘

Coverage

Capability	Status	Why
Conversations and subagents	Yes	Root and nested conversations with parent edges
Model-call ownership	Partial	Exact identifier matches only; unmatched calls stay unowned
Parallel tool timing	Yes	Independent overlapping intervals and outcomes
Context compaction	Partial	Lifecycle is covered; hidden calls require a unique match and `first_kept_item_id` is unavailable
Invocation outcome	Yes	Status, duration, and error type
Model evidence	Yes	Captured requests, responses, and metrics
Gap reporting	Yes	Missing or ambiguous evidence is explicit
Sandbox CPU and memory	No	The current integration runs Claude Code on the host

Compatibility

no new required configuration
existing command construction, MCP wiring, response/reward/verification, and training contracts remain unchanged
collection is enabled only with rollout observability
observation and join failures do not fail the rollout
ng_agent_observations remains optional and additive

This producer complements Model Server capture. It does not define a response or training contract.

Validation

158 Claude Code, shared observation-contract, and Model Server capture tests pass
Claude Code transcript fixtures cover a three-level invocation tree and overlapping tool intervals
Ruff, formatting, and diff checks pass

Stack

copy-pr-bot · 2026-07-27T18:47:14Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Michal Bien <mbien@nvidia.com>

…lation contract (#2114) ## Summary This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage. It also joins Agent Server observations with Model Server call capture: ```text Agent or harness ──> ng_agent_observations ─┐ ├─ exact correlation ─> rollout record Model Servers ─────> ng_model_call_capture ─┘ ``` This is the base of the rollout-observability work and follows up on #1867. ## Contract | Evidence | Representation | |---|---| | Root agent and subagents | `AgentInvocation` | | Model calls owned by an invocation | `ModelCallRef` | | Tool execution and parallel timing | `ToolCallObservation` | | Context compaction | `ContextCompactionObservation` | | Sandbox outcome and lifetime usage | `SandboxObservation` | | Missing or unsupported evidence | `ObservationGap` | Model calls are joined through `model_call_id`, or the exact `(model_ref, response_id)` pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps. This is an observability view, not a training trajectory or a replacement for `NeMoGymResponse`. ## Changes - add the shared observation models and exact join logic - extend `ModelCallRecord` with protocol response ID, model metadata, and raw-payload fallbacks - preserve valid records around damaged capture lines and report incomplete captures - preserve upstream status and response evidence when a Model Server call raises - join Agent Server observations with Model Server capture during rollout-record assembly - propagate rollout correlation through standard Agent, Resources, and Model Server calls - exclude observation payloads from aggregate-metrics requests All behavior remains opt-in through the existing observability configuration. ## Scope - No external trajectory serialization is introduced. - Response, reward, token-ID, and log-probability contracts are unchanged. - Missing hierarchy, ownership, and timing evidence is reported rather than estimated. - Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls. - Harness and sandbox-provider producers remain separate follow-ups. ## Validation Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass. ## Stack - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw and PinchBench rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

Signed-off-by: Michal Bien <mbien@nvidia.com>

github-actions · 2026-07-28T18:24:02Z

🌿 Preview your docs: https://nvidia-preview-feat-rollout-observability-claude-code.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

nemo/gym/model-server/model-call-capture

mlazuka

Looks good

…verage (#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by #2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / #1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs #2115 and #2117-#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. #2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from #2115 through #2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | #2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | #2116 | Claude Code | Superseded by #2153 | See #2153 and this PR | | #2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | #2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | #2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | #2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | #2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with #2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of #1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…lation contract (NVIDIA-NeMo#2114) ## Summary This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage. It also joins Agent Server observations with Model Server call capture: ```text Agent or harness ──> ng_agent_observations ─┐ ├─ exact correlation ─> rollout record Model Servers ─────> ng_model_call_capture ─┘ ``` This is the base of the rollout-observability work and follows up on NVIDIA-NeMo#1867. ## Contract | Evidence | Representation | |---|---| | Root agent and subagents | `AgentInvocation` | | Model calls owned by an invocation | `ModelCallRef` | | Tool execution and parallel timing | `ToolCallObservation` | | Context compaction | `ContextCompactionObservation` | | Sandbox outcome and lifetime usage | `SandboxObservation` | | Missing or unsupported evidence | `ObservationGap` | Model calls are joined through `model_call_id`, or the exact `(model_ref, response_id)` pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps. This is an observability view, not a training trajectory or a replacement for `NeMoGymResponse`. ## Changes - add the shared observation models and exact join logic - extend `ModelCallRecord` with protocol response ID, model metadata, and raw-payload fallbacks - preserve valid records around damaged capture lines and report incomplete captures - preserve upstream status and response evidence when a Model Server call raises - join Agent Server observations with Model Server capture during rollout-record assembly - propagate rollout correlation through standard Agent, Resources, and Model Server calls - exclude observation payloads from aggregate-metrics requests All behavior remains opt-in through the existing observability configuration. ## Scope - No external trajectory serialization is introduced. - Response, reward, token-ID, and log-probability contracts are unchanged. - Missing hierarchy, ownership, and timing evidence is reported rather than estimated. - Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls. - Harness and sandbox-provider producers remain separate follow-ups. ## Validation Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass. ## Stack - [1/7 NVIDIA-NeMo#2114 — Define the rollout observation and correlation contract](NVIDIA-NeMo#2114) - [2/7 NVIDIA-NeMo#2153 — Add Claude Code rollout observations](NVIDIA-NeMo#2153) - [3/7 NVIDIA-NeMo#2115 — Add OpenClaw and PinchBench rollout observations](NVIDIA-NeMo#2115) - [4/7 NVIDIA-NeMo#2117 — Add Hermes rollout observations](NVIDIA-NeMo#2117) - [5/7 NVIDIA-NeMo#2118 — Add Pi rollout observations](NVIDIA-NeMo#2118) - [6/7 NVIDIA-NeMo#2119 — Add OpenCode rollout observations](NVIDIA-NeMo#2119) - [7/7 NVIDIA-NeMo#2120 — Add SWE and OpenHands rollout observations](NVIDIA-NeMo#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…IDIA-NeMo#2153) ## Summary Adds rollout observations to the existing Claude Code Agent integration. Claude Code transcripts are read before temporary configuration cleanup and returned as `ng_agent_observations`; rollout collection then joins them with Model Server capture. ## Data flow ```text Claude transcript + CLI events ──► agent observations ──┐ Model Server capture ────────────────────────────────────┼──► rollout evidence Unknown or ambiguous evidence ──► explicit gaps ─────────┘ ``` ## Coverage | Capability | Status | Why | | --- | --- | --- | | Conversations and subagents | Yes | Root and nested conversations with parent edges | | Model-call ownership | Partial | Exact identifier matches only; unmatched calls stay unowned | | Parallel tool timing | Yes | Independent overlapping intervals and outcomes | | Context compaction | Partial | Lifecycle is covered; hidden calls require a unique match and `first_kept_item_id` is unavailable | | Invocation outcome | Yes | Status, duration, and error type | | Model evidence | Yes | Captured requests, responses, and metrics | | Gap reporting | Yes | Missing or ambiguous evidence is explicit | | Sandbox CPU and memory | No | The current integration runs Claude Code on the host | ## Compatibility - no new required configuration - existing command construction, MCP wiring, response/reward/verification, and training contracts remain unchanged - collection is enabled only with rollout observability - observation and join failures do not fail the rollout - `ng_agent_observations` remains optional and additive This producer complements Model Server capture. It does not define a response or training contract. ## Validation - 158 Claude Code, shared observation-contract, and Model Server capture tests pass - Claude Code transcript fixtures cover a three-level invocation tree and overlapping tool intervals - Ruff, formatting, and diff checks pass ## Stack - [1/7 NVIDIA-NeMo#2114 — Define the rollout observation and correlation contract](NVIDIA-NeMo#2114) - [2/7 NVIDIA-NeMo#2153 — Add Claude Code rollout observations](NVIDIA-NeMo#2153) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…verage (NVIDIA-NeMo#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by NVIDIA-NeMo#2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / NVIDIA-NeMo#1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs NVIDIA-NeMo#2115 and NVIDIA-NeMo#2117-NVIDIA-NeMo#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. NVIDIA-NeMo#2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from NVIDIA-NeMo#2115 through NVIDIA-NeMo#2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | NVIDIA-NeMo#2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | NVIDIA-NeMo#2116 | Claude Code | Superseded by NVIDIA-NeMo#2153 | See NVIDIA-NeMo#2153 and this PR | | NVIDIA-NeMo#2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | NVIDIA-NeMo#2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | NVIDIA-NeMo#2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | NVIDIA-NeMo#2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | NVIDIA-NeMo#2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with NVIDIA-NeMo#2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of NVIDIA-NeMo#1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Add OpenClaw rollout observations for both the standalone agent endpoint and the PinchBench benchmark path. OpenClaw is the observation producer in both paths. PinchBench exports retained OpenClaw session artifacts and attaches the resulting OpenClaw observation bundle; it does not define a separate observation format or producer identity. Observations are additive and are emitted only when observability and rollout correlation are enabled. Existing agent responses, rewards, and grading fields are unchanged. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 are evaluated on the correlated Gym Model Server path. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | OpenClaw | V | V | X | O | V | V | V | C3 remains unsupported because OpenClaw does not emit standardized per-turn records. C4 is partial because retained branched sessions are reported but cannot always reconstruct a single model-visible branch. ## Changes - Normalize retained OpenClaw session records into user, assistant, reasoning, tool-call, and tool-result observations. - Reconstruct subagent lineage from retained OpenClaw session stores. - Correlate policy and judge model calls through rollout-prefixed Gym Model Servers when configured. - Preserve independent tool-call timing, status, duration, and output. - Record context compaction and explicit observation gaps. - Preserve direct model endpoint support when no Gym Model Server is configured. - Isolate observation capture failures from agent responses and benchmark results. ## Evidence boundaries - Model-call correlation requires transcript response IDs and a configured Gym Model Server. - OpenClaw does not identify the exact tool call that spawned a child session. - Branches within one session are reported but are not reconstructed as separate invocations. - Missing timestamps, transcripts, or correlation evidence are represented as explicit gaps. ## Validation - OpenClaw and PinchBench suites: 88 passed. - Core trajectory, collector, correlation, base-agent, and model suites: 155 passed. - Producer-to-collector projection verifies model calls, token details, tool output, status, and independent timing in `ng_trajectory`. - Ruff, formatting, compilation, shell, and diff checks passed. ## Follow-up pull requests - [#2117 — Add Hermes rollout observations](#2117) - [#2118 — Add Pi rollout observations](#2118) - [#2119 — Add OpenCode rollout observations](#2119) - [#2120 — Add SWE and OpenHands rollout observations](#2120) - [#2122 — Add Harbor rollout observations](#2122) Foundation work is available in [#2114](#2114) and [#2153](#2153), both merged into `main`. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Add Hermes production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Hermes observations use instance-local hooks. Uncorrelated requests retain the existing response path. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `hermes_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Hermes does not emit standardized per-turn records. ## Captured evidence - root and delegated-agent conversations, including system prompts and plain reasoning - parent and `delegate_task` spawn relationships when directly observed - model-call references from response IDs - model-visible tool outputs, execution status, timestamps, duration, and independent concurrent-call intervals - context-compaction outcomes and available pre/post token estimates - invocation outcomes Existing Hermes callbacks are chained. Observation failures do not mask agent failures. ## Evidence boundaries - Missing Hermes hooks and unattributed child spawns are reported as explicit gaps. - Opaque `reasoning_details` are not normalized. - Compaction summaries and exact adjacent model calls are unavailable; failed compactions do not report a post-compaction token count. - Local terminal execution reports `no_sandbox_runtime`. - External terminal backends report `sandbox_observation_unavailable` because Gym does not receive their lifecycle or resource telemetry. ## Compatibility - observation capture is enabled only for correlated rollouts - the private self-call attachment is removed before verification - verifier payloads, rewards, and token-bearing output retain existing semantics - `ng_agent_observations` is optional and additive ## Validation - Hermes suite: 44 passed - compatibility suite: 239 passed - local E2E: pinned Hermes agent, concurrent tools, model-call correlation, normalized conversation, and trajectory projection - Ruff, formatting, compile, and diff checks passed ## Related pull requests This is **4/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`.  - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Add Pi production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Pi's JSON event stream is captured while the process runs. Uncorrelated requests retain the existing response path and provider configuration. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `pi_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Pi does not emit standardized per-turn records. ## Captured evidence - model-visible system prompt, user instruction, assistant output, tool calls, and tool results - model-call references from response IDs when `model_server` is configured - independent tool-call intervals and outcomes, including overlapping calls - context-compaction outcomes, summaries, token counts, retained boundaries, and adjacent model-call references when available - invocation status from Pi's terminal `agent_end` event - partial evidence from incomplete or timed-out runs ## Evidence boundaries - Pi does not expose subagent hierarchy. - Tool timings use harness receipt timestamps rather than executor timestamps. - Incomplete model-call, tool, and compaction pairs are reported as gaps. - Unknown terminal outcomes remain `unknown` with an `invocation_outcome_unavailable` gap. - The integration runs on the host and reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional; direct-provider configuration is unchanged - configured Gym Model Servers use rollout-prefixed routing - response, verification, reward, and token-accounting behavior is unchanged - observation parsing failures do not fail the rollout - the private self-call attachment is removed before verification - `ng_agent_observations` is optional and additive ## Validation - Pi suite: 34 passed - Shared observability, correlation, and collector suites: 82 passed - Prefixed FastAPI response round-trip covered - Ruff, formatting, and diff checks passed ## Related pull requests This is **5/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`.  - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary - emit OpenCode session, conversation, tool, and compaction evidence as `ng_agent_observations` - route Gym Model Server calls through the rollout-prefixed endpoint so model-call capture correlates with the rollout - document OpenCode coverage in the trajectory capability matrix ## Capability coverage | Criterion | Status | Scope | | --- | --- | --- | | C1 | V | Correlated Gym Model Server calls use the standard model-call schema. | | C2 | V | Correlated capture retains standard token fields when provided. | | C3 | X | OpenCode does not emit standardized semantic turns. | | C4 | V | Session artifacts retain model-visible invocation conversations, including compaction behavior. | | C5 | V | Tool output, status, start/end timestamps, and duration project into `ng_trajectory`. | | C6 | V | Each tool call retains its own artifact-derived interval. | | C7 | V | The OpenCode host-agent path emits the shared observation contract. | `V` applies to the correlated Gym Model Server path for C1, C2, and C4, consistent with the capability-matrix definition. ## Evidence boundaries - OpenCode artifacts do not expose stable response IDs, so model calls are not assigned to individual OpenCode invocations. - Compaction token counts and adjacent model-call boundaries are reported as unavailable. - A child invocation is linked to a spawning tool only when the artifact identifies one unambiguous call. - The host-run integration reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional. - Uncorrelated `/v1/responses` payloads retain their existing public shape. - Observation metadata is removed before the resource-server verifier receives the response. - Artifact parsing failures produce observation gaps without failing the rollout. - Existing OpenCode provider configuration is copied, not mutated. ## Validation - 24 OpenCode agent tests - 106 combined OpenCode, base-agent, rollout-observability, and rollout-collection tests - subprocess -> OpenCode SQLite artifact -> observations -> `ng_trajectory` E2E - Ruff, formatting, and diff checks ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2120 SWE/OpenHands producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary - emit standardized `ng_agent_observations` for the legacy SWE OpenCode/OpenHands paths and the decoupled `opencode_sandboxed_agent` + SWE-bench path - correlate OpenCode calls through rollout-prefixed Gym Model Server routes when rollout observability or token capture is active - compose OpenCode invocation/tool evidence with separate agent- and verifier-sandbox observations - preserve direct response behavior, grading outputs, retained artifacts, and `subagent_trajectories` ## What changed ### Decoupled OpenCode / SWE-bench - parse OpenCode SQLite sessions into invocations, parent relationships, cumulative model-visible conversations, tool timing/outcomes, compaction events, and exact response IDs - keep the sandboxed agent independently installable by copying the minimal parser closure locally; it no longer imports the standalone `opencode_agent`, and a fresh-process test enforces that boundary - isolate OpenCode data per observed run, download and parse the database before teardown, and remove the local scratch database afterward - record the connected agent sandbox's real provider/ID and compose verifier-sandbox lifecycle evidence returned by `resources_servers/swebench` - emit observations only when a capture-derived rollout ID exists; direct `/v1/responses` behavior remains unchanged ### Legacy SWE harness - OpenCode records retained session invocations, parent relationships, exact response IDs, and the latest cumulative conversation; rollout-prefixed model calls provide capture and token accounting - OpenHands records its available cumulative root conversation and sandbox evidence, while explicitly reporting that exact model-call correlation is unavailable with the pinned fork - legacy Apptainer records leave `sandbox_id` unset and report `sandbox_identity_unavailable` because the runner exposes no real sandbox handle Observation construction fails open. Missing or malformed evidence becomes an explicit gap; provider sentinel values are not exposed as process exit codes, and unavailable resource or lifecycle measurements remain unset rather than inferred. ## Capability coverage | Producer | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `opencode_sandboxed_agent` | V | V | X | V | V | V | V | | `swe_agents` / OpenCode | V | V | X | V | X | X | V | | `swe_agents` / OpenHands | X | X | X | O | X | X | V | The capability matrix documents these evidence boundaries. Legacy artifacts do not provide standardized semantic turns, authoritative per-tool timing, or independent parallel-tool timing. ## Validation - 34 combined standalone and sandboxed OpenCode tests, including fresh-process import isolation - focused SWE-bench resource-server and legacy SWE-agent tests across disabled, observability-only, token-only, and combined capture states - 114 shared rollout-observability, correlation, and collection regressions rerun after the final rebase - SQLite artifact -> parser -> decoupled `/run` composition test covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup primitives - Ruff, formatting, Python compilation, `git diff --check`, and scoped pre-commit hooks A real artifact-compatibility smoke test used Docker Server 29.6.2 on Linux/aarch64 and the exact `swebench/sweb.eval.x86_64.astropy_1776_astropy-12907` image under x86_64 emulation. OpenCode 1.17.11 was installed only inside the temporary container and ran a real gpt-5.5-backed session whose bash tool executed `printf opencode-observability-smoke`; the actual tool result persisted and `opencode export` succeeded. The WAL-mode database contained 1 session, 3 messages, and 7 parts. After closing/exporting, only `opencode.db` was copied and parsed by the final sandbox-local parser, producing 1 completed invocation, 1 tool call, 0 compactions, and only the expected `model_call_ownership_unavailable` gap. The temporary container was stopped and auto-removed; nothing was installed on the host. ## Limitations - the live smoke used the direct NVIDIA gateway rather than Gym's rollout-prefixed model proxy, so it validates the real OpenCode artifact schema/parser but not model-call ownership or capture joining - OpenCode's final prose stream did not terminate after the tool result and was gracefully interrupted - the full decoupled `/run` + verifier flow was not run live because `DockerProvider` cannot reconnect across the resource-server and agent processes; that path requires OpenSandbox - the SQLite parser is intentionally duplicated to keep the two agent servers dependency-isolated and must remain synchronized ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2119 standalone OpenCode producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

mlazuka self-requested a review July 28, 2026 10:57

mlazuka reviewed Jul 28, 2026

View reviewed changes

Comment thread responses_api_agents/claude_code_agent/observability.py Outdated

mlazuka reviewed Jul 28, 2026

View reviewed changes

Comment thread responses_api_agents/claude_code_agent/observability.py Outdated

laszkiewiczp reviewed Jul 28, 2026

View reviewed changes

Comment thread responses_api_agents/claude_code_agent/observability.py

Glorf force-pushed the feat/rollout-observability-base branch from 9058409 to e960d68 Compare July 28, 2026 16:12

Glorf force-pushed the feat/rollout-observability-claude-code branch from a9c7877 to 85de508 Compare July 28, 2026 16:13

Glorf added 13 commits July 28, 2026 18:33

Define rollout observation and correlation contract

3c973bb

Signed-off-by: Michal Bien <mbien@nvidia.com>

Propagate rollout correlation across servers

3309fb1

Signed-off-by: Michal Bien <mbien@nvidia.com>

Tighten rollout observation foundation

dc136ea

Signed-off-by: Michal Bien <mbien@nvidia.com>

Record agent invocation outcomes

671177c

Signed-off-by: Michal Bien <mbien@nvidia.com>

Preserve upstream failure evidence

ffc1eaf

Signed-off-by: Michal Bien <mbien@nvidia.com>

Harden rollout observation contract

662c2b0

Signed-off-by: Michal Bien <mbien@nvidia.com>

Address rollout observation review feedback

698e1e4

Signed-off-by: Michal Bien <mbien@nvidia.com>

Add Claude Code and AnyTerminal rollout observations

fd2f7d5

Signed-off-by: Michal Bien <mbien@nvidia.com>

Report Claude and AnyTerminal observation gaps

8d13d5c

Signed-off-by: Michal Bien <mbien@nvidia.com>

Focus Claude Code rollout observations

08d8e3a

Signed-off-by: Michal Bien <mbien@nvidia.com>

Preserve Claude Code run outcomes

4653953

Signed-off-by: Michal Bien <mbien@nvidia.com>

Harden Claude Code observation extraction

9f68db2

Signed-off-by: Michal Bien <mbien@nvidia.com>

Address Claude Code observation review feedback

9af9174

Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf force-pushed the feat/rollout-observability-base branch from e960d68 to 698e1e4 Compare July 28, 2026 16:33

Glorf force-pushed the feat/rollout-observability-claude-code branch from 85de508 to 9af9174 Compare July 28, 2026 16:33

Base automatically changed from feat/rollout-observability-base to main July 28, 2026 16:57

Glorf added 2 commits July 28, 2026 18:58

Merge branch 'main' into feat/rollout-observability-claude-code

a06aa39

Complete Claude Code compaction observations

4562b9b

Signed-off-by: Michal Bien <mbien@nvidia.com>

Merge branch 'main' into feat/rollout-observability-claude-code

0168ded

mlazuka approved these changes Jul 28, 2026

View reviewed changes

Merge branch 'main' into feat/rollout-observability-claude-code

86e3671

Glorf enabled auto-merge (squash) July 28, 2026 19:50

Glorf merged commit d4cd918 into main Jul 28, 2026
15 checks passed

Glorf deleted the feat/rollout-observability-claude-code branch July 28, 2026 19:53

sephmard linked an issue Jul 29, 2026 that may be closed by this pull request

Add Claude Code rollout observations #2198

Closed

qiaochuz-nv mentioned this pull request Aug 4, 2026

[epic] standardised trajectories format #1867

Closed

7 tasks

giuliolovisotto mentioned this pull request Aug 4, 2026

[Feat] Port evalpipeline validation and artifact quality gates into Gym #2136

Closed

6 tasks

Glorf mentioned this pull request Aug 4, 2026

[fix] Standardize rollout trajectory records and document producer coverage #2293

Merged

Navigation Menu

Uh oh!

[rollout-observability][2/7] Add Claude Code rollout observations - #2153

[rollout-observability][2/7] Add Claude Code rollout observations#2153
Glorf merged 17 commits into
mainfrom
feat/rollout-observability-claude-code

Glorf commented Jul 27, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

mlazuka left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Glorf commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Data flow

Coverage

Compatibility

Validation

Stack

Uh oh!

copy-pr-bot Bot commented Jul 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

mlazuka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Glorf commented Jul 27, 2026 •

edited

Loading