[rollout-observability][1/7] Define the rollout observation and correlation contract by Glorf · Pull Request #2114 · NVIDIA-NeMo/Gym

Glorf · 2026-07-23T12:30:38Z

Summary

This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage.

It also joins Agent Server observations with Model Server call capture:

Agent or harness ──> ng_agent_observations ─┐
                                            ├─ exact correlation ─> rollout record
Model Servers ─────> ng_model_call_capture ─┘

This is the base of the rollout-observability work and follows up on #1867.

Contract

Evidence	Representation
Root agent and subagents	`AgentInvocation`
Model calls owned by an invocation	`ModelCallRef`
Tool execution and parallel timing	`ToolCallObservation`
Context compaction	`ContextCompactionObservation`
Sandbox outcome and lifetime usage	`SandboxObservation`
Missing or unsupported evidence	`ObservationGap`

Model calls are joined through model_call_id, or the exact (model_ref, response_id) pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps.

This is an observability view, not a training trajectory or a replacement for NeMoGymResponse.

Changes

add the shared observation models and exact join logic
extend ModelCallRecord with protocol response ID, model metadata, and raw-payload fallbacks
preserve valid records around damaged capture lines and report incomplete captures
preserve upstream status and response evidence when a Model Server call raises
join Agent Server observations with Model Server capture during rollout-record assembly
propagate rollout correlation through standard Agent, Resources, and Model Server calls
exclude observation payloads from aggregate-metrics requests

All behavior remains opt-in through the existing observability configuration.

Scope

No external trajectory serialization is introduced.
Response, reward, token-ID, and log-probability contracts are unchanged.
Missing hierarchy, ownership, and timing evidence is reported rather than estimated.
Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls.
Harness and sandbox-provider producers remain separate follow-ups.

Validation

Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass.

Stack

copy-pr-bot · 2026-07-23T12:30:42Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-07-23T12:31:42Z

🌿 Preview your docs: https://nvidia-preview-feat-rollout-observability-base.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

nemo/gym/model-server/model-call-capture

laszkiewiczp · 2026-07-28T11:16:35Z

+    tokens_before: Optional[int] = None
+    tokens_after: Optional[int] = None


maybe we could validate if after < before?

I kept these as non-negative producer-reported counts rather than enforcing after < before, since opaque producers may use different accounting boundaries. The field descriptions now make that explicit.

mlazuka

Thanks for the extra changes :) LGTM now

Signed-off-by: Michal Bien <mbien@nvidia.com>

) ## Summary Adds rollout observations to the existing Claude Code Agent integration. Claude Code transcripts are read before temporary configuration cleanup and returned as `ng_agent_observations`; rollout collection then joins them with Model Server capture. ## Data flow ```text Claude transcript + CLI events ──► agent observations ──┐ Model Server capture ────────────────────────────────────┼──► rollout evidence Unknown or ambiguous evidence ──► explicit gaps ─────────┘ ``` ## Coverage | Capability | Status | Why | | --- | --- | --- | | Conversations and subagents | Yes | Root and nested conversations with parent edges | | Model-call ownership | Partial | Exact identifier matches only; unmatched calls stay unowned | | Parallel tool timing | Yes | Independent overlapping intervals and outcomes | | Context compaction | Partial | Lifecycle is covered; hidden calls require a unique match and `first_kept_item_id` is unavailable | | Invocation outcome | Yes | Status, duration, and error type | | Model evidence | Yes | Captured requests, responses, and metrics | | Gap reporting | Yes | Missing or ambiguous evidence is explicit | | Sandbox CPU and memory | No | The current integration runs Claude Code on the host | ## Compatibility - no new required configuration - existing command construction, MCP wiring, response/reward/verification, and training contracts remain unchanged - collection is enabled only with rollout observability - observation and join failures do not fail the rollout - `ng_agent_observations` remains optional and additive This producer complements Model Server capture. It does not define a response or training contract. ## Validation - 158 Claude Code, shared observation-contract, and Model Server capture tests pass - Claude Code transcript fixtures cover a three-level invocation tree and overlapping tool intervals - Ruff, formatting, and diff checks pass ## Stack - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…g it Merging main brought in nemo_gym.rollout_correlation (#2114), which makes the rollout-id charset explicit: RolloutContextMiddleware matches [A-Za-z0-9][A-Za-z0-9._-]* and _validate_rollout_id enforces the same set on the capture path. This server predates that contract and matched [^/]+. That is too permissive here specifically. ASGI hands over a percent-decoded path, and this server does something the core middleware does not -- it puts the id into an outbound HTTP header. A crafted prefix such as /ng-rollout/x%0d%0ax-injected:%201/v1/responses parses as a rollout id carrying CRLF under the old pattern. The client library rejects such a value, so the practical outcome was a failed request rather than a forged header, but relying on a downstream library to catch what this regex let through is the wrong place to draw the line. Matching the core charset means an off-contract id is simply not published: the call goes out uncorrelated, exactly as it does for a request that never carried a prefix. The unused `rest` capture group goes too; the path is forwarded untouched, so it was never read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>

…lation contract (NVIDIA-NeMo#2114) ## Summary This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage. It also joins Agent Server observations with Model Server call capture: ```text Agent or harness ──> ng_agent_observations ─┐ ├─ exact correlation ─> rollout record Model Servers ─────> ng_model_call_capture ─┘ ``` This is the base of the rollout-observability work and follows up on NVIDIA-NeMo#1867. ## Contract | Evidence | Representation | |---|---| | Root agent and subagents | `AgentInvocation` | | Model calls owned by an invocation | `ModelCallRef` | | Tool execution and parallel timing | `ToolCallObservation` | | Context compaction | `ContextCompactionObservation` | | Sandbox outcome and lifetime usage | `SandboxObservation` | | Missing or unsupported evidence | `ObservationGap` | Model calls are joined through `model_call_id`, or the exact `(model_ref, response_id)` pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps. This is an observability view, not a training trajectory or a replacement for `NeMoGymResponse`. ## Changes - add the shared observation models and exact join logic - extend `ModelCallRecord` with protocol response ID, model metadata, and raw-payload fallbacks - preserve valid records around damaged capture lines and report incomplete captures - preserve upstream status and response evidence when a Model Server call raises - join Agent Server observations with Model Server capture during rollout-record assembly - propagate rollout correlation through standard Agent, Resources, and Model Server calls - exclude observation payloads from aggregate-metrics requests All behavior remains opt-in through the existing observability configuration. ## Scope - No external trajectory serialization is introduced. - Response, reward, token-ID, and log-probability contracts are unchanged. - Missing hierarchy, ownership, and timing evidence is reported rather than estimated. - Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls. - Harness and sandbox-provider producers remain separate follow-ups. ## Validation Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass. ## Stack - [1/7 NVIDIA-NeMo#2114 — Define the rollout observation and correlation contract](NVIDIA-NeMo#2114) - [2/7 NVIDIA-NeMo#2153 — Add Claude Code rollout observations](NVIDIA-NeMo#2153) - [3/7 NVIDIA-NeMo#2115 — Add OpenClaw and PinchBench rollout observations](NVIDIA-NeMo#2115) - [4/7 NVIDIA-NeMo#2117 — Add Hermes rollout observations](NVIDIA-NeMo#2117) - [5/7 NVIDIA-NeMo#2118 — Add Pi rollout observations](NVIDIA-NeMo#2118) - [6/7 NVIDIA-NeMo#2119 — Add OpenCode rollout observations](NVIDIA-NeMo#2119) - [7/7 NVIDIA-NeMo#2120 — Add SWE and OpenHands rollout observations](NVIDIA-NeMo#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…IDIA-NeMo#2153) ## Summary Adds rollout observations to the existing Claude Code Agent integration. Claude Code transcripts are read before temporary configuration cleanup and returned as `ng_agent_observations`; rollout collection then joins them with Model Server capture. ## Data flow ```text Claude transcript + CLI events ──► agent observations ──┐ Model Server capture ────────────────────────────────────┼──► rollout evidence Unknown or ambiguous evidence ──► explicit gaps ─────────┘ ``` ## Coverage | Capability | Status | Why | | --- | --- | --- | | Conversations and subagents | Yes | Root and nested conversations with parent edges | | Model-call ownership | Partial | Exact identifier matches only; unmatched calls stay unowned | | Parallel tool timing | Yes | Independent overlapping intervals and outcomes | | Context compaction | Partial | Lifecycle is covered; hidden calls require a unique match and `first_kept_item_id` is unavailable | | Invocation outcome | Yes | Status, duration, and error type | | Model evidence | Yes | Captured requests, responses, and metrics | | Gap reporting | Yes | Missing or ambiguous evidence is explicit | | Sandbox CPU and memory | No | The current integration runs Claude Code on the host | ## Compatibility - no new required configuration - existing command construction, MCP wiring, response/reward/verification, and training contracts remain unchanged - collection is enabled only with rollout observability - observation and join failures do not fail the rollout - `ng_agent_observations` remains optional and additive This producer complements Model Server capture. It does not define a response or training contract. ## Validation - 158 Claude Code, shared observation-contract, and Model Server capture tests pass - Claude Code transcript fixtures cover a three-level invocation tree and overlapping tool intervals - Ruff, formatting, and diff checks pass ## Stack - [1/7 NVIDIA-NeMo#2114 — Define the rollout observation and correlation contract](NVIDIA-NeMo#2114) - [2/7 NVIDIA-NeMo#2153 — Add Claude Code rollout observations](NVIDIA-NeMo#2153) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Add OpenClaw rollout observations for both the standalone agent endpoint and the PinchBench benchmark path. OpenClaw is the observation producer in both paths. PinchBench exports retained OpenClaw session artifacts and attaches the resulting OpenClaw observation bundle; it does not define a separate observation format or producer identity. Observations are additive and are emitted only when observability and rollout correlation are enabled. Existing agent responses, rewards, and grading fields are unchanged. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 are evaluated on the correlated Gym Model Server path. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | OpenClaw | V | V | X | O | V | V | V | C3 remains unsupported because OpenClaw does not emit standardized per-turn records. C4 is partial because retained branched sessions are reported but cannot always reconstruct a single model-visible branch. ## Changes - Normalize retained OpenClaw session records into user, assistant, reasoning, tool-call, and tool-result observations. - Reconstruct subagent lineage from retained OpenClaw session stores. - Correlate policy and judge model calls through rollout-prefixed Gym Model Servers when configured. - Preserve independent tool-call timing, status, duration, and output. - Record context compaction and explicit observation gaps. - Preserve direct model endpoint support when no Gym Model Server is configured. - Isolate observation capture failures from agent responses and benchmark results. ## Evidence boundaries - Model-call correlation requires transcript response IDs and a configured Gym Model Server. - OpenClaw does not identify the exact tool call that spawned a child session. - Branches within one session are reported but are not reconstructed as separate invocations. - Missing timestamps, transcripts, or correlation evidence are represented as explicit gaps. ## Validation - OpenClaw and PinchBench suites: 88 passed. - Core trajectory, collector, correlation, base-agent, and model suites: 155 passed. - Producer-to-collector projection verifies model calls, token details, tool output, status, and independent timing in `ng_trajectory`. - Ruff, formatting, compilation, shell, and diff checks passed. ## Follow-up pull requests - [#2117 — Add Hermes rollout observations](#2117) - [#2118 — Add Pi rollout observations](#2118) - [#2119 — Add OpenCode rollout observations](#2119) - [#2120 — Add SWE and OpenHands rollout observations](#2120) - [#2122 — Add Harbor rollout observations](#2122) Foundation work is available in [#2114](#2114) and [#2153](#2153), both merged into `main`. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Add Hermes production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Hermes observations use instance-local hooks. Uncorrelated requests retain the existing response path. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `hermes_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Hermes does not emit standardized per-turn records. ## Captured evidence - root and delegated-agent conversations, including system prompts and plain reasoning - parent and `delegate_task` spawn relationships when directly observed - model-call references from response IDs - model-visible tool outputs, execution status, timestamps, duration, and independent concurrent-call intervals - context-compaction outcomes and available pre/post token estimates - invocation outcomes Existing Hermes callbacks are chained. Observation failures do not mask agent failures. ## Evidence boundaries - Missing Hermes hooks and unattributed child spawns are reported as explicit gaps. - Opaque `reasoning_details` are not normalized. - Compaction summaries and exact adjacent model calls are unavailable; failed compactions do not report a post-compaction token count. - Local terminal execution reports `no_sandbox_runtime`. - External terminal backends report `sandbox_observation_unavailable` because Gym does not receive their lifecycle or resource telemetry. ## Compatibility - observation capture is enabled only for correlated rollouts - the private self-call attachment is removed before verification - verifier payloads, rewards, and token-bearing output retain existing semantics - `ng_agent_observations` is optional and additive ## Validation - Hermes suite: 44 passed - compatibility suite: 239 passed - local E2E: pinned Hermes agent, concurrent tools, model-call correlation, normalized conversation, and trajectory projection - Ruff, formatting, compile, and diff checks passed ## Related pull requests This is **4/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`.  - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Add Pi production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Pi's JSON event stream is captured while the process runs. Uncorrelated requests retain the existing response path and provider configuration. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `pi_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Pi does not emit standardized per-turn records. ## Captured evidence - model-visible system prompt, user instruction, assistant output, tool calls, and tool results - model-call references from response IDs when `model_server` is configured - independent tool-call intervals and outcomes, including overlapping calls - context-compaction outcomes, summaries, token counts, retained boundaries, and adjacent model-call references when available - invocation status from Pi's terminal `agent_end` event - partial evidence from incomplete or timed-out runs ## Evidence boundaries - Pi does not expose subagent hierarchy. - Tool timings use harness receipt timestamps rather than executor timestamps. - Incomplete model-call, tool, and compaction pairs are reported as gaps. - Unknown terminal outcomes remain `unknown` with an `invocation_outcome_unavailable` gap. - The integration runs on the host and reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional; direct-provider configuration is unchanged - configured Gym Model Servers use rollout-prefixed routing - response, verification, reward, and token-accounting behavior is unchanged - observation parsing failures do not fail the rollout - the private self-call attachment is removed before verification - `ng_agent_observations` is optional and additive ## Validation - Pi suite: 34 passed - Shared observability, correlation, and collector suites: 82 passed - Prefixed FastAPI response round-trip covered - Ruff, formatting, and diff checks passed ## Related pull requests This is **5/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`.  - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary - emit OpenCode session, conversation, tool, and compaction evidence as `ng_agent_observations` - route Gym Model Server calls through the rollout-prefixed endpoint so model-call capture correlates with the rollout - document OpenCode coverage in the trajectory capability matrix ## Capability coverage | Criterion | Status | Scope | | --- | --- | --- | | C1 | V | Correlated Gym Model Server calls use the standard model-call schema. | | C2 | V | Correlated capture retains standard token fields when provided. | | C3 | X | OpenCode does not emit standardized semantic turns. | | C4 | V | Session artifacts retain model-visible invocation conversations, including compaction behavior. | | C5 | V | Tool output, status, start/end timestamps, and duration project into `ng_trajectory`. | | C6 | V | Each tool call retains its own artifact-derived interval. | | C7 | V | The OpenCode host-agent path emits the shared observation contract. | `V` applies to the correlated Gym Model Server path for C1, C2, and C4, consistent with the capability-matrix definition. ## Evidence boundaries - OpenCode artifacts do not expose stable response IDs, so model calls are not assigned to individual OpenCode invocations. - Compaction token counts and adjacent model-call boundaries are reported as unavailable. - A child invocation is linked to a spawning tool only when the artifact identifies one unambiguous call. - The host-run integration reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional. - Uncorrelated `/v1/responses` payloads retain their existing public shape. - Observation metadata is removed before the resource-server verifier receives the response. - Artifact parsing failures produce observation gaps without failing the rollout. - Existing OpenCode provider configuration is copied, not mutated. ## Validation - 24 OpenCode agent tests - 106 combined OpenCode, base-agent, rollout-observability, and rollout-collection tests - subprocess -> OpenCode SQLite artifact -> observations -> `ng_trajectory` E2E - Ruff, formatting, and diff checks ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2120 SWE/OpenHands producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…g it Merging main brought in nemo_gym.rollout_correlation (#2114), which makes the rollout-id charset explicit: RolloutContextMiddleware matches [A-Za-z0-9][A-Za-z0-9._-]* and _validate_rollout_id enforces the same set on the capture path. This server predates that contract and matched [^/]+. That is too permissive here specifically. ASGI hands over a percent-decoded path, and this server does something the core middleware does not -- it puts the id into an outbound HTTP header. A crafted prefix such as /ng-rollout/x%0d%0ax-injected:%201/v1/responses parses as a rollout id carrying CRLF under the old pattern. The client library rejects such a value, so the practical outcome was a failed request rather than a forged header, but relying on a downstream library to catch what this regex let through is the wrong place to draw the line. Matching the core charset means an off-contract id is simply not published: the call goes out uncorrelated, exactly as it does for a request that never carried a prefix. The unused `rest` capture group goes too; the path is forwarded untouched, so it was never read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>

…elation without full capture NVIDIA-NeMo#2114 landed the rollout correlation contract, but two gaps keep production incident attribution manual. It is one-directional. BaseSeedSessionResponse is an empty model and BaseVerifyResponse carries no identifier, so the environment-side handle - the container, browser context or provider session that actually consumed quota - never reaches the training side. Add an optional env_session_id to both, opaque to Gym and absent unless an environment reports it. It is gated on model-call capture. server_utils only applies the rollout prefix to resources servers when observability_enabled is true, and that flag defaults off, so current_rollout_id() is None inside resources servers exactly in the runs where an incident happens. Add rollout_correlation_enabled, which turns on the prefix without turning on capture. It defaults to false, so nothing changes for existing runs; happy to flip the default if maintainers prefer. No training framework needs a transport change: verl and NeMo-RL already carry the whole verify response as full_result. Closes NVIDIA-NeMo#2610 Signed-off-by: waple0820 <232305951+waple0820@users.noreply.github.com>

…g it Merging main brought in nemo_gym.rollout_correlation (#2114), which makes the rollout-id charset explicit: RolloutContextMiddleware matches [A-Za-z0-9][A-Za-z0-9._-]* and _validate_rollout_id enforces the same set on the capture path. This server predates that contract and matched [^/]+. That is too permissive here specifically. ASGI hands over a percent-decoded path, and this server does something the core middleware does not -- it puts the id into an outbound HTTP header. A crafted prefix such as /ng-rollout/x%0d%0ax-injected:%201/v1/responses parses as a rollout id carrying CRLF under the old pattern. The client library rejects such a value, so the practical outcome was a failed request rather than a forged header, but relying on a downstream library to catch what this regex let through is the wrong place to draw the line. Matching the core charset means an off-contract id is simply not published: the call goes out uncorrelated, exactly as it does for a request that never carried a prefix. The unused `rest` capture group goes too; the path is forwarded untouched, so it was never read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>

## Summary - emit standardized `ng_agent_observations` for the legacy SWE OpenCode/OpenHands paths and the decoupled `opencode_sandboxed_agent` + SWE-bench path - correlate OpenCode calls through rollout-prefixed Gym Model Server routes when rollout observability or token capture is active - compose OpenCode invocation/tool evidence with separate agent- and verifier-sandbox observations - preserve direct response behavior, grading outputs, retained artifacts, and `subagent_trajectories` ## What changed ### Decoupled OpenCode / SWE-bench - parse OpenCode SQLite sessions into invocations, parent relationships, cumulative model-visible conversations, tool timing/outcomes, compaction events, and exact response IDs - keep the sandboxed agent independently installable by copying the minimal parser closure locally; it no longer imports the standalone `opencode_agent`, and a fresh-process test enforces that boundary - isolate OpenCode data per observed run, download and parse the database before teardown, and remove the local scratch database afterward - record the connected agent sandbox's real provider/ID and compose verifier-sandbox lifecycle evidence returned by `resources_servers/swebench` - emit observations only when a capture-derived rollout ID exists; direct `/v1/responses` behavior remains unchanged ### Legacy SWE harness - OpenCode records retained session invocations, parent relationships, exact response IDs, and the latest cumulative conversation; rollout-prefixed model calls provide capture and token accounting - OpenHands records its available cumulative root conversation and sandbox evidence, while explicitly reporting that exact model-call correlation is unavailable with the pinned fork - legacy Apptainer records leave `sandbox_id` unset and report `sandbox_identity_unavailable` because the runner exposes no real sandbox handle Observation construction fails open. Missing or malformed evidence becomes an explicit gap; provider sentinel values are not exposed as process exit codes, and unavailable resource or lifecycle measurements remain unset rather than inferred. ## Capability coverage | Producer | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `opencode_sandboxed_agent` | V | V | X | V | V | V | V | | `swe_agents` / OpenCode | V | V | X | V | X | X | V | | `swe_agents` / OpenHands | X | X | X | O | X | X | V | The capability matrix documents these evidence boundaries. Legacy artifacts do not provide standardized semantic turns, authoritative per-tool timing, or independent parallel-tool timing. ## Validation - 34 combined standalone and sandboxed OpenCode tests, including fresh-process import isolation - focused SWE-bench resource-server and legacy SWE-agent tests across disabled, observability-only, token-only, and combined capture states - 114 shared rollout-observability, correlation, and collection regressions rerun after the final rebase - SQLite artifact -> parser -> decoupled `/run` composition test covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup primitives - Ruff, formatting, Python compilation, `git diff --check`, and scoped pre-commit hooks A real artifact-compatibility smoke test used Docker Server 29.6.2 on Linux/aarch64 and the exact `swebench/sweb.eval.x86_64.astropy_1776_astropy-12907` image under x86_64 emulation. OpenCode 1.17.11 was installed only inside the temporary container and ran a real gpt-5.5-backed session whose bash tool executed `printf opencode-observability-smoke`; the actual tool result persisted and `opencode export` succeeded. The WAL-mode database contained 1 session, 3 messages, and 7 parts. After closing/exporting, only `opencode.db` was copied and parsed by the final sandbox-local parser, producing 1 completed invocation, 1 tool call, 0 compactions, and only the expected `model_call_ownership_unavailable` gap. The temporary container was stopped and auto-removed; nothing was installed on the host. ## Limitations - the live smoke used the direct NVIDIA gateway rather than Gym's rollout-prefixed model proxy, so it validates the real OpenCode artifact schema/parser but not model-call ownership or capture joining - OpenCode's final prose stream did not terminate after the tool result and was gracefully interrupted - the full decoupled `/run` + verifier flow was not run live because `DockerProvider` cannot reconnect across the resource-server and agent processes; that path requires OpenSandbox - the SQLite parser is intentionally duplicated to keep the two agent servers dependency-isolated and must remain synchronized ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2119 standalone OpenCode producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…g it Merging main brought in nemo_gym.rollout_correlation (#2114), which makes the rollout-id charset explicit: RolloutContextMiddleware matches [A-Za-z0-9][A-Za-z0-9._-]* and _validate_rollout_id enforces the same set on the capture path. This server predates that contract and matched [^/]+. That is too permissive here specifically. ASGI hands over a percent-decoded path, and this server does something the core middleware does not -- it puts the id into an outbound HTTP header. A crafted prefix such as /ng-rollout/x%0d%0ax-injected:%201/v1/responses parses as a rollout id carrying CRLF under the old pattern. The client library rejects such a value, so the practical outcome was a failed request rather than a forged header, but relying on a downstream library to catch what this regex let through is the wrong place to draw the line. Matching the core charset means an off-contract id is simply not published: the call goes out uncorrelated, exactly as it does for a request that never carried a prefix. The unused `rest` capture group goes too; the path is forwarded untouched, so it was never read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>

## What does this PR do? Adds two capabilities to the SWE agent harness (`responses_api_agents/swe_agents/`), plus fixes to the DeepSWE verifier path found while baselining that dataset. **1. opencode patch mode.** New `opencode_patch_mode` config field (`worktree` | `committed`, default `worktree`), exported to the harness as `PATCH_MODE` and forwarded to the bench CLI as `--patch-mode`. `worktree` is the existing capture — `git diff` of the working tree with untracked files marked intent-to-add — and is unchanged. `committed` instead diffs the pre-run HEAD against the most advanced commit the agent left behind, searched across HEAD and every local branch. That mode is required by task families whose problem statement tells the agent to commit its solution: DeepSWE's statements end with *"work on this in a new branch from main and commit everything when you are done"*, so those rollouts finish with a clean tree and `git diff` records every patch as 0 bytes regardless of whether the model solved the task. Both modes emit a plain `base -> final tree` unified diff, so the eval side is unchanged. **2. opencode trajectory replay.** New `opencode_replay.py` module plus wiring, so a rollout can resume a partially-completed trajectory on a fresh container instead of restarting the task. When a request's `input` carries a prior trajectory (`function_call` / `function_call_output` items beyond the seed messages), it is converted to chat-completion format and surfaced as `problem_info["replay_messages"]`; the processors materialize it (plus the recorded system prompt and, for opencode, a subagent manifest) and forward the paths to `run_infer.sh` as positional args (opencode `#13`/`#14`, openhands `#18`). Recorded subagent sessions are linked to the exact parent task call that spawned them rather than by metadata order, so parallel siblings and nested agents can't consume each other's turns, and live continuations are merged back onto the recorded root. Per-session trajectory records now also carry replay linkage and global ordering (`recorded_session_id`, `spawn_call_id`, `spawn_index`, `global_turn`, …), and `SWEBenchVerifyResponse.subagent_trajectories` is populated so returned create-params are directly replay-ready. **3. DeepSWE verifier fixes.** - Carry optional `tests/grader.py` / `tests/config.json` from Harbor bundles (DeepSWE v1.1 delegates preparation and scoring to them) and mount them into the eval container. Optional, so older bundles still convert and grade. - Read the reward from `/logs/verifier/reward.json` (new tasks) as well as `reward.txt` (older tasks and crash sentinels). - Fix `mkdir -p /logs/vserifier` → `/logs/verifier`; the typo meant the directory the verifier writes its reward into was never created, so the reward read back empty and the task graded 0. - Skip injecting the online Maven mirror for verifiers that deliberately run offline (`mvn -o` / `--offline`) against their image's build-time JVM cache — Maven records which repository supplied each cached artifact, so a differently-named mirror makes cached plugins look unavailable. **4. Reverts #2120** (SWE agent rollout observations), removing `observability.py`, `tests/test_observability.py` and their call sites.  Tests: `responses_api_agents/swe_agents/tests/test_app.py` gains 26 tests covering the replay path — gym-side message conversion (openhands and opencode), replay file materialization and system-prompt pinning, positional-arg ordering, subagent manifest mounting with `ENABLE_SUBAGENTS=1`, parent-task-call linkage for parallel and nested children, completed-invocation truncation, legacy payload parsing, live-continuation merging, and preservation of the new trajectory record fields. Note: end-to-end opencode replay also needs `--replay-messages-file` / `--replay-subagents-file` in the opencode fork. The commit currently pinned by `configs/swebench_opencode.yaml` (`nv-opencode@sdd/dev`) does not have them — its `run_infer.sh` stops at positional `#12` — so until that lands and the pin moves, an opencode replay request is a no-op and the agent starts the task from scratch. openhands replay is unaffected. Bumping the pin also requires clearing `cache/swe_agents/swe_opencode_setup`, since `OpenCodeHarnessProcessor.setup()` returns early once the tree exists and never re-checks-out the configured commit. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Sugam Devare <sdevare@nvidia.com>

…rify NVIDIA-NeMo#2114 landed the rollout correlation contract, but it is one-directional. The training side learns nothing about the handle the environment actually allocated — the container, the browser context, the provider session that consumed quota — so a rollout record and a provider-side log can only be joined on a timestamp. `env_session_id` is optional on both `BaseSeedSessionResponse` and `BaseVerifyResponse`, opaque to Gym, and absent unless an environment reports one, so nothing changes for an environment that does not. This originally also added a `rollout_correlation_enabled` key so the rollout prefix could reach resources servers without turning on model-call capture. NVIDIA-NeMo#2783 removes that need by making correlation independent of the observability gate rather than adding a second flag, which is the better shape, so that half is dropped here. No training framework needs a transport change: verl and NeMo-RL already carry the whole verify response as `full_result`. Terminology: this is the environment session created by `/seed_session`, not the vLLM router KV-cache affinity of NVIDIA-NeMo#2570 / NVIDIA-NeMo#2347 / NVIDIA-NeMo#2369. Signed-off-by: waple0820 <232305951+waple0820@users.noreply.github.com>

Glorf marked this pull request as ready for review July 23, 2026 12:34

Glorf requested a review from a team as a code owner July 23, 2026 12:34

Glorf force-pushed the feat/rollout-observability-base branch 2 times, most recently from 4fd8fdf to 03b751f Compare July 23, 2026 14:13

github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 24, 2026

github-actions Bot mentioned this pull request Jul 24, 2026

[Tracker] Pull request handoff SLA #2067

Open

Glorf changed the title ~~[rollout-observability][1/7] Define the rollout observation and correlation contract~~ [rollout-observability][1/8] Define the rollout observation and correlation contract Jul 25, 2026

Glorf mentioned this pull request Jul 25, 2026

[rollout-observability][follow-up] Correlate auxiliary Model Server calls #2122

Closed

Glorf force-pushed the feat/rollout-observability-base branch from edce744 to b851fdf Compare July 27, 2026 12:05

sephmard mentioned this pull request Jul 27, 2026

Epic: Reproducible eval output: run metadata, health signals, and consumable artifacts #1863

Closed

Glorf changed the title ~~[rollout-observability][1/8] Define the rollout observation and correlation contract~~ [rollout-observability][1/2] Define the rollout observation and correlation contract Jul 27, 2026

Glorf mentioned this pull request Jul 27, 2026

[rollout-observability][2/7] Add Claude Code rollout observations #2153

Merged

Glorf changed the title ~~[rollout-observability][1/2] Define the rollout observation and correlation contract~~ [rollout-observability][1/7] Define the rollout observation and correlation contract Jul 27, 2026

mlazuka self-requested a review July 28, 2026 10:57

github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 28, 2026

laszkiewiczp reviewed Jul 28, 2026

View reviewed changes

Comment thread nemo_gym/rollout_observability.py

laszkiewiczp reviewed Jul 28, 2026

View reviewed changes

Glorf force-pushed the feat/rollout-observability-base branch from 9058409 to e960d68 Compare July 28, 2026 16:12

mlazuka approved these changes Jul 28, 2026

View reviewed changes

Glorf added 2 commits July 28, 2026 18:33

Define rollout observation and correlation contract

3c973bb

Signed-off-by: Michal Bien <mbien@nvidia.com>

Propagate rollout correlation across servers

3309fb1

Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf added 2 commits July 28, 2026 18:33

Harden rollout observation contract

662c2b0

Signed-off-by: Michal Bien <mbien@nvidia.com>

Address rollout observation review feedback

698e1e4

Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf force-pushed the feat/rollout-observability-base branch from e960d68 to 698e1e4 Compare July 28, 2026 16:33

Glorf merged commit bbf58c1 into main Jul 28, 2026
16 checks passed

Glorf deleted the feat/rollout-observability-base branch July 28, 2026 16:57

linj-glitch mentioned this pull request Jul 28, 2026

feat: add switchyard_model server for router-backed evals with condition provenance #2141

Merged

sephmard linked an issue Jul 29, 2026 that may be closed by this pull request

[epic] standardised trajectories format #1867

Closed

7 tasks

qiaochuz-nv mentioned this pull request Aug 4, 2026

[epic] standardised trajectories format #1867

Closed

7 tasks

This was referenced Aug 19, 2026

feat(observability): return the environment session id so infrastructure failures can be attributed to rollouts #2610

Open

feat(observability): return the environment session id and allow correlation without full capture #2613

Open

Navigation Menu

Uh oh!

[rollout-observability][1/7] Define the rollout observation and correlation contract - #2114

[rollout-observability][1/7] Define the rollout observation and correlation contract#2114
Glorf merged 7 commits into
mainfrom
feat/rollout-observability-base

Glorf commented Jul 23, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

Uh oh!

laszkiewiczp Jul 28, 2026

Uh oh!

Glorf Jul 28, 2026

Uh oh!

Uh oh!

mlazuka left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		tokens_before: Optional[int] = None
		tokens_after: Optional[int] = None

Uh oh!

Conversation

Glorf commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Contract

Changes

Scope

Validation

Stack

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

Uh oh!

laszkiewiczp Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Glorf Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mlazuka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Glorf commented Jul 23, 2026 •

edited

Loading