[rollout-observability][4/7] Add Hermes rollout observations by Glorf · Pull Request #2117 · NVIDIA-NeMo/Gym

Glorf · 2026-07-23T12:30:46Z

Summary

Add Hermes production of ng_agent_observations for correlated rollouts and update the trajectory capability matrix.

Hermes observations use instance-local hooks. Uncorrelated requests retain the existing response path.

Capability coverage

V = supported, O = partial or path-dependent, X = unavailable. C1, C2, and C4 require correlated Gym Model Server capture.

Agent	C1	C2	C3	C4	C5	C6	C7
`hermes_agent`	V	V	X	V	V	V	V

C3 remains unavailable because Hermes does not emit standardized per-turn records.

Captured evidence

root and delegated-agent conversations, including system prompts and plain reasoning
parent and delegate_task spawn relationships when directly observed
model-call references from response IDs
model-visible tool outputs, execution status, timestamps, duration, and independent concurrent-call intervals
context-compaction outcomes and available pre/post token estimates
invocation outcomes

Existing Hermes callbacks are chained. Observation failures do not mask agent failures.

Evidence boundaries

Missing Hermes hooks and unattributed child spawns are reported as explicit gaps.
Opaque reasoning_details are not normalized.
Compaction summaries and exact adjacent model calls are unavailable; failed compactions do not report a post-compaction token count.
Local terminal execution reports no_sandbox_runtime.
External terminal backends report sandbox_observation_unavailable because Gym does not receive their lifecycle or resource telemetry.

Compatibility

observation capture is enabled only for correlated rollouts
the private self-call attachment is removed before verification
verifier payloads, rewards, and token-bearing output retain existing semantics
ng_agent_observations is optional and additive

Validation

Hermes suite: 44 passed
compatibility suite: 239 passed
local E2E: pinned Hermes agent, concurrent tools, model-call correlation, normalized conversation, and trajectory projection
Ruff, formatting, compile, and diff checks passed

Related pull requests

This is 4/7 in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on main.

copy-pr-bot · 2026-07-23T12:30:51Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…lation contract (#2114) ## Summary This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage. It also joins Agent Server observations with Model Server call capture: ```text Agent or harness ──> ng_agent_observations ─┐ ├─ exact correlation ─> rollout record Model Servers ─────> ng_model_call_capture ─┘ ``` This is the base of the rollout-observability work and follows up on #1867. ## Contract | Evidence | Representation | |---|---| | Root agent and subagents | `AgentInvocation` | | Model calls owned by an invocation | `ModelCallRef` | | Tool execution and parallel timing | `ToolCallObservation` | | Context compaction | `ContextCompactionObservation` | | Sandbox outcome and lifetime usage | `SandboxObservation` | | Missing or unsupported evidence | `ObservationGap` | Model calls are joined through `model_call_id`, or the exact `(model_ref, response_id)` pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps. This is an observability view, not a training trajectory or a replacement for `NeMoGymResponse`. ## Changes - add the shared observation models and exact join logic - extend `ModelCallRecord` with protocol response ID, model metadata, and raw-payload fallbacks - preserve valid records around damaged capture lines and report incomplete captures - preserve upstream status and response evidence when a Model Server call raises - join Agent Server observations with Model Server capture during rollout-record assembly - propagate rollout correlation through standard Agent, Resources, and Model Server calls - exclude observation payloads from aggregate-metrics requests All behavior remains opt-in through the existing observability configuration. ## Scope - No external trajectory serialization is introduced. - Response, reward, token-ID, and log-probability contracts are unchanged. - Missing hierarchy, ownership, and timing evidence is reported rather than estimated. - Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls. - Harness and sandbox-provider producers remain separate follow-ups. ## Validation Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass. ## Stack - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw and PinchBench rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…verage (#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by #2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / #1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs #2115 and #2117-#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. #2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from #2115 through #2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | #2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | #2116 | Claude Code | Superseded by #2153 | See #2153 and this PR | | #2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | #2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | #2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | #2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | #2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with #2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of #1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…lation contract (NVIDIA-NeMo#2114) ## Summary This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage. It also joins Agent Server observations with Model Server call capture: ```text Agent or harness ──> ng_agent_observations ─┐ ├─ exact correlation ─> rollout record Model Servers ─────> ng_model_call_capture ─┘ ``` This is the base of the rollout-observability work and follows up on NVIDIA-NeMo#1867. ## Contract | Evidence | Representation | |---|---| | Root agent and subagents | `AgentInvocation` | | Model calls owned by an invocation | `ModelCallRef` | | Tool execution and parallel timing | `ToolCallObservation` | | Context compaction | `ContextCompactionObservation` | | Sandbox outcome and lifetime usage | `SandboxObservation` | | Missing or unsupported evidence | `ObservationGap` | Model calls are joined through `model_call_id`, or the exact `(model_ref, response_id)` pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps. This is an observability view, not a training trajectory or a replacement for `NeMoGymResponse`. ## Changes - add the shared observation models and exact join logic - extend `ModelCallRecord` with protocol response ID, model metadata, and raw-payload fallbacks - preserve valid records around damaged capture lines and report incomplete captures - preserve upstream status and response evidence when a Model Server call raises - join Agent Server observations with Model Server capture during rollout-record assembly - propagate rollout correlation through standard Agent, Resources, and Model Server calls - exclude observation payloads from aggregate-metrics requests All behavior remains opt-in through the existing observability configuration. ## Scope - No external trajectory serialization is introduced. - Response, reward, token-ID, and log-probability contracts are unchanged. - Missing hierarchy, ownership, and timing evidence is reported rather than estimated. - Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls. - Harness and sandbox-provider producers remain separate follow-ups. ## Validation Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass. ## Stack - [1/7 NVIDIA-NeMo#2114 — Define the rollout observation and correlation contract](NVIDIA-NeMo#2114) - [2/7 NVIDIA-NeMo#2153 — Add Claude Code rollout observations](NVIDIA-NeMo#2153) - [3/7 NVIDIA-NeMo#2115 — Add OpenClaw and PinchBench rollout observations](NVIDIA-NeMo#2115) - [4/7 NVIDIA-NeMo#2117 — Add Hermes rollout observations](NVIDIA-NeMo#2117) - [5/7 NVIDIA-NeMo#2118 — Add Pi rollout observations](NVIDIA-NeMo#2118) - [6/7 NVIDIA-NeMo#2119 — Add OpenCode rollout observations](NVIDIA-NeMo#2119) - [7/7 NVIDIA-NeMo#2120 — Add SWE and OpenHands rollout observations](NVIDIA-NeMo#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…verage (NVIDIA-NeMo#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by NVIDIA-NeMo#2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / NVIDIA-NeMo#1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs NVIDIA-NeMo#2115 and NVIDIA-NeMo#2117-NVIDIA-NeMo#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. NVIDIA-NeMo#2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from NVIDIA-NeMo#2115 through NVIDIA-NeMo#2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | NVIDIA-NeMo#2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | NVIDIA-NeMo#2116 | Claude Code | Superseded by NVIDIA-NeMo#2153 | See NVIDIA-NeMo#2153 and this PR | | NVIDIA-NeMo#2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | NVIDIA-NeMo#2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | NVIDIA-NeMo#2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | NVIDIA-NeMo#2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | NVIDIA-NeMo#2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with NVIDIA-NeMo#2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of NVIDIA-NeMo#1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

github-actions · 2026-08-10T08:54:06Z

🌿 Preview your docs: https://nvidia-preview-feat-rollout-observability-hermes.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

nemo/gym/reference/trajectory-capabilities

## Summary Add OpenClaw rollout observations for both the standalone agent endpoint and the PinchBench benchmark path. OpenClaw is the observation producer in both paths. PinchBench exports retained OpenClaw session artifacts and attaches the resulting OpenClaw observation bundle; it does not define a separate observation format or producer identity. Observations are additive and are emitted only when observability and rollout correlation are enabled. Existing agent responses, rewards, and grading fields are unchanged. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 are evaluated on the correlated Gym Model Server path. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | OpenClaw | V | V | X | O | V | V | V | C3 remains unsupported because OpenClaw does not emit standardized per-turn records. C4 is partial because retained branched sessions are reported but cannot always reconstruct a single model-visible branch. ## Changes - Normalize retained OpenClaw session records into user, assistant, reasoning, tool-call, and tool-result observations. - Reconstruct subagent lineage from retained OpenClaw session stores. - Correlate policy and judge model calls through rollout-prefixed Gym Model Servers when configured. - Preserve independent tool-call timing, status, duration, and output. - Record context compaction and explicit observation gaps. - Preserve direct model endpoint support when no Gym Model Server is configured. - Isolate observation capture failures from agent responses and benchmark results. ## Evidence boundaries - Model-call correlation requires transcript response IDs and a configured Gym Model Server. - OpenClaw does not identify the exact tool call that spawned a child session. - Branches within one session are reported but are not reconstructed as separate invocations. - Missing timestamps, transcripts, or correlation evidence are represented as explicit gaps. ## Validation - OpenClaw and PinchBench suites: 88 passed. - Core trajectory, collector, correlation, base-agent, and model suites: 155 passed. - Producer-to-collector projection verifies model calls, token details, tool output, status, and independent timing in `ng_trajectory`. - Ruff, formatting, compilation, shell, and diff checks passed. ## Follow-up pull requests - [#2117 — Add Hermes rollout observations](#2117) - [#2118 — Add Pi rollout observations](#2118) - [#2119 — Add OpenCode rollout observations](#2119) - [#2120 — Add SWE and OpenHands rollout observations](#2120) - [#2122 — Add Harbor rollout observations](#2122) Foundation work is available in [#2114](#2114) and [#2153](#2153), both merged into `main`. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Add Pi production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Pi's JSON event stream is captured while the process runs. Uncorrelated requests retain the existing response path and provider configuration. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `pi_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Pi does not emit standardized per-turn records. ## Captured evidence - model-visible system prompt, user instruction, assistant output, tool calls, and tool results - model-call references from response IDs when `model_server` is configured - independent tool-call intervals and outcomes, including overlapping calls - context-compaction outcomes, summaries, token counts, retained boundaries, and adjacent model-call references when available - invocation status from Pi's terminal `agent_end` event - partial evidence from incomplete or timed-out runs ## Evidence boundaries - Pi does not expose subagent hierarchy. - Tool timings use harness receipt timestamps rather than executor timestamps. - Incomplete model-call, tool, and compaction pairs are reported as gaps. - Unknown terminal outcomes remain `unknown` with an `invocation_outcome_unavailable` gap. - The integration runs on the host and reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional; direct-provider configuration is unchanged - configured Gym Model Servers use rollout-prefixed routing - response, verification, reward, and token-accounting behavior is unchanged - observation parsing failures do not fail the rollout - the private self-call attachment is removed before verification - `ng_agent_observations` is optional and additive ## Validation - Pi suite: 34 passed - Shared observability, correlation, and collector suites: 82 passed - Prefixed FastAPI response round-trip covered - Ruff, formatting, and diff checks passed ## Related pull requests This is **5/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`.  - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary - emit OpenCode session, conversation, tool, and compaction evidence as `ng_agent_observations` - route Gym Model Server calls through the rollout-prefixed endpoint so model-call capture correlates with the rollout - document OpenCode coverage in the trajectory capability matrix ## Capability coverage | Criterion | Status | Scope | | --- | --- | --- | | C1 | V | Correlated Gym Model Server calls use the standard model-call schema. | | C2 | V | Correlated capture retains standard token fields when provided. | | C3 | X | OpenCode does not emit standardized semantic turns. | | C4 | V | Session artifacts retain model-visible invocation conversations, including compaction behavior. | | C5 | V | Tool output, status, start/end timestamps, and duration project into `ng_trajectory`. | | C6 | V | Each tool call retains its own artifact-derived interval. | | C7 | V | The OpenCode host-agent path emits the shared observation contract. | `V` applies to the correlated Gym Model Server path for C1, C2, and C4, consistent with the capability-matrix definition. ## Evidence boundaries - OpenCode artifacts do not expose stable response IDs, so model calls are not assigned to individual OpenCode invocations. - Compaction token counts and adjacent model-call boundaries are reported as unavailable. - A child invocation is linked to a spawning tool only when the artifact identifies one unambiguous call. - The host-run integration reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional. - Uncorrelated `/v1/responses` payloads retain their existing public shape. - Observation metadata is removed before the resource-server verifier receives the response. - Artifact parsing failures produce observation gaps without failing the rollout. - Existing OpenCode provider configuration is copied, not mutated. ## Validation - 24 OpenCode agent tests - 106 combined OpenCode, base-agent, rollout-observability, and rollout-collection tests - subprocess -> OpenCode SQLite artifact -> observations -> `ng_trajectory` E2E - Ruff, formatting, and diff checks ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2120 SWE/OpenHands producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary - emit standardized `ng_agent_observations` for the legacy SWE OpenCode/OpenHands paths and the decoupled `opencode_sandboxed_agent` + SWE-bench path - correlate OpenCode calls through rollout-prefixed Gym Model Server routes when rollout observability or token capture is active - compose OpenCode invocation/tool evidence with separate agent- and verifier-sandbox observations - preserve direct response behavior, grading outputs, retained artifacts, and `subagent_trajectories` ## What changed ### Decoupled OpenCode / SWE-bench - parse OpenCode SQLite sessions into invocations, parent relationships, cumulative model-visible conversations, tool timing/outcomes, compaction events, and exact response IDs - keep the sandboxed agent independently installable by copying the minimal parser closure locally; it no longer imports the standalone `opencode_agent`, and a fresh-process test enforces that boundary - isolate OpenCode data per observed run, download and parse the database before teardown, and remove the local scratch database afterward - record the connected agent sandbox's real provider/ID and compose verifier-sandbox lifecycle evidence returned by `resources_servers/swebench` - emit observations only when a capture-derived rollout ID exists; direct `/v1/responses` behavior remains unchanged ### Legacy SWE harness - OpenCode records retained session invocations, parent relationships, exact response IDs, and the latest cumulative conversation; rollout-prefixed model calls provide capture and token accounting - OpenHands records its available cumulative root conversation and sandbox evidence, while explicitly reporting that exact model-call correlation is unavailable with the pinned fork - legacy Apptainer records leave `sandbox_id` unset and report `sandbox_identity_unavailable` because the runner exposes no real sandbox handle Observation construction fails open. Missing or malformed evidence becomes an explicit gap; provider sentinel values are not exposed as process exit codes, and unavailable resource or lifecycle measurements remain unset rather than inferred. ## Capability coverage | Producer | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `opencode_sandboxed_agent` | V | V | X | V | V | V | V | | `swe_agents` / OpenCode | V | V | X | V | X | X | V | | `swe_agents` / OpenHands | X | X | X | O | X | X | V | The capability matrix documents these evidence boundaries. Legacy artifacts do not provide standardized semantic turns, authoritative per-tool timing, or independent parallel-tool timing. ## Validation - 34 combined standalone and sandboxed OpenCode tests, including fresh-process import isolation - focused SWE-bench resource-server and legacy SWE-agent tests across disabled, observability-only, token-only, and combined capture states - 114 shared rollout-observability, correlation, and collection regressions rerun after the final rebase - SQLite artifact -> parser -> decoupled `/run` composition test covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup primitives - Ruff, formatting, Python compilation, `git diff --check`, and scoped pre-commit hooks A real artifact-compatibility smoke test used Docker Server 29.6.2 on Linux/aarch64 and the exact `swebench/sweb.eval.x86_64.astropy_1776_astropy-12907` image under x86_64 emulation. OpenCode 1.17.11 was installed only inside the temporary container and ran a real gpt-5.5-backed session whose bash tool executed `printf opencode-observability-smoke`; the actual tool result persisted and `opencode export` succeeded. The WAL-mode database contained 1 session, 3 messages, and 7 parts. After closing/exporting, only `opencode.db` was copied and parsed by the final sandbox-local parser, producing 1 completed invocation, 1 tool call, 0 compactions, and only the expected `model_call_ownership_unavailable` gap. The temporary container was stopped and auto-removed; nothing was installed on the host. ## Limitations - the live smoke used the direct NVIDIA gateway rather than Gym's rollout-prefixed model proxy, so it validates the real OpenCode artifact schema/parser but not model-call ownership or capture joining - OpenCode's final prose stream did not terminate after the tool result and was gracefully interrupted - the full decoupled `/run` + verifier flow was not run live because `DockerProvider` cannot reconnect across the resource-server and agent processes; that path requires OpenSandbox - the SQLite parser is intentionally duplicated to keep the two agent servers dependency-isolated and must remain synchronized ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2119 standalone OpenCode producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf marked this pull request as ready for review July 23, 2026 12:34

Glorf force-pushed the feat/rollout-observability-claude-code branch from cafe10a to 9a4657f Compare July 23, 2026 13:24

Glorf force-pushed the feat/rollout-observability-hermes branch from 0c9188b to 75c70e5 Compare July 23, 2026 13:24

Glorf force-pushed the feat/rollout-observability-claude-code branch from 9a4657f to a442a8d Compare July 23, 2026 14:16

Glorf requested a review from a team as a code owner July 23, 2026 14:16

Glorf force-pushed the feat/rollout-observability-hermes branch from 75c70e5 to f64f5a3 Compare July 23, 2026 14:18

Glorf force-pushed the feat/rollout-observability-claude-code branch from a442a8d to fdc7283 Compare July 23, 2026 18:20

Glorf force-pushed the feat/rollout-observability-hermes branch from f64f5a3 to b7b4597 Compare July 23, 2026 18:20

github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 24, 2026

github-actions Bot mentioned this pull request Jul 24, 2026

[Tracker] Pull request handoff SLA #2067

Open

Glorf force-pushed the feat/rollout-observability-claude-code branch from fdc7283 to 3f622a0 Compare July 24, 2026 16:38

Glorf force-pushed the feat/rollout-observability-hermes branch from b7b4597 to 5334e1e Compare July 24, 2026 16:38

Glorf mentioned this pull request Jul 25, 2026

[rollout-observability][follow-up] Correlate auxiliary Model Server calls #2122

Closed

Glorf changed the title ~~[rollout-observability][4/7] Add Hermes rollout observations~~ [rollout-observability][5/8] Add Hermes rollout observations Jul 25, 2026

Glorf force-pushed the feat/rollout-observability-hermes branch from 5334e1e to 8c1a4b5 Compare July 27, 2026 12:05

Glorf force-pushed the feat/rollout-observability-claude-code branch 2 times, most recently from 3c424af to 82c3dc1 Compare July 27, 2026 17:38

Glorf force-pushed the feat/rollout-observability-hermes branch from 8c1a4b5 to 8b43262 Compare July 27, 2026 18:01

Base automatically changed from feat/rollout-observability-claude-code to feat/rollout-observability-openclaw-pinchbench July 27, 2026 18:01

Glorf changed the title ~~[rollout-observability][5/8] Add Hermes rollout observations~~ [rollout-observability][4/7] Add Hermes rollout observations Jul 27, 2026

sephmard linked an issue Jul 29, 2026 that may be closed by this pull request

Add Hermes rollout observations #2200

Closed

giuliolovisotto mentioned this pull request Aug 4, 2026

[Feat] Port evalpipeline validation and artifact quality gates into Gym #2136

Closed

6 tasks

Glorf mentioned this pull request Aug 4, 2026

[fix] Standardize rollout trajectory records and document producer coverage #2293

Merged

Glorf force-pushed the feat/rollout-observability-openclaw-pinchbench branch from 57d2b1c to 3b2f32b Compare August 6, 2026 08:58

Glorf force-pushed the feat/rollout-observability-hermes branch from 8b43262 to 8cba221 Compare August 10, 2026 08:52

Glorf force-pushed the feat/rollout-observability-hermes branch from 8cba221 to 1266a1d Compare August 10, 2026 10:29

Glorf changed the base branch from feat/rollout-observability-openclaw-pinchbench to main August 10, 2026 10:29

laszkiewiczp reviewed Aug 11, 2026

View reviewed changes

Comment thread responses_api_agents/hermes_agent/observability.py Outdated

laszkiewiczp previously approved these changes Aug 11, 2026

View reviewed changes

github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 11, 2026

Glorf force-pushed the feat/rollout-observability-hermes branch from 1266a1d to c435c05 Compare August 11, 2026 10:37

Glorf added 3 commits August 11, 2026 12:39

Add Hermes rollout observations

de5ac5a

Signed-off-by: Michal Bien <mbien@nvidia.com>

Report unavailable Hermes sandbox usage

7c49b1c

Signed-off-by: Michal Bien <mbien@nvidia.com>

Fix Hermes observation fidelity after stack rebase

ba8c32e

Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf force-pushed the feat/rollout-observability-hermes branch from c435c05 to ba8c32e Compare August 11, 2026 10:41

Remove unreachable Hermes timing gap

98e8aed

Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf dismissed laszkiewiczp’s stale review via 98e8aed August 11, 2026 10:49

laszkiewiczp approved these changes Aug 11, 2026

View reviewed changes

Glorf merged commit 6a2fc8d into main Aug 11, 2026
10 checks passed

Glorf deleted the feat/rollout-observability-hermes branch August 11, 2026 10:58

cwing-nvidia mentioned this pull request Aug 25, 2026

enhancement(hermes_agent): improve rollout observability fidelity #2741

Open

6 tasks

Navigation Menu

Uh oh!

[rollout-observability][4/7] Add Hermes rollout observations - #2117

[rollout-observability][4/7] Add Hermes rollout observations#2117
Glorf merged 4 commits into
mainfrom
feat/rollout-observability-hermes

Glorf commented Jul 23, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Glorf commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Capability coverage

Captured evidence

Evidence boundaries

Compatibility

Validation

Related pull requests

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Glorf commented Jul 23, 2026 •

edited

Loading