[rollout-observability][3/7] Add OpenClaw rollout observations - #2115
Merged
Conversation
This was referenced Jul 23, 2026
Glorf
marked this pull request as ready for review
July 23, 2026 12:34
Glorf
force-pushed
the
feat/rollout-observability-openclaw-pinchbench
branch
from
July 23, 2026 13:24
035493b to
93657c3
Compare
Glorf
force-pushed
the
feat/rollout-observability-base
branch
2 times, most recently
from
July 23, 2026 14:13
4fd8fdf to
03b751f
Compare
Glorf
force-pushed
the
feat/rollout-observability-openclaw-pinchbench
branch
2 times, most recently
from
July 23, 2026 18:20
6987c61 to
3cd07b8
Compare
Glorf
force-pushed
the
feat/rollout-observability-openclaw-pinchbench
branch
from
July 24, 2026 16:38
3cd07b8 to
5d5ec46
Compare
Glorf
changed the base branch from
feat/rollout-observability-base
to
feat/rollout-observability-aux-model-correlation
July 25, 2026 10:52
Glorf
force-pushed
the
feat/rollout-observability-aux-model-correlation
branch
from
July 27, 2026 12:05
e0856b3 to
f8b97bf
Compare
Glorf
force-pushed
the
feat/rollout-observability-openclaw-pinchbench
branch
from
July 27, 2026 12:05
5d5ec46 to
6b747cf
Compare
Glorf
force-pushed
the
feat/rollout-observability-aux-model-correlation
branch
from
July 27, 2026 17:37
f8b97bf to
0ca9488
Compare
Glorf
force-pushed
the
feat/rollout-observability-openclaw-pinchbench
branch
from
July 27, 2026 18:01
6b747cf to
57d2b1c
Compare
Glorf
changed the base branch from
feat/rollout-observability-aux-model-correlation
to
feat/rollout-observability-claude-code
July 27, 2026 18:47
Glorf
force-pushed
the
feat/rollout-observability-claude-code
branch
2 times, most recently
from
July 28, 2026 16:33
85de508 to
9af9174
Compare
Glorf
added a commit
that referenced
this pull request
Jul 28, 2026
…lation contract (#2114) ## Summary This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage. It also joins Agent Server observations with Model Server call capture: ```text Agent or harness ──> ng_agent_observations ─┐ ├─ exact correlation ─> rollout record Model Servers ─────> ng_model_call_capture ─┘ ``` This is the base of the rollout-observability work and follows up on #1867. ## Contract | Evidence | Representation | |---|---| | Root agent and subagents | `AgentInvocation` | | Model calls owned by an invocation | `ModelCallRef` | | Tool execution and parallel timing | `ToolCallObservation` | | Context compaction | `ContextCompactionObservation` | | Sandbox outcome and lifetime usage | `SandboxObservation` | | Missing or unsupported evidence | `ObservationGap` | Model calls are joined through `model_call_id`, or the exact `(model_ref, response_id)` pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps. This is an observability view, not a training trajectory or a replacement for `NeMoGymResponse`. ## Changes - add the shared observation models and exact join logic - extend `ModelCallRecord` with protocol response ID, model metadata, and raw-payload fallbacks - preserve valid records around damaged capture lines and report incomplete captures - preserve upstream status and response evidence when a Model Server call raises - join Agent Server observations with Model Server capture during rollout-record assembly - propagate rollout correlation through standard Agent, Resources, and Model Server calls - exclude observation payloads from aggregate-metrics requests All behavior remains opt-in through the existing observability configuration. ## Scope - No external trajectory serialization is introduced. - Response, reward, token-ID, and log-probability contracts are unchanged. - Missing hierarchy, ownership, and timing evidence is reported rather than estimated. - Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls. - Harness and sandbox-provider producers remain separate follow-ups. ## Validation Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass. ## Stack - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw and PinchBench rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
…verage (NVIDIA-NeMo#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by NVIDIA-NeMo#2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / NVIDIA-NeMo#1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs NVIDIA-NeMo#2115 and NVIDIA-NeMo#2117-NVIDIA-NeMo#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. NVIDIA-NeMo#2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from NVIDIA-NeMo#2115 through NVIDIA-NeMo#2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | NVIDIA-NeMo#2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | NVIDIA-NeMo#2116 | Claude Code | Superseded by NVIDIA-NeMo#2153 | See NVIDIA-NeMo#2153 and this PR | | NVIDIA-NeMo#2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | NVIDIA-NeMo#2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | NVIDIA-NeMo#2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | NVIDIA-NeMo#2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | NVIDIA-NeMo#2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with NVIDIA-NeMo#2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of NVIDIA-NeMo#1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
mlazuka
reviewed
Aug 7, 2026
mlazuka
reviewed
Aug 7, 2026
Contributor
Author
|
@kajalj22 can I have the CI review? (it's only .secrets.baseline file modification) |
kajalj22
previously approved these changes
Aug 10, 2026
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
force-pushed
the
feat/rollout-observability-openclaw-pinchbench
branch
from
August 10, 2026 15:25
d0bf618 to
1834d39
Compare
Glorf
enabled auto-merge (squash)
August 10, 2026 16:30
kajalj22
self-requested a review
August 10, 2026 17:12
kajalj22
approved these changes
Aug 10, 2026
Glorf
added a commit
that referenced
this pull request
Aug 11, 2026
## Summary Add Hermes production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Hermes observations use instance-local hooks. Uncorrelated requests retain the existing response path. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `hermes_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Hermes does not emit standardized per-turn records. ## Captured evidence - root and delegated-agent conversations, including system prompts and plain reasoning - parent and `delegate_task` spawn relationships when directly observed - model-call references from response IDs - model-visible tool outputs, execution status, timestamps, duration, and independent concurrent-call intervals - context-compaction outcomes and available pre/post token estimates - invocation outcomes Existing Hermes callbacks are chained. Observation failures do not mask agent failures. ## Evidence boundaries - Missing Hermes hooks and unattributed child spawns are reported as explicit gaps. - Opaque `reasoning_details` are not normalized. - Compaction summaries and exact adjacent model calls are unavailable; failed compactions do not report a post-compaction token count. - Local terminal execution reports `no_sandbox_runtime`. - External terminal backends report `sandbox_observation_unavailable` because Gym does not receive their lifecycle or resource telemetry. ## Compatibility - observation capture is enabled only for correlated rollouts - the private self-call attachment is removed before verification - verifier payloads, rewards, and token-bearing output retain existing semantics - `ng_agent_observations` is optional and additive ## Validation - Hermes suite: 44 passed - compatibility suite: 239 passed - local E2E: pinned Hermes agent, concurrent tools, model-call correlation, normalized conversation, and trajectory projection - Ruff, formatting, compile, and diff checks passed ## Related pull requests This is **4/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`. <!-- stack-links --> - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
added a commit
that referenced
this pull request
Aug 11, 2026
## Summary Add Pi production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Pi's JSON event stream is captured while the process runs. Uncorrelated requests retain the existing response path and provider configuration. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `pi_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Pi does not emit standardized per-turn records. ## Captured evidence - model-visible system prompt, user instruction, assistant output, tool calls, and tool results - model-call references from response IDs when `model_server` is configured - independent tool-call intervals and outcomes, including overlapping calls - context-compaction outcomes, summaries, token counts, retained boundaries, and adjacent model-call references when available - invocation status from Pi's terminal `agent_end` event - partial evidence from incomplete or timed-out runs ## Evidence boundaries - Pi does not expose subagent hierarchy. - Tool timings use harness receipt timestamps rather than executor timestamps. - Incomplete model-call, tool, and compaction pairs are reported as gaps. - Unknown terminal outcomes remain `unknown` with an `invocation_outcome_unavailable` gap. - The integration runs on the host and reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional; direct-provider configuration is unchanged - configured Gym Model Servers use rollout-prefixed routing - response, verification, reward, and token-accounting behavior is unchanged - observation parsing failures do not fail the rollout - the private self-call attachment is removed before verification - `ng_agent_observations` is optional and additive ## Validation - Pi suite: 34 passed - Shared observability, correlation, and collector suites: 82 passed - Prefixed FastAPI response round-trip covered - Ruff, formatting, and diff checks passed ## Related pull requests This is **5/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`. <!-- stack-links --> - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
added a commit
that referenced
this pull request
Aug 12, 2026
## Summary - emit OpenCode session, conversation, tool, and compaction evidence as `ng_agent_observations` - route Gym Model Server calls through the rollout-prefixed endpoint so model-call capture correlates with the rollout - document OpenCode coverage in the trajectory capability matrix ## Capability coverage | Criterion | Status | Scope | | --- | --- | --- | | C1 | V | Correlated Gym Model Server calls use the standard model-call schema. | | C2 | V | Correlated capture retains standard token fields when provided. | | C3 | X | OpenCode does not emit standardized semantic turns. | | C4 | V | Session artifacts retain model-visible invocation conversations, including compaction behavior. | | C5 | V | Tool output, status, start/end timestamps, and duration project into `ng_trajectory`. | | C6 | V | Each tool call retains its own artifact-derived interval. | | C7 | V | The OpenCode host-agent path emits the shared observation contract. | `V` applies to the correlated Gym Model Server path for C1, C2, and C4, consistent with the capability-matrix definition. ## Evidence boundaries - OpenCode artifacts do not expose stable response IDs, so model calls are not assigned to individual OpenCode invocations. - Compaction token counts and adjacent model-call boundaries are reported as unavailable. - A child invocation is linked to a spawning tool only when the artifact identifies one unambiguous call. - The host-run integration reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional. - Uncorrelated `/v1/responses` payloads retain their existing public shape. - Observation metadata is removed before the resource-server verifier receives the response. - Artifact parsing failures produce observation gaps without failing the rollout. - Existing OpenCode provider configuration is copied, not mutated. ## Validation - 24 OpenCode agent tests - 106 combined OpenCode, base-agent, rollout-observability, and rollout-collection tests - subprocess -> OpenCode SQLite artifact -> observations -> `ng_trajectory` E2E - Ruff, formatting, and diff checks ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2120 SWE/OpenHands producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - emit standardized `ng_agent_observations` for the legacy SWE OpenCode/OpenHands paths and the decoupled `opencode_sandboxed_agent` + SWE-bench path - correlate OpenCode calls through rollout-prefixed Gym Model Server routes when rollout observability or token capture is active - compose OpenCode invocation/tool evidence with separate agent- and verifier-sandbox observations - preserve direct response behavior, grading outputs, retained artifacts, and `subagent_trajectories` ## What changed ### Decoupled OpenCode / SWE-bench - parse OpenCode SQLite sessions into invocations, parent relationships, cumulative model-visible conversations, tool timing/outcomes, compaction events, and exact response IDs - keep the sandboxed agent independently installable by copying the minimal parser closure locally; it no longer imports the standalone `opencode_agent`, and a fresh-process test enforces that boundary - isolate OpenCode data per observed run, download and parse the database before teardown, and remove the local scratch database afterward - record the connected agent sandbox's real provider/ID and compose verifier-sandbox lifecycle evidence returned by `resources_servers/swebench` - emit observations only when a capture-derived rollout ID exists; direct `/v1/responses` behavior remains unchanged ### Legacy SWE harness - OpenCode records retained session invocations, parent relationships, exact response IDs, and the latest cumulative conversation; rollout-prefixed model calls provide capture and token accounting - OpenHands records its available cumulative root conversation and sandbox evidence, while explicitly reporting that exact model-call correlation is unavailable with the pinned fork - legacy Apptainer records leave `sandbox_id` unset and report `sandbox_identity_unavailable` because the runner exposes no real sandbox handle Observation construction fails open. Missing or malformed evidence becomes an explicit gap; provider sentinel values are not exposed as process exit codes, and unavailable resource or lifecycle measurements remain unset rather than inferred. ## Capability coverage | Producer | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `opencode_sandboxed_agent` | V | V | X | V | V | V | V | | `swe_agents` / OpenCode | V | V | X | V | X | X | V | | `swe_agents` / OpenHands | X | X | X | O | X | X | V | The capability matrix documents these evidence boundaries. Legacy artifacts do not provide standardized semantic turns, authoritative per-tool timing, or independent parallel-tool timing. ## Validation - 34 combined standalone and sandboxed OpenCode tests, including fresh-process import isolation - focused SWE-bench resource-server and legacy SWE-agent tests across disabled, observability-only, token-only, and combined capture states - 114 shared rollout-observability, correlation, and collection regressions rerun after the final rebase - SQLite artifact -> parser -> decoupled `/run` composition test covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup primitives - Ruff, formatting, Python compilation, `git diff --check`, and scoped pre-commit hooks A real artifact-compatibility smoke test used Docker Server 29.6.2 on Linux/aarch64 and the exact `swebench/sweb.eval.x86_64.astropy_1776_astropy-12907` image under x86_64 emulation. OpenCode 1.17.11 was installed only inside the temporary container and ran a real gpt-5.5-backed session whose bash tool executed `printf opencode-observability-smoke`; the actual tool result persisted and `opencode export` succeeded. The WAL-mode database contained 1 session, 3 messages, and 7 parts. After closing/exporting, only `opencode.db` was copied and parsed by the final sandbox-local parser, producing 1 completed invocation, 1 tool call, 0 compactions, and only the expected `model_call_ownership_unavailable` gap. The temporary container was stopped and auto-removed; nothing was installed on the host. ## Limitations - the live smoke used the direct NVIDIA gateway rather than Gym's rollout-prefixed model proxy, so it validates the real OpenCode artifact schema/parser but not model-call ownership or capture joining - OpenCode's final prose stream did not terminate after the tool result and was gracefully interrupted - the full decoupled `/run` + verifier flow was not run live because `DockerProvider` cannot reconnect across the resource-server and agent processes; that path requires OpenSandbox - the SQLite parser is intentionally duplicated to keep the two agent servers dependency-isolated and must remain synchronized ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2119 standalone OpenCode producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add OpenClaw rollout observations for both the standalone agent endpoint and the PinchBench benchmark path.
OpenClaw is the observation producer in both paths. PinchBench exports retained OpenClaw session artifacts and attaches the resulting OpenClaw observation bundle; it does not define a separate observation format or producer identity.
Observations are additive and are emitted only when observability and rollout correlation are enabled. Existing agent responses, rewards, and grading fields are unchanged.
Capability coverage
V= supported,O= partial or path-dependent,X= unavailable. C1, C2, and C4 are evaluated on the correlated Gym Model Server path.C3 remains unsupported because OpenClaw does not emit standardized per-turn records. C4 is partial because retained branched sessions are reported but cannot always reconstruct a single model-visible branch.
Changes
Evidence boundaries
Validation
ng_trajectory.Follow-up pull requests
Foundation work is available in #2114 and #2153, both merged into
main.