Skip to content

[rollout-observability][3/7] Add OpenClaw rollout observations - #2115

Merged
Glorf merged 10 commits into
mainfrom
feat/rollout-observability-openclaw-pinchbench
Aug 10, 2026
Merged

[rollout-observability][3/7] Add OpenClaw rollout observations#2115
Glorf merged 10 commits into
mainfrom
feat/rollout-observability-openclaw-pinchbench

Conversation

@Glorf

@Glorf Glorf commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Add OpenClaw rollout observations for both the standalone agent endpoint and the PinchBench benchmark path.

OpenClaw is the observation producer in both paths. PinchBench exports retained OpenClaw session artifacts and attaches the resulting OpenClaw observation bundle; it does not define a separate observation format or producer identity.

Observations are additive and are emitted only when observability and rollout correlation are enabled. Existing agent responses, rewards, and grading fields are unchanged.

Capability coverage

V = supported, O = partial or path-dependent, X = unavailable. C1, C2, and C4 are evaluated on the correlated Gym Model Server path.

Agent C1 C2 C3 C4 C5 C6 C7
OpenClaw V V X O V V V

C3 remains unsupported because OpenClaw does not emit standardized per-turn records. C4 is partial because retained branched sessions are reported but cannot always reconstruct a single model-visible branch.

Changes

  • Normalize retained OpenClaw session records into user, assistant, reasoning, tool-call, and tool-result observations.
  • Reconstruct subagent lineage from retained OpenClaw session stores.
  • Correlate policy and judge model calls through rollout-prefixed Gym Model Servers when configured.
  • Preserve independent tool-call timing, status, duration, and output.
  • Record context compaction and explicit observation gaps.
  • Preserve direct model endpoint support when no Gym Model Server is configured.
  • Isolate observation capture failures from agent responses and benchmark results.

Evidence boundaries

  • Model-call correlation requires transcript response IDs and a configured Gym Model Server.
  • OpenClaw does not identify the exact tool call that spawned a child session.
  • Branches within one session are reported but are not reconstructed as separate invocations.
  • Missing timestamps, transcripts, or correlation evidence are represented as explicit gaps.

Validation

  • OpenClaw and PinchBench suites: 88 passed.
  • Core trajectory, collector, correlation, base-agent, and model suites: 155 passed.
  • Producer-to-collector projection verifies model calls, token details, tool output, status, and independent timing in ng_trajectory.
  • Ruff, formatting, compilation, shell, and diff checks passed.

Follow-up pull requests

Foundation work is available in #2114 and #2153, both merged into main.

@copy-pr-bot

copy-pr-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Glorf
Glorf marked this pull request as ready for review July 23, 2026 12:34
@Glorf
Glorf force-pushed the feat/rollout-observability-openclaw-pinchbench branch from 035493b to 93657c3 Compare July 23, 2026 13:24
@Glorf
Glorf requested a review from a team as a code owner July 23, 2026 13:24
@Glorf
Glorf force-pushed the feat/rollout-observability-base branch 2 times, most recently from 4fd8fdf to 03b751f Compare July 23, 2026 14:13
@Glorf
Glorf force-pushed the feat/rollout-observability-openclaw-pinchbench branch 2 times, most recently from 6987c61 to 3cd07b8 Compare July 23, 2026 18:20
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 24, 2026
@Glorf
Glorf force-pushed the feat/rollout-observability-openclaw-pinchbench branch from 3cd07b8 to 5d5ec46 Compare July 24, 2026 16:38
@Glorf Glorf changed the title [rollout-observability][2/7] Add OpenClaw and PinchBench rollout observations [rollout-observability][3/8] Add OpenClaw and PinchBench rollout observations Jul 25, 2026
@Glorf
Glorf changed the base branch from feat/rollout-observability-base to feat/rollout-observability-aux-model-correlation July 25, 2026 10:52
@Glorf
Glorf force-pushed the feat/rollout-observability-aux-model-correlation branch from e0856b3 to f8b97bf Compare July 27, 2026 12:05
@Glorf
Glorf force-pushed the feat/rollout-observability-openclaw-pinchbench branch from 5d5ec46 to 6b747cf Compare July 27, 2026 12:05
@Glorf
Glorf force-pushed the feat/rollout-observability-aux-model-correlation branch from f8b97bf to 0ca9488 Compare July 27, 2026 17:37
@Glorf
Glorf force-pushed the feat/rollout-observability-openclaw-pinchbench branch from 6b747cf to 57d2b1c Compare July 27, 2026 18:01
@Glorf
Glorf changed the base branch from feat/rollout-observability-aux-model-correlation to feat/rollout-observability-claude-code July 27, 2026 18:47
@Glorf Glorf changed the title [rollout-observability][3/8] Add OpenClaw and PinchBench rollout observations [rollout-observability][3/7] Add OpenClaw and PinchBench rollout observations Jul 27, 2026
@Glorf
Glorf force-pushed the feat/rollout-observability-claude-code branch 2 times, most recently from 85de508 to 9af9174 Compare July 28, 2026 16:33
Glorf added a commit that referenced this pull request Jul 28, 2026
…lation contract (#2114)

## Summary

This PR defines a shared contract for rollout evidence that is not
visible at the Model Server boundary: agent and subagent structure, tool
execution intervals, context compaction, and sandbox outcome/resource
usage.

It also joins Agent Server observations with Model Server call capture:

```text
Agent or harness ──> ng_agent_observations ─┐
                                            ├─ exact correlation ─> rollout record
Model Servers ─────> ng_model_call_capture ─┘
```

This is the base of the rollout-observability work and follows up on
#1867.

## Contract

| Evidence | Representation |
|---|---|
| Root agent and subagents | `AgentInvocation` |
| Model calls owned by an invocation | `ModelCallRef` |
| Tool execution and parallel timing | `ToolCallObservation` |
| Context compaction | `ContextCompactionObservation` |
| Sandbox outcome and lifetime usage | `SandboxObservation` |
| Missing or unsupported evidence | `ObservationGap` |

Model calls are joined through `model_call_id`, or the exact
`(model_ref, response_id)` pair when the harness exposes the protocol
response ID. A compaction may own exact model calls from its enclosing
invocation; boundary references do not imply ownership. Ambiguous,
unmatched, conflicting, and unowned calls remain visible as gaps.

This is an observability view, not a training trajectory or a
replacement for `NeMoGymResponse`.

## Changes

- add the shared observation models and exact join logic
- extend `ModelCallRecord` with protocol response ID, model metadata,
and raw-payload fallbacks
- preserve valid records around damaged capture lines and report
incomplete captures
- preserve upstream status and response evidence when a Model Server
call raises
- join Agent Server observations with Model Server capture during
rollout-record assembly
- propagate rollout correlation through standard Agent, Resources, and
Model Server calls
- exclude observation payloads from aggregate-metrics requests

All behavior remains opt-in through the existing observability
configuration.

## Scope

- No external trajectory serialization is introduced.
- Response, reward, token-ID, and log-probability contracts are
unchanged.
- Missing hierarchy, ownership, and timing evidence is reported rather
than estimated.
- Sandbox usage describes the enclosing sandbox, not individual or
overlapping tool calls.
- Harness and sandbox-provider producers remain separate follow-ups.

## Validation

Focused observation, correlation, capture, streaming,
rollout-attachment, resource-server, and upstream-failure tests pass.
Ruff, formatting, and diff checks pass.

## Stack

- [1/7 #2114 — Define the rollout observation and correlation
contract](#2114)
- [2/7 #2153 — Add Claude Code rollout
observations](#2153)
- [3/7 #2115 — Add OpenClaw and PinchBench rollout
observations](#2115)
- [4/7 #2117 — Add Hermes rollout
observations](#2117)
- [5/7 #2118 — Add Pi rollout
observations](#2118)
- [6/7 #2119 — Add OpenCode rollout
observations](#2119)
- [7/7 #2120 — Add SWE and OpenHands rollout
observations](#2120)

---------

Signed-off-by: Michal Bien <mbien@nvidia.com>
OlegSudakov pushed a commit to OlegSudakov/Gym that referenced this pull request Aug 7, 2026
…verage (NVIDIA-NeMo#2293)

## Summary

- add a versioned `ng_trajectory` projection for normalized model calls,
token statistics, semantic turns, tool observations, and
invocation-scoped model-visible history
- preserve provider-reported cached, reasoning, and total token usage
across capture, conversion, and multi-turn agent loops
- preserve Responses lifecycle status as `response_status` without
reinterpreting dialect-specific `finish_reason`
- add an opt-in reference trajectory producer for Simple Agent and
document current V/O/X coverage for C1-C7 across all 34 agents

This PR establishes the shared schema and one reference producer. The
capability matrix reports current support; it does not claim that every
producer satisfies C1-C7.

## Correctness and compatibility

- collector-derived task and rollout identities are canonical; producer
mismatches are recorded and turn identities are normalized without
dropping the producer trajectory
- producer invocations remain authoritative on duplicate IDs,
observation-only invocations are appended, and model and tool records
are merged only by exact identifiers
- projection failures retain an explicit gap and captured request and
response payloads; successful projections remove the duplicate capture
payloads
- Simple Agent records the actual per-turn model input, cumulative
tool-step count, and an explicit gap when resolution is unavailable
- no existing fields are removed; response lifecycle status, provider
token details and totals, and cached-token aggregation are intentional
correctness changes; `ng_trajectory` is additive
- Simple Agent retains self-dispatch when observability is disabled or
`responses()` is overridden
- trajectory collection remains gated by observability and independent
of the token-ID capture correlation introduced by NVIDIA-NeMo#2124
- captured request and response payloads are retained in persisted
`ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits
those copies
- W&B rollout tables omit `ng_trajectory` and model-call request and
response payloads during collection and reverification
- payload projection has no separate size cap or trajectory-specific
opt-out; model-call capture remains opt-in

## NVBug alignment

- NVBug 6535274 is addressed with its preferred lossless option:
completed Responses calls preserve `response_status="completed"`;
`finish_reason` remains unset when the dialect does not provide one. The
`inference_provider` path now uses the shared Chat-to-Responses
converter, so the fix applies to the backend named in the report.
- This is the schema and reference-producer phase of NVBug 6555643 /
NVIDIA-NeMo#1867. The capability matrix marks unsupported and path-dependent
producers as `X` or `O`; it is not a claim of full producer coverage.

## Follow-up pull requests

PRs NVIDIA-NeMo#2115 and NVIDIA-NeMo#2117-NVIDIA-NeMo#2120 are stacked; later PRs include earlier stack
changes. Open PRs describe planned evidence and do not affect the
current capability matrix. NVIDIA-NeMo#2153 provides Claude Code tool timing and
status; this PR joins tool results to those observations. No available
PR from NVIDIA-NeMo#2115 through NVIDIA-NeMo#2122 adds canonical C3 turns.

| PR | Producer or path | Evidence added | Remaining gap |
|---|---|---|---|
| NVIDIA-NeMo#2115 | OpenClaw, PinchBench | Correlated model calls, conversations,
parallel tool timing, and sandbox observations | No C3 turns |
| NVIDIA-NeMo#2116 | Claude Code | Superseded by NVIDIA-NeMo#2153 | See NVIDIA-NeMo#2153 and this PR |
| NVIDIA-NeMo#2117 | Hermes | Parallel tool timing and agent observations | No C3
turns |
| NVIDIA-NeMo#2118 | Pi | Correlated model calls, conversations, and parallel tool
timing | No C3 turns |
| NVIDIA-NeMo#2119 | OpenCode | Rollout-level model calls, retained conversations,
and parallel tool timing | No exact per-invocation model-call ownership;
no C3 turns |
| NVIDIA-NeMo#2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus
retained conversations and sandbox observations for both paths |
OpenHands calls, tool timing, and C3 turns remain unavailable |
| #2121 | — | No PR exists | — |
| NVIDIA-NeMo#2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and
Tau2 model-call correlation is already present | No agent turns or tool
observations |

## Validation

- 354 capture, conversion, trajectory, collector, reverification,
Fern-link, and inference-provider tests passed, plus 5 subtests
- 10 Simple Agent producer and dispatch tests passed
- in-process endpoint-to-record test passed: prefixed Responses request
→ `inference_provider` → capture middleware → `CaptureStore` →
`ModelCallRecord`
- producer-to-collector-to-JSON trajectory round trip passed with both
available and unavailable resolution status
- 110 vLLM model and Responses conversion tests passed
- 67 Claude Code observation tests and 4 LabBench redaction tests passed
- the trajectory patch merges cleanly with NVIDIA-NeMo#2124; combined-stack
token-capture and trajectory tests passed
- Ruff check, Ruff format check, and `git diff --check` passed

Part of NVIDIA-NeMo#1867.

---------

Signed-off-by: Michal Bien <mbien@nvidia.com>
Comment thread responses_api_agents/pinchbench/app.py
Comment thread responses_api_agents/openclaw_agent/observability.py
mlazuka
mlazuka previously approved these changes Aug 10, 2026

@mlazuka mlazuka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@Glorf
Glorf requested a review from kajalj22 August 10, 2026 09:48
@Glorf

Glorf commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@kajalj22 can I have the CI review? (it's only .secrets.baseline file modification)

kajalj22
kajalj22 previously approved these changes Aug 10, 2026
Glorf added 9 commits August 10, 2026 17:06
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
@Glorf
Glorf dismissed stale reviews from kajalj22 and mlazuka via 1834d39 August 10, 2026 15:25
@Glorf
Glorf force-pushed the feat/rollout-observability-openclaw-pinchbench branch from d0bf618 to 1834d39 Compare August 10, 2026 15:25

@mlazuka mlazuka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Glorf
Glorf enabled auto-merge (squash) August 10, 2026 16:30
@kajalj22
kajalj22 self-requested a review August 10, 2026 17:12
@Glorf
Glorf merged commit abc21eb into main Aug 10, 2026
15 of 17 checks passed
@Glorf
Glorf deleted the feat/rollout-observability-openclaw-pinchbench branch August 10, 2026 17:12
Glorf added a commit that referenced this pull request Aug 11, 2026
## Summary

Add Hermes production of `ng_agent_observations` for correlated rollouts
and update the trajectory capability matrix.

Hermes observations use instance-local hooks. Uncorrelated requests
retain the existing response path.

## Capability coverage

`V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1,
C2, and C4 require correlated Gym Model Server capture.

| Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `hermes_agent` | V | V | X | V | V | V | V |

C3 remains unavailable because Hermes does not emit standardized
per-turn records.

## Captured evidence

- root and delegated-agent conversations, including system prompts and
plain reasoning
- parent and `delegate_task` spawn relationships when directly observed
- model-call references from response IDs
- model-visible tool outputs, execution status, timestamps, duration,
and independent concurrent-call intervals
- context-compaction outcomes and available pre/post token estimates
- invocation outcomes

Existing Hermes callbacks are chained. Observation failures do not mask
agent failures.

## Evidence boundaries

- Missing Hermes hooks and unattributed child spawns are reported as
explicit gaps.
- Opaque `reasoning_details` are not normalized.
- Compaction summaries and exact adjacent model calls are unavailable;
failed compactions do not report a post-compaction token count.
- Local terminal execution reports `no_sandbox_runtime`.
- External terminal backends report `sandbox_observation_unavailable`
because Gym does not receive their lifecycle or resource telemetry.

## Compatibility

- observation capture is enabled only for correlated rollouts
- the private self-call attachment is removed before verification
- verifier payloads, rewards, and token-bearing output retain existing
semantics
- `ng_agent_observations` is optional and additive

## Validation

- Hermes suite: 44 passed
- compatibility suite: 239 passed
- local E2E: pinned Hermes agent, concurrent tools, model-call
correlation, normalized conversation, and trajectory projection
- Ruff, formatting, compile, and diff checks passed

## Related pull requests

This is **4/7** in the rollout-observability series. Producer PRs 4/7
through 7/7 are independently based on `main`.

<!-- stack-links -->

- [1/7 #2114 — Define the rollout observation and correlation
contract](#2114)
- [2/7 #2153 — Add Claude Code rollout
observations](#2153)
- [3/7 #2115 — Add OpenClaw rollout
observations](#2115)
- [4/7 #2117 — Add Hermes rollout
observations](#2117)
- [5/7 #2118 — Add Pi rollout
observations](#2118)
- [6/7 #2119 — Add OpenCode rollout
observations](#2119)
- [7/7 #2120 — Add SWE and OpenHands rollout
observations](#2120)

---------

Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf added a commit that referenced this pull request Aug 11, 2026
## Summary

Add Pi production of `ng_agent_observations` for correlated rollouts and
update the trajectory capability matrix.

Pi's JSON event stream is captured while the process runs. Uncorrelated
requests retain the existing response path and provider configuration.

## Capability coverage

`V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1,
C2, and C4 require correlated Gym Model Server capture.

| Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `pi_agent` | V | V | X | V | V | V | V |

C3 remains unavailable because Pi does not emit standardized per-turn
records.

## Captured evidence

- model-visible system prompt, user instruction, assistant output, tool
calls, and tool results
- model-call references from response IDs when `model_server` is
configured
- independent tool-call intervals and outcomes, including overlapping
calls
- context-compaction outcomes, summaries, token counts, retained
boundaries, and adjacent model-call references when available
- invocation status from Pi's terminal `agent_end` event
- partial evidence from incomplete or timed-out runs

## Evidence boundaries

- Pi does not expose subagent hierarchy.
- Tool timings use harness receipt timestamps rather than executor
timestamps.
- Incomplete model-call, tool, and compaction pairs are reported as
gaps.
- Unknown terminal outcomes remain `unknown` with an
`invocation_outcome_unavailable` gap.
- The integration runs on the host and reports `no_sandbox_runtime`.

## Compatibility

- `model_server` remains optional; direct-provider configuration is
unchanged
- configured Gym Model Servers use rollout-prefixed routing
- response, verification, reward, and token-accounting behavior is
unchanged
- observation parsing failures do not fail the rollout
- the private self-call attachment is removed before verification
- `ng_agent_observations` is optional and additive

## Validation

- Pi suite: 34 passed
- Shared observability, correlation, and collector suites: 82 passed
- Prefixed FastAPI response round-trip covered
- Ruff, formatting, and diff checks passed

## Related pull requests

This is **5/7** in the rollout-observability series. Producer PRs 4/7
through 7/7 are independently based on `main`.

<!-- stack-links -->

- [1/7 #2114 — Define the rollout observation and correlation
contract](#2114)
- [2/7 #2153 — Add Claude Code rollout
observations](#2153)
- [3/7 #2115 — Add OpenClaw rollout
observations](#2115)
- [4/7 #2117 — Add Hermes rollout
observations](#2117)
- [5/7 #2118 — Add Pi rollout
observations](#2118)
- [6/7 #2119 — Add OpenCode rollout
observations](#2119)
- [7/7 #2120 — Add SWE and OpenHands rollout
observations](#2120)

---------

Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf added a commit that referenced this pull request Aug 12, 2026
## Summary

- emit OpenCode session, conversation, tool, and compaction evidence as
`ng_agent_observations`
- route Gym Model Server calls through the rollout-prefixed endpoint so
model-call capture correlates with the rollout
- document OpenCode coverage in the trajectory capability matrix

## Capability coverage

| Criterion | Status | Scope |
| --- | --- | --- |
| C1 | V | Correlated Gym Model Server calls use the standard model-call
schema. |
| C2 | V | Correlated capture retains standard token fields when
provided. |
| C3 | X | OpenCode does not emit standardized semantic turns. |
| C4 | V | Session artifacts retain model-visible invocation
conversations, including compaction behavior. |
| C5 | V | Tool output, status, start/end timestamps, and duration
project into `ng_trajectory`. |
| C6 | V | Each tool call retains its own artifact-derived interval. |
| C7 | V | The OpenCode host-agent path emits the shared observation
contract. |

`V` applies to the correlated Gym Model Server path for C1, C2, and C4,
consistent with the capability-matrix definition.

## Evidence boundaries

- OpenCode artifacts do not expose stable response IDs, so model calls
are not assigned to individual OpenCode invocations.
- Compaction token counts and adjacent model-call boundaries are
reported as unavailable.
- A child invocation is linked to a spawning tool only when the artifact
identifies one unambiguous call.
- The host-run integration reports `no_sandbox_runtime`.

## Compatibility

- `model_server` remains optional.
- Uncorrelated `/v1/responses` payloads retain their existing public
shape.
- Observation metadata is removed before the resource-server verifier
receives the response.
- Artifact parsing failures produce observation gaps without failing the
rollout.
- Existing OpenCode provider configuration is copied, not mutated.

## Validation

- 24 OpenCode agent tests
- 106 combined OpenCode, base-agent, rollout-observability, and
rollout-collection tests
- subprocess -> OpenCode SQLite artifact -> observations ->
`ng_trajectory` E2E
- Ruff, formatting, and diff checks

## Related rollout-observability work

- #2114 shared observation and correlation contract
- #2153 Claude Code producer
- #2115 OpenClaw producer
- #2117 Hermes producer
- #2118 Pi producer
- #2120 SWE/OpenHands producer

---------

Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf added a commit that referenced this pull request Aug 25, 2026
## Summary

- emit standardized `ng_agent_observations` for the legacy SWE
OpenCode/OpenHands paths and the decoupled `opencode_sandboxed_agent` +
SWE-bench path
- correlate OpenCode calls through rollout-prefixed Gym Model Server
routes when rollout observability or token capture is active
- compose OpenCode invocation/tool evidence with separate agent- and
verifier-sandbox observations
- preserve direct response behavior, grading outputs, retained
artifacts, and `subagent_trajectories`

## What changed

### Decoupled OpenCode / SWE-bench

- parse OpenCode SQLite sessions into invocations, parent relationships,
cumulative model-visible conversations, tool timing/outcomes, compaction
events, and exact response IDs
- keep the sandboxed agent independently installable by copying the
minimal parser closure locally; it no longer imports the standalone
`opencode_agent`, and a fresh-process test enforces that boundary
- isolate OpenCode data per observed run, download and parse the
database before teardown, and remove the local scratch database
afterward
- record the connected agent sandbox's real provider/ID and compose
verifier-sandbox lifecycle evidence returned by
`resources_servers/swebench`
- emit observations only when a capture-derived rollout ID exists;
direct `/v1/responses` behavior remains unchanged

### Legacy SWE harness

- OpenCode records retained session invocations, parent relationships,
exact response IDs, and the latest cumulative conversation;
rollout-prefixed model calls provide capture and token accounting
- OpenHands records its available cumulative root conversation and
sandbox evidence, while explicitly reporting that exact model-call
correlation is unavailable with the pinned fork
- legacy Apptainer records leave `sandbox_id` unset and report
`sandbox_identity_unavailable` because the runner exposes no real
sandbox handle

Observation construction fails open. Missing or malformed evidence
becomes an explicit gap; provider sentinel values are not exposed as
process exit codes, and unavailable resource or lifecycle measurements
remain unset rather than inferred.

## Capability coverage

| Producer | C1 | C2 | C3 | C4 | C5 | C6 | C7 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `opencode_sandboxed_agent` | V | V | X | V | V | V | V |
| `swe_agents` / OpenCode | V | V | X | V | X | X | V |
| `swe_agents` / OpenHands | X | X | X | O | X | X | V |

The capability matrix documents these evidence boundaries. Legacy
artifacts do not provide standardized semantic turns, authoritative
per-tool timing, or independent parallel-tool timing.

## Validation

- 34 combined standalone and sandboxed OpenCode tests, including
fresh-process import isolation
- focused SWE-bench resource-server and legacy SWE-agent tests across
disabled, observability-only, token-only, and combined capture states
- 114 shared rollout-observability, correlation, and collection
regressions rerun after the final rebase
- SQLite artifact -> parser -> decoupled `/run` composition test
covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup
primitives
- Ruff, formatting, Python compilation, `git diff --check`, and scoped
pre-commit hooks

A real artifact-compatibility smoke test used Docker Server 29.6.2 on
Linux/aarch64 and the exact
`swebench/sweb.eval.x86_64.astropy_1776_astropy-12907` image under
x86_64 emulation. OpenCode 1.17.11 was installed only inside the
temporary container and ran a real gpt-5.5-backed session whose bash
tool executed `printf opencode-observability-smoke`; the actual tool
result persisted and `opencode export` succeeded.

The WAL-mode database contained 1 session, 3 messages, and 7 parts.
After closing/exporting, only `opencode.db` was copied and parsed by the
final sandbox-local parser, producing 1 completed invocation, 1 tool
call, 0 compactions, and only the expected
`model_call_ownership_unavailable` gap. The temporary container was
stopped and auto-removed; nothing was installed on the host.

## Limitations

- the live smoke used the direct NVIDIA gateway rather than Gym's
rollout-prefixed model proxy, so it validates the real OpenCode artifact
schema/parser but not model-call ownership or capture joining
- OpenCode's final prose stream did not terminate after the tool result
and was gracefully interrupted
- the full decoupled `/run` + verifier flow was not run live because
`DockerProvider` cannot reconnect across the resource-server and agent
processes; that path requires OpenSandbox
- the SQLite parser is intentionally duplicated to keep the two agent
servers dependency-isolated and must remain synchronized

## Related rollout-observability work

- #2114 shared observation and correlation contract
- #2153 Claude Code producer
- #2115 OpenClaw producer
- #2117 Hermes producer
- #2118 Pi producer
- #2119 standalone OpenCode producer

---------

Signed-off-by: Michal Bien <mbien@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add OpenClaw and PinchBench rollout observations

4 participants