[rollout-observability][7/7] Add SWE agent rollout observations - #2120
Merged
Conversation
This was referenced Jul 23, 2026
Glorf
marked this pull request as ready for review
July 23, 2026 12:34
Glorf
force-pushed
the
feat/rollout-observability-swe
branch
from
July 23, 2026 13:24
8ac07bc to
c71c1e6
Compare
Glorf
force-pushed
the
feat/rollout-observability-opencode
branch
2 times, most recently
from
July 23, 2026 14:21
1de03b3 to
cc0a73c
Compare
Glorf
force-pushed
the
feat/rollout-observability-swe
branch
from
July 23, 2026 14:23
c71c1e6 to
9174912
Compare
Glorf
force-pushed
the
feat/rollout-observability-opencode
branch
from
July 23, 2026 18:20
cc0a73c to
b2f65bb
Compare
Glorf
force-pushed
the
feat/rollout-observability-swe
branch
from
July 23, 2026 18:20
9174912 to
52da3af
Compare
Glorf
force-pushed
the
feat/rollout-observability-opencode
branch
from
July 24, 2026 16:38
b2f65bb to
c2e3908
Compare
Glorf
force-pushed
the
feat/rollout-observability-swe
branch
from
July 24, 2026 16:38
52da3af to
03a27bd
Compare
Glorf
force-pushed
the
feat/rollout-observability-swe
branch
from
July 27, 2026 12:05
03a27bd to
e31cd5f
Compare
Glorf
force-pushed
the
feat/rollout-observability-opencode
branch
2 times, most recently
from
July 27, 2026 18:01
75268e7 to
61aa400
Compare
Glorf
force-pushed
the
feat/rollout-observability-swe
branch
from
July 27, 2026 18:01
e31cd5f to
2117cdc
Compare
Glorf
added a commit
that referenced
this pull request
Jul 28, 2026
…lation contract (#2114) ## Summary This PR defines a shared contract for rollout evidence that is not visible at the Model Server boundary: agent and subagent structure, tool execution intervals, context compaction, and sandbox outcome/resource usage. It also joins Agent Server observations with Model Server call capture: ```text Agent or harness ──> ng_agent_observations ─┐ ├─ exact correlation ─> rollout record Model Servers ─────> ng_model_call_capture ─┘ ``` This is the base of the rollout-observability work and follows up on #1867. ## Contract | Evidence | Representation | |---|---| | Root agent and subagents | `AgentInvocation` | | Model calls owned by an invocation | `ModelCallRef` | | Tool execution and parallel timing | `ToolCallObservation` | | Context compaction | `ContextCompactionObservation` | | Sandbox outcome and lifetime usage | `SandboxObservation` | | Missing or unsupported evidence | `ObservationGap` | Model calls are joined through `model_call_id`, or the exact `(model_ref, response_id)` pair when the harness exposes the protocol response ID. A compaction may own exact model calls from its enclosing invocation; boundary references do not imply ownership. Ambiguous, unmatched, conflicting, and unowned calls remain visible as gaps. This is an observability view, not a training trajectory or a replacement for `NeMoGymResponse`. ## Changes - add the shared observation models and exact join logic - extend `ModelCallRecord` with protocol response ID, model metadata, and raw-payload fallbacks - preserve valid records around damaged capture lines and report incomplete captures - preserve upstream status and response evidence when a Model Server call raises - join Agent Server observations with Model Server capture during rollout-record assembly - propagate rollout correlation through standard Agent, Resources, and Model Server calls - exclude observation payloads from aggregate-metrics requests All behavior remains opt-in through the existing observability configuration. ## Scope - No external trajectory serialization is introduced. - Response, reward, token-ID, and log-probability contracts are unchanged. - Missing hierarchy, ownership, and timing evidence is reported rather than estimated. - Sandbox usage describes the enclosing sandbox, not individual or overlapping tool calls. - Harness and sandbox-provider producers remain separate follow-ups. ## Validation Focused observation, correlation, capture, streaming, rollout-attachment, resource-server, and upstream-failure tests pass. Ruff, formatting, and diff checks pass. ## Stack - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw and PinchBench rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
7 tasks
Contributor
Author
|
/ok to test 949f70f |
Contributor
Author
|
/ok to test 8a176b3 |
Contributor
Author
|
/ok to test 2251a8b |
Glorf
disabled auto-merge
August 25, 2026 18:56
Glorf
enabled auto-merge (squash)
August 25, 2026 19:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ng_agent_observationsfor the legacy SWE OpenCode/OpenHands paths and the decoupledopencode_sandboxed_agent+ SWE-bench pathsubagent_trajectoriesWhat changed
Decoupled OpenCode / SWE-bench
opencode_agent, and a fresh-process test enforces that boundaryresources_servers/swebench/v1/responsesbehavior remains unchangedLegacy SWE harness
sandbox_idunset and reportsandbox_identity_unavailablebecause the runner exposes no real sandbox handleObservation construction fails open. Missing or malformed evidence becomes an explicit gap; provider sentinel values are not exposed as process exit codes, and unavailable resource or lifecycle measurements remain unset rather than inferred.
Capability coverage
opencode_sandboxed_agentswe_agents/ OpenCodeswe_agents/ OpenHandsThe capability matrix documents these evidence boundaries. Legacy artifacts do not provide standardized semantic turns, authoritative per-tool timing, or independent parallel-tool timing.
Validation
/runcomposition test covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup primitivesgit diff --check, and scoped pre-commit hooksA real artifact-compatibility smoke test used Docker Server 29.6.2 on Linux/aarch64 and the exact
swebench/sweb.eval.x86_64.astropy_1776_astropy-12907image under x86_64 emulation. OpenCode 1.17.11 was installed only inside the temporary container and ran a real gpt-5.5-backed session whose bash tool executedprintf opencode-observability-smoke; the actual tool result persisted andopencode exportsucceeded.The WAL-mode database contained 1 session, 3 messages, and 7 parts. After closing/exporting, only
opencode.dbwas copied and parsed by the final sandbox-local parser, producing 1 completed invocation, 1 tool call, 0 compactions, and only the expectedmodel_call_ownership_unavailablegap. The temporary container was stopped and auto-removed; nothing was installed on the host.Limitations
/run+ verifier flow was not run live becauseDockerProvidercannot reconnect across the resource-server and agent processes; that path requires OpenSandboxRelated rollout-observability work