Skip to content

[rollout-observability][7/7] Add SWE agent rollout observations - #2120

Merged
Glorf merged 11 commits into
mainfrom
feat/rollout-observability-swe
Aug 25, 2026
Merged

[rollout-observability][7/7] Add SWE agent rollout observations#2120
Glorf merged 11 commits into
mainfrom
feat/rollout-observability-swe

Conversation

@Glorf

@Glorf Glorf commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • emit standardized ng_agent_observations for the legacy SWE OpenCode/OpenHands paths and the decoupled opencode_sandboxed_agent + SWE-bench path
  • correlate OpenCode calls through rollout-prefixed Gym Model Server routes when rollout observability or token capture is active
  • compose OpenCode invocation/tool evidence with separate agent- and verifier-sandbox observations
  • preserve direct response behavior, grading outputs, retained artifacts, and subagent_trajectories

What changed

Decoupled OpenCode / SWE-bench

  • parse OpenCode SQLite sessions into invocations, parent relationships, cumulative model-visible conversations, tool timing/outcomes, compaction events, and exact response IDs
  • keep the sandboxed agent independently installable by copying the minimal parser closure locally; it no longer imports the standalone opencode_agent, and a fresh-process test enforces that boundary
  • isolate OpenCode data per observed run, download and parse the database before teardown, and remove the local scratch database afterward
  • record the connected agent sandbox's real provider/ID and compose verifier-sandbox lifecycle evidence returned by resources_servers/swebench
  • emit observations only when a capture-derived rollout ID exists; direct /v1/responses behavior remains unchanged

Legacy SWE harness

  • OpenCode records retained session invocations, parent relationships, exact response IDs, and the latest cumulative conversation; rollout-prefixed model calls provide capture and token accounting
  • OpenHands records its available cumulative root conversation and sandbox evidence, while explicitly reporting that exact model-call correlation is unavailable with the pinned fork
  • legacy Apptainer records leave sandbox_id unset and report sandbox_identity_unavailable because the runner exposes no real sandbox handle

Observation construction fails open. Missing or malformed evidence becomes an explicit gap; provider sentinel values are not exposed as process exit codes, and unavailable resource or lifecycle measurements remain unset rather than inferred.

Capability coverage

Producer C1 C2 C3 C4 C5 C6 C7
opencode_sandboxed_agent V V X V V V V
swe_agents / OpenCode V V X V X X V
swe_agents / OpenHands X X X O X X V

The capability matrix documents these evidence boundaries. Legacy artifacts do not provide standardized semantic turns, authoritative per-tool timing, or independent parallel-tool timing.

Validation

  • 34 combined standalone and sandboxed OpenCode tests, including fresh-process import isolation
  • focused SWE-bench resource-server and legacy SWE-agent tests across disabled, observability-only, token-only, and combined capture states
  • 114 shared rollout-observability, correlation, and collection regressions rerun after the final rebase
  • SQLite artifact -> parser -> decoupled /run composition test covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup primitives
  • Ruff, formatting, Python compilation, git diff --check, and scoped pre-commit hooks

A real artifact-compatibility smoke test used Docker Server 29.6.2 on Linux/aarch64 and the exact swebench/sweb.eval.x86_64.astropy_1776_astropy-12907 image under x86_64 emulation. OpenCode 1.17.11 was installed only inside the temporary container and ran a real gpt-5.5-backed session whose bash tool executed printf opencode-observability-smoke; the actual tool result persisted and opencode export succeeded.

The WAL-mode database contained 1 session, 3 messages, and 7 parts. After closing/exporting, only opencode.db was copied and parsed by the final sandbox-local parser, producing 1 completed invocation, 1 tool call, 0 compactions, and only the expected model_call_ownership_unavailable gap. The temporary container was stopped and auto-removed; nothing was installed on the host.

Limitations

  • the live smoke used the direct NVIDIA gateway rather than Gym's rollout-prefixed model proxy, so it validates the real OpenCode artifact schema/parser but not model-call ownership or capture joining
  • OpenCode's final prose stream did not terminate after the tool result and was gracefully interrupted
  • the full decoupled /run + verifier flow was not run live because DockerProvider cannot reconnect across the resource-server and agent processes; that path requires OpenSandbox
  • the SQLite parser is intentionally duplicated to keep the two agent servers dependency-isolated and must remain synchronized

Related rollout-observability work

@copy-pr-bot

copy-pr-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Glorf
Glorf marked this pull request as ready for review July 23, 2026 12:34
@Glorf
Glorf force-pushed the feat/rollout-observability-swe branch from 8ac07bc to c71c1e6 Compare July 23, 2026 13:24
@Glorf
Glorf force-pushed the feat/rollout-observability-opencode branch 2 times, most recently from 1de03b3 to cc0a73c Compare July 23, 2026 14:21
@Glorf
Glorf requested a review from a team as a code owner July 23, 2026 14:21
@Glorf
Glorf force-pushed the feat/rollout-observability-swe branch from c71c1e6 to 9174912 Compare July 23, 2026 14:23
@Glorf
Glorf force-pushed the feat/rollout-observability-opencode branch from cc0a73c to b2f65bb Compare July 23, 2026 18:20
@Glorf
Glorf force-pushed the feat/rollout-observability-swe branch from 9174912 to 52da3af Compare July 23, 2026 18:20
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 24, 2026
@Glorf
Glorf force-pushed the feat/rollout-observability-opencode branch from b2f65bb to c2e3908 Compare July 24, 2026 16:38
@Glorf
Glorf force-pushed the feat/rollout-observability-swe branch from 52da3af to 03a27bd Compare July 24, 2026 16:38
@Glorf Glorf changed the title [rollout-observability][7/7] Add SWE and OpenHands rollout observations [rollout-observability][8/8] Add SWE and OpenHands rollout observations Jul 25, 2026
@Glorf
Glorf force-pushed the feat/rollout-observability-swe branch from 03a27bd to e31cd5f Compare July 27, 2026 12:05
@Glorf
Glorf force-pushed the feat/rollout-observability-opencode branch 2 times, most recently from 75268e7 to 61aa400 Compare July 27, 2026 18:01
@Glorf
Glorf force-pushed the feat/rollout-observability-swe branch from e31cd5f to 2117cdc Compare July 27, 2026 18:01
@Glorf Glorf changed the title [rollout-observability][8/8] Add SWE and OpenHands rollout observations [rollout-observability][7/7] Add SWE and OpenHands rollout observations Jul 27, 2026
Glorf added a commit that referenced this pull request Jul 28, 2026
…lation contract (#2114)

## Summary

This PR defines a shared contract for rollout evidence that is not
visible at the Model Server boundary: agent and subagent structure, tool
execution intervals, context compaction, and sandbox outcome/resource
usage.

It also joins Agent Server observations with Model Server call capture:

```text
Agent or harness ──> ng_agent_observations ─┐
                                            ├─ exact correlation ─> rollout record
Model Servers ─────> ng_model_call_capture ─┘
```

This is the base of the rollout-observability work and follows up on
#1867.

## Contract

| Evidence | Representation |
|---|---|
| Root agent and subagents | `AgentInvocation` |
| Model calls owned by an invocation | `ModelCallRef` |
| Tool execution and parallel timing | `ToolCallObservation` |
| Context compaction | `ContextCompactionObservation` |
| Sandbox outcome and lifetime usage | `SandboxObservation` |
| Missing or unsupported evidence | `ObservationGap` |

Model calls are joined through `model_call_id`, or the exact
`(model_ref, response_id)` pair when the harness exposes the protocol
response ID. A compaction may own exact model calls from its enclosing
invocation; boundary references do not imply ownership. Ambiguous,
unmatched, conflicting, and unowned calls remain visible as gaps.

This is an observability view, not a training trajectory or a
replacement for `NeMoGymResponse`.

## Changes

- add the shared observation models and exact join logic
- extend `ModelCallRecord` with protocol response ID, model metadata,
and raw-payload fallbacks
- preserve valid records around damaged capture lines and report
incomplete captures
- preserve upstream status and response evidence when a Model Server
call raises
- join Agent Server observations with Model Server capture during
rollout-record assembly
- propagate rollout correlation through standard Agent, Resources, and
Model Server calls
- exclude observation payloads from aggregate-metrics requests

All behavior remains opt-in through the existing observability
configuration.

## Scope

- No external trajectory serialization is introduced.
- Response, reward, token-ID, and log-probability contracts are
unchanged.
- Missing hierarchy, ownership, and timing evidence is reported rather
than estimated.
- Sandbox usage describes the enclosing sandbox, not individual or
overlapping tool calls.
- Harness and sandbox-provider producers remain separate follow-ups.

## Validation

Focused observation, correlation, capture, streaming,
rollout-attachment, resource-server, and upstream-failure tests pass.
Ruff, formatting, and diff checks pass.

## Stack

- [1/7 #2114 — Define the rollout observation and correlation
contract](#2114)
- [2/7 #2153 — Add Claude Code rollout
observations](#2153)
- [3/7 #2115 — Add OpenClaw and PinchBench rollout
observations](#2115)
- [4/7 #2117 — Add Hermes rollout
observations](#2117)
- [5/7 #2118 — Add Pi rollout
observations](#2118)
- [6/7 #2119 — Add OpenCode rollout
observations](#2119)
- [7/7 #2120 — Add SWE and OpenHands rollout
observations](#2120)

---------

Signed-off-by: Michal Bien <mbien@nvidia.com>
@sephmard sephmard linked an issue Jul 29, 2026 that may be closed by this pull request
@Glorf

Glorf commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 949f70f

@Glorf

Glorf commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8a176b3

@Glorf

Glorf commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 2251a8b

@Glorf
Glorf disabled auto-merge August 25, 2026 19:09
@Glorf
Glorf merged commit 9a87438 into main Aug 25, 2026
6 checks passed
@Glorf
Glorf deleted the feat/rollout-observability-swe branch August 25, 2026 19:09
@sdevare-nv sdevare-nv mentioned this pull request Aug 27, 2026
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add SWE and OpenHands rollout observations

2 participants