[rollout-observability][follow-up] Correlate auxiliary Model Server calls - #2122
Closed
Glorf wants to merge 8 commits into
Closed
[rollout-observability][follow-up] Correlate auxiliary Model Server calls#2122Glorf wants to merge 8 commits into
Glorf wants to merge 8 commits into
Conversation
Glorf
force-pushed
the
feat/rollout-observability-aux-model-correlation
branch
from
July 24, 2026 16:38
1da9a2c to
e0856b3
Compare
Glorf
marked this pull request as ready for review
July 24, 2026 18:54
Glorf
changed the title
This was referenced Jul 25, 2026
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
force-pushed
the
feat/rollout-observability-aux-model-correlation
branch
from
July 27, 2026 12:05
e0856b3 to
f8b97bf
Compare
Glorf
force-pushed
the
feat/rollout-observability-base
branch
from
July 27, 2026 12:05
edce744 to
b851fdf
Compare
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
force-pushed
the
feat/rollout-observability-aux-model-correlation
branch
from
July 27, 2026 17:37
f8b97bf to
0ca9488
Compare
Glorf
changed the title
github-actions
Bot
added
the
sla:triage-overdue
Glorf
force-pushed
the
feat/rollout-observability-base
branch
2 times, most recently
from
July 28, 2026 16:33
e960d68 to
698e1e4
Compare
6 tasks
Glorf
added a commit
that referenced
this pull request
Aug 5, 2026
…verage (#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by #2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / #1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs #2115 and #2117-#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. #2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from #2115 through #2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | #2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | #2116 | Claude Code | Superseded by #2153 | See #2153 and this PR | | #2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | #2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | #2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | #2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | #2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with #2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of #1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
…verage (NVIDIA-NeMo#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by NVIDIA-NeMo#2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / NVIDIA-NeMo#1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs NVIDIA-NeMo#2115 and NVIDIA-NeMo#2117-NVIDIA-NeMo#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. NVIDIA-NeMo#2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from NVIDIA-NeMo#2115 through NVIDIA-NeMo#2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | NVIDIA-NeMo#2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | NVIDIA-NeMo#2116 | Claude Code | Superseded by NVIDIA-NeMo#2153 | See NVIDIA-NeMo#2153 and this PR | | NVIDIA-NeMo#2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | NVIDIA-NeMo#2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | NVIDIA-NeMo#2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | NVIDIA-NeMo#2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | NVIDIA-NeMo#2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with NVIDIA-NeMo#2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of NVIDIA-NeMo#1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
added a commit
that referenced
this pull request
Aug 10, 2026
## Summary Add OpenClaw rollout observations for both the standalone agent endpoint and the PinchBench benchmark path. OpenClaw is the observation producer in both paths. PinchBench exports retained OpenClaw session artifacts and attaches the resulting OpenClaw observation bundle; it does not define a separate observation format or producer identity. Observations are additive and are emitted only when observability and rollout correlation are enabled. Existing agent responses, rewards, and grading fields are unchanged. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 are evaluated on the correlated Gym Model Server path. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | OpenClaw | V | V | X | O | V | V | V | C3 remains unsupported because OpenClaw does not emit standardized per-turn records. C4 is partial because retained branched sessions are reported but cannot always reconstruct a single model-visible branch. ## Changes - Normalize retained OpenClaw session records into user, assistant, reasoning, tool-call, and tool-result observations. - Reconstruct subagent lineage from retained OpenClaw session stores. - Correlate policy and judge model calls through rollout-prefixed Gym Model Servers when configured. - Preserve independent tool-call timing, status, duration, and output. - Record context compaction and explicit observation gaps. - Preserve direct model endpoint support when no Gym Model Server is configured. - Isolate observation capture failures from agent responses and benchmark results. ## Evidence boundaries - Model-call correlation requires transcript response IDs and a configured Gym Model Server. - OpenClaw does not identify the exact tool call that spawned a child session. - Branches within one session are reported but are not reconstructed as separate invocations. - Missing timestamps, transcripts, or correlation evidence are represented as explicit gaps. ## Validation - OpenClaw and PinchBench suites: 88 passed. - Core trajectory, collector, correlation, base-agent, and model suites: 155 passed. - Producer-to-collector projection verifies model calls, token details, tool output, status, and independent timing in `ng_trajectory`. - Ruff, formatting, compilation, shell, and diff checks passed. ## Follow-up pull requests - [#2117 — Add Hermes rollout observations](#2117) - [#2118 — Add Pi rollout observations](#2118) - [#2119 — Add OpenCode rollout observations](#2119) - [#2120 — Add SWE and OpenHands rollout observations](#2120) - [#2122 — Add Harbor rollout observations](#2122) Foundation work is available in [#2114](#2114) and [#2153](#2153), both merged into `main`. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR correlates auxiliary Model Server calls that bypass Gym's standard
ServerClientpath.Policy, user-simulator, and judge calls can therefore be attributed to the same rollout while remaining distinguishable by
model_ref.Changes
No rollout identifier is added to public request schemas. Unobserved runs retain their existing Model Server URLs.
Scope
/v1/messagesand does not depend on Chat Completions streaming.Validation
Focused GDPVal, Stirrup, and Tau2 tests pass, including observed and unobserved URL behavior.
Relationship to the P0 stack
This is a parallel follow-up to #2114. Claude Code support in #2116 does not depend on it.