[rollout-observability][superseded] Add Claude Code rollout observations - #2116
Merged
Glorf merged 0 commit intoJul 27, 2026
Conversation
This was referenced Jul 23, 2026
Glorf
marked this pull request as ready for review
July 23, 2026 12:34
Glorf
force-pushed
the
feat/rollout-observability-claude-code
branch
from
July 23, 2026 13:24
cafe10a to
9a4657f
Compare
Glorf
changed the title
Glorf
force-pushed
the
feat/rollout-observability-claude-code
branch
from
July 23, 2026 14:16
9a4657f to
a442a8d
Compare
Glorf
force-pushed
the
feat/rollout-observability-claude-code
branch
from
July 23, 2026 18:20
a442a8d to
fdc7283
Compare
github-actions
Bot
added
the
sla:triage-overdue
Glorf
force-pushed
the
feat/rollout-observability-claude-code
branch
from
July 24, 2026 16:38
fdc7283 to
3f622a0
Compare
Glorf
changed the title
Glorf
force-pushed
the
feat/rollout-observability-claude-code
branch
2 times, most recently
from
July 27, 2026 17:38
3c424af to
82c3dc1
Compare
Glorf
changed the title
Glorf
changed the title
Glorf
added a commit
that referenced
this pull request
Aug 5, 2026
…verage (#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by #2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / #1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs #2115 and #2117-#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. #2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from #2115 through #2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | #2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | #2116 | Claude Code | Superseded by #2153 | See #2153 and this PR | | #2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | #2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | #2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | #2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | #2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with #2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of #1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
…verage (NVIDIA-NeMo#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by NVIDIA-NeMo#2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / NVIDIA-NeMo#1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs NVIDIA-NeMo#2115 and NVIDIA-NeMo#2117-NVIDIA-NeMo#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. NVIDIA-NeMo#2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from NVIDIA-NeMo#2115 through NVIDIA-NeMo#2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | NVIDIA-NeMo#2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | NVIDIA-NeMo#2116 | Claude Code | Superseded by NVIDIA-NeMo#2153 | See NVIDIA-NeMo#2153 and this PR | | NVIDIA-NeMo#2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | NVIDIA-NeMo#2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | NVIDIA-NeMo#2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | NVIDIA-NeMo#2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | NVIDIA-NeMo#2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with NVIDIA-NeMo#2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of NVIDIA-NeMo#1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Superseded by #2153. GitHub marked this PR merged when its head became an ancestor of its former feature-branch base during stack maintenance. This PR did not merge into
main.Summary
This PR adds rollout observations for the existing Claude Code Agent integration.
Claude Code transcripts are read before temporary configuration cleanup and returned as
ng_agent_observationsfor correlated/runrollouts.Captured evidence
(model_ref, response_id)Evidence boundaries
no_sandbox_runtime; sandbox CPU and peak-memory data are unavailable on this path.model_serverand protocol response IDs.Compatibility
ng_agent_observationsremains optional and additiveThis producer complements Model Server capture. It does not define a response or training contract.
Validation
56 focused Claude Code tests pass, covering nested subagents, exact model references, parallel tool intervals, compaction, root outcomes, failure isolation, rollout correlation, and MCP wiring. Focused pre-commit checks pass.
Replacement