Skip to content

Add agent-side rollout observability - #2106

Closed
Glorf wants to merge 1 commit into
mainfrom
feat/rollout-observability-p0
Closed

Add agent-side rollout observability#2106
Glorf wants to merge 1 commit into
mainfrom
feat/rollout-observability-p0

Conversation

@Glorf

@Glorf Glorf commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add the opt-in ng_agent_observations contract for agent invocations, tool-call intervals, context-compaction events, model-call references, and explicit observation gaps
  • add producers for Claude Code, Hermes, OpenClaw, and PinchBench, plus optional AnyTerminal sidecar transport
  • preserve protocol response IDs in model-call records for exact (model_ref, response_id) joins
  • propagate rollout correlation to Tau, SWE/OpenCode, GDPVal judge calls, and Stirrup
  • exclude observation payloads from aggregate-metrics requests and document capability boundaries

Producer coverage

Producer Evidence exposed
Claude Code Transcript-derived invocation hierarchy, model-call references, artifact-derived tool intervals, and compaction markers
Hermes Invocation hierarchy, executor tool timing, and compaction markers through the pinned Hermes integration; exact model-call ownership is unavailable
OpenClaw / PinchBench Normalized root response items with explicit gaps for hierarchy, timing, compaction, and model-call ownership
AnyTerminal Optional normalized sidecar bundle; otherwise an explicit availability gap

The full model request and response remain in CaptureStore. Agent observations are additive rollout evidence, not a canonical trajectory or training contract.

Compatibility

  • emission remains gated by observability_enabled
  • the disabled execution and output paths remain unchanged
  • observation data does not replace the agent response, reward, token IDs, or log probabilities
  • producer parsing failures covered here return explicit gaps instead of inferred data

Review status

This is a combined draft for cross-agent validation. Before it is ready to merge:

  • preserve the existing self-HTTP /v1/responses boundary for observed Hermes and OpenClaw runs; the current observed path executes in-process to collect integration hooks
  • complete real-agent validation across the target benchmark matrix
  • split the shared contract and producer integrations into stacked PRs

Canonical trajectory assembly, model-call/agent-observation export joins, ATIF projection, Pi/OpenHands producers, and sandbox CPU/memory telemetry are not included.

Validation

  • producer integration suites: 213 passed
  • core contract, capture, rollout aggregation, GDPVal, and Stirrup suites: 194 passed
  • Tau policy/user correlation: 2 passed
  • SWE/OpenCode correlation and disabled serialization: 3 passed
  • Ruff check and format checks passed
  • staged diff and credential-pattern checks passed

Related to #1867.

Signed-off-by: Michal Bien <mbien@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

🌿 Preview your docs: https://nvidia-preview-feat-rollout-observability-p0.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

@Glorf

Glorf commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by the reviewed rollout-observability stack: #2114, #2115, #2116, #2117, #2118, #2119, and #2120.

@Glorf Glorf closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant