[fix] Standardize rollout trajectory records and document producer coverage

fern/versions/latest/pages/about/concepts/key-terminology.mdx

@@ -7,7 +7,7 @@ Essential vocabulary for model training, RL workflows, and NeMo Gym. This glossa
 ## Rollout & Data Collection Terms
 **Rollout / Trajectory**
 
-**Rollout** (verb) refers to the process of executing a policy in an environment to generate data: stepping through the environment, taking actions, and recording what happens. **Rollout** (noun) is also used synonymously with **trajectory**: the resulting sequence of states, actions, and rewards: the ordered record of what happened. In practice, many people use "rollout" and "trajectory" interchangeably since a rollout produces exactly one trajectory.
+**Rollout** (verb) is the process of executing a policy in an environment. **Rollout** (noun) is one execution and its result. A **trajectory** is the ordered record of what happened during that execution. The terms are often used informally as synonyms, but a Responses API result or rollout metrics record is not necessarily a complete standardized trajectory record. See the [trajectory capability matrix](/reference/trajectory-capabilities).
 
 **Rollout Batch**
 
@@ -125,7 +125,7 @@ Models invoking external capabilities (APIs, calculators, databases) to accompli
 
 **Responses API**
 
-OpenAI's standard interface for rollouts, including function calls and multi-turn conversations. NeMo Gym's native format.
+OpenAI's interface for model interactions, including function calls and multi-turn conversation items. NeMo Gym uses it as its native agent/model protocol. It does not include every trajectory field, such as executor timing and resolution status.
 
 **Chat Completions API**
 

fern/versions/latest/pages/model-server/model-call-capture.mdx

@@ -81,8 +81,8 @@ totals = aggregate_model_call_metrics(store, rollout_id)
 `ModelCallRecord` is an observability serialization model derived from captured HTTP exchanges. It
 contains a unique server-generated `model_call_id`, the protocol `response_id` when present, typed
 `model_ref`, wall-clock `started_at` and `completed_at`, a `call_index`, API dialect, token and cache
-usage, latency, error details, tool calls, reasoning content, and the captured request and response.
-`started_at` is recorded immediately
+usage, `response_status`, normalized `finish_reason`, latency, error details, tool calls, reasoning
+content, and the captured request and response. `started_at` is recorded immediately
 before invoking the downstream ASGI application; `completed_at` is recorded when that invocation
 returns or raises, before capture parsing and persistence. Both are UTC Unix seconds for external
 trace correlation; durations use the monotonic latency fields. `call_index` reflects durable append
@@ -108,9 +108,14 @@ ng_model_call_capture = {
 }
 ```
 
-`metrics.num_calls` is the number of captured model calls. Attached calls omit raw request and
-response payloads; those remain in `CaptureStore`. Resume attempts use an `-a<n>` suffix so their
-data does not mix with an earlier attempt. Before dispatch, the collector clears any existing capture
+`metrics.num_calls` is the number of captured model calls. After successful trajectory projection, attached calls omit request
+and response payloads; those remain in `CaptureStore`. If projection fails, they remain attached with a
+`trajectory_projection_failed` gap. [`ng_trajectory`](/reference/trajectory-capabilities) includes payload copies in the rollout
+record, except for LabBench rows carrying the `multimodal_history_redacted` gap. Each call stores the exact model request, so
+cumulative multi-turn inputs can make rollout records large and should be handled with the same access controls as the capture
+directory. Persisted rollout JSONL retains these payloads; W&B rollout tables omit `ng_trajectory` and model-call request and
+response payloads. Resume attempts use an
+`-a<n>` suffix so their data does not mix with an earlier attempt. Before dispatch, the collector clears any existing capture
 for that exact rollout-attempt id, including a kill-shaped attempt being redispatched.
 
 The attachment is additive: it does not replace or rewrite the existing response, reward,
@@ -119,17 +124,16 @@ and aggregate-metrics requests exclude it.
 
 ## Agent observations
 
-Supported Agent Servers may also attach `ng_agent_observations` when observability is enabled. It
-contains an unordered `records` list of typed agent invocations, tool-call intervals, explicit
-context-compaction events, and sandbox observations. `gaps` reports unavailable evidence. An
+Supported Agent Servers may also attach `ng_agent_observations` when observability is enabled. Agent evidence and model-call
+capture are collected independently. The attachment contains an unordered `records` list of typed agent invocations,
+tool-call intervals, explicit context-compaction events, and sandbox observations. `gaps` reports unavailable evidence. An
 invocation's `conversation` contains the ordered, normalized items exposed by that integration.
 
 Agent observations and model-call capture are separate evidence. Join an invocation's model-call
 references by `model_call_id`, or by the exact `(model_ref, response_id)` pair when the harness sees
 the protocol response ID. An integration may resolve an otherwise hidden call only through a
 producer-specific, unique exact match against its retained artifact and the raw capture. Ambiguous
-matches remain unowned; timestamps or list position alone are never sufficient. The full model
-request and response remain in `CaptureStore`; rollout attachments intentionally omit them.
+matches remain unowned; timestamps or list position alone are never sufficient.
 
 Compaction records distinguish the calls immediately before and after the context change from
 `model_calls` used to perform the compaction. Compaction calls are exact references to calls owned by
@@ -141,8 +145,9 @@ measured interval, and `timing_source` identifies executor, harness, or artifact
 
 Model-visible tool calls and results remain in `AgentInvocation.conversation` as `function_call` and
 `function_call_output` items. Execution timing and outcome, when observable, live in
-`ToolCallObservation` and join through `(invocation_id, tool_call_id)`. A tool observation may also
-reference its enclosing `sandbox_id`; concurrent calls retain independent timing and outcome.
+`ToolCallObservation` and join through `(invocation_id, tool_call_id)`. The `ng_trajectory` projection combines these fields
+with the matching model-visible output. A tool observation may also reference its enclosing `sandbox_id`; concurrent calls
+retain independent timing and outcome.
 
 Each `SandboxObservation` covers one sandbox execution. Usage fields contain measured values only;
 configured limits are never reported as usage. Integrations emit only facts available at their

fern/versions/latest/pages/reference/index.mdx

@@ -22,4 +22,8 @@ All CLI commands, arguments, and usage examples.
 Common questions about debugging, performance, and configuration.
 </Card>
 
+<Card title="Trajectory Capability Matrix" href="/reference/trajectory-capabilities">
+Review standardized trajectory support by criterion and producer.
+</Card>
+
 </Cards>

fern/versions/latest/pages/reference/trajectory-capabilities.mdx

@@ -0,0 +1,83 @@
+---
+title: "Trajectory capability matrix"
+description: "Standardized trajectory support by criterion and producer"
+position: 5
+---
+
+# Trajectory capability matrix
+
+`ng_trajectory` schema version `1.0` combines available model-call capture and agent observations in a rollout record. It
+contains task and rollout identity, model calls and token usage, semantic turns, invocation-scoped model-visible histories,
+tool attempts, resolution, step count, and explicit evidence gaps. Source attachments remain under `ng_model_call_capture` and
+`ng_agent_observations`; aggregate-metrics requests omit all three attachments.
+
+Enable model-call evidence with `observability_enabled: true` and `model_call_capture_dir: /data/model-calls`. Calls must use
+a `/ng-rollout/<rollout_id>` Gym Model Server prefix. See [Model-call capture](/model-server/model-call-capture).
+Captured request and response payloads are included in `ng_trajectory`. LabBench rows that report the
+`multimodal_history_redacted` gap omit those payload copies; direct provider calls are not captured. See
+[Model-call capture](/model-server/model-call-capture) for payload retention and access-control requirements. Token fields are
+`prompt_tokens`, `completion_tokens`, `reasoning_tokens`, `total_tokens`, and `cached_tokens`. The fields are nullable, and
+provider-reported details are preserved when available.
+
+## Acceptance criteria
+
+| ID | Criterion |
+|---|---|
+| C1 | Standard schema for per-model-call token statistics and response metadata |
+| C2 | Prompt, completion, reasoning, total, and cached token counts when available |
+| C3 | Task and rollout identity, turn number, timestamp, question, answer, reasoning, resolution, and step count per turn |
+| C4 | Model-visible input and output history can be reconstructed from persisted rollout JSONL |
+| C5 | Tool output, status, start and completion timestamps, and duration |
+| C6 | Independent timing for parallel tool calls |
+| C7 | Resource-server-backed and custom or sandbox-backed benchmarks |
+
+## Agent coverage
+
+- `V`: complete for the evaluated path under the documented observability configuration.
+- `O`: currently partial or path-dependent.
+- `X`: currently unavailable.
+
+For C1, C2, and C4, `V` evaluates the correlated Gym Model Server path; direct-provider alternatives are outside capture.
+
+| Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 |
+|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `anyswe_agent` | X | X | X | X | X | X | X |
+| `anyterminal_agent` | V | V | X | V | X | X | X |
+| `aviary_agent` | V | V | X | V | X | X | X |
+| `browsecomp_agent` | V | V | X | V | X | X | X |
+| `claude_code_agent` | V | V | X | V | V | V | V |
+| `codex_agent` | V | V | X | V | X | X | X |
+| `critpt_agent` | V | V | X | V | X | X | X |
+| `cvdp_agent` | O | O | X | O | X | X | X |
+| `finance_agent` | X | O | X | X | X | X | X |
+| `gymnasium_agent` | V | V | X | V | X | X | X |
+| `harbor_agent` | X | X | X | X | X | X | X |
+| `hermes_agent` | V | V | X | V | X | X | X |
+| `kilocode_agent` | V | V | X | V | X | X | X |
+| `labbench2_vlm_agent` | V | V | O | O | O | X | V |
+| `langgraph_agent` | X | X | X | X | X | X | X |
+| `mini_swe_agent` | X | X | X | X | X | X | X |
+| `mini_swe_agent_2` | V | V | X | V | X | X | X |
+| `non_executing_simple_agent` | V | V | X | V | X | X | X |
+| `openclaw_agent` | X | X | X | X | X | X | X |
+| `opencode_agent` | X | X | X | X | X | X | X |
+| `osworld_agent` | O | O | X | O | X | X | X |
+| `pi_agent` | X | X | X | X | X | X | X |
+| `pinchbench` | X | X | X | X | X | X | X |
+| `proof_refinement_agent` | V | V | X | V | X | X | X |
+| `remote_agent` | X | O | X | X | X | X | X |
+| `scicode_agent` | X | X | X | X | X | X | X |
+| `simple_agent` | V | V | O | V | O | X | V |
+| `speed_bench_agent` | V | V | X | V | X | X | X |
+| `stirrup_agent` | O | O | X | O | X | X | X |
+| `swe_agents` | X | X | X | X | X | X | X |
+| `tau2` | V | V | X | V | X | X | X |
+| `tool_simulation_agent` | V | V | X | V | X | X | X |
+| `toolsandbox_agent` | V | V | X | V | X | X | X |
+| `verifiers_agent` | X | X | X | X | X | X | X |
+
+Simple and LabBench C3 support is partial because verifier responses may omit resolution status. Current `O` values reflect the
+CVDP Simple path, Finance and Remote aggregate usage, LabBench image redaction, OSWorld M3 and Pointer direct-provider routes,
+omitted raised failures in Simple-derived agents, and Stirrup calls outside its policy path. The matrix reports producer output,
+not schema capacity.
+C7 requires standardized agent-side trajectory evidence; model HTTP capture alone does not satisfy it.

nemo_gym/base_responses_api_model.py

@@ -371,6 +371,21 @@ def _token_count(value: Any) -> Optional[int]:
     return value if type(value) is int and value >= 0 else None
 
 
+def _usage_detail_token(
+    usage: Mapping[str, Any], detail_groups: tuple[str, ...], field_names: tuple[str, ...]
+) -> Optional[int]:
+    """Return the first valid token count across equivalent provider detail shapes."""
+    for group_name in detail_groups:
+        details = usage.get(group_name)
+        if not isinstance(details, Mapping):
+            continue
+        for field_name in field_names:
+            value = _token_count(details.get(field_name))
+            if value is not None:
+                return value
+    return None
+
+
 def extract_token_stats(usage: Any) -> dict[str, Optional[int]]:
     """Normalize token totals across Responses, Chat Completions, and Anthropic Messages usage.
 
@@ -402,20 +417,22 @@ def extract_token_stats(usage: Any) -> dict[str, Optional[int]]:
     cache_read = _token_count(usage.get("cache_read_input_tokens"))
     cache_creation = _token_count(usage.get("cache_creation_input_tokens"))
     if cache_read is not None or cache_creation is not None:
-        # A fully-cached response can omit input_tokens; use a 0 base so the folded prompt size is
-        # preserved rather than dropped to null. (Top-level cache_* keys are Anthropic-only, so the
-        # OpenAI/Responses path -- nested prompt_tokens_details.cached_tokens -- never enters here.)
-        tokens_in = (tokens_in or 0) + (cache_read or 0) + (cache_creation or 0)
+        cache_total = (cache_read or 0) + (cache_creation or 0)
+        # Zero cache fields alone do not establish a missing prompt count.
+        if tokens_in is not None or cache_total > 0:
+            tokens_in = (tokens_in or 0) + cache_total
     tokens_total = _token_count(usage.get("total_tokens"))
     if tokens_total is None and tokens_in is not None and tokens_out is not None:
         tokens_total = tokens_in + tokens_out
-    details = usage.get("output_tokens_details") or usage.get("completion_tokens_details") or {}
-    if not isinstance(details, Mapping):
-        details = {}
+    tokens_reasoning = _usage_detail_token(
+        usage, ("output_tokens_details", "completion_tokens_details"), ("reasoning_tokens",)
+    )
+    if tokens_reasoning is None:
+        tokens_reasoning = _token_count(usage.get("reasoning_output_tokens"))
     return {
         "tokens_in": tokens_in,
         "tokens_out": tokens_out,
-        "tokens_reasoning": _token_count(details.get("reasoning_tokens")),
+        "tokens_reasoning": tokens_reasoning,
         "tokens_total": tokens_total,
         "cache_creation_tokens": cache_creation,
     }
@@ -425,12 +442,15 @@ def _cache_signal(usage: Any) -> tuple[Optional[bool], Optional[int]]:
     """Cache hit/miss + cached-token count, from usage cache fields (OpenAI / Anthropic)."""
     if not isinstance(usage, Mapping):
         return None, None
-    details = usage.get("prompt_tokens_details") or usage.get("input_tokens_details") or {}
-    if not isinstance(details, Mapping):
-        details = {}
-    cached = _token_count(details.get("cached_tokens"))
+    cached = _usage_detail_token(
+        usage,
+        ("prompt_tokens_details", "input_tokens_details"),
+        ("cached_tokens", "cached_input_tokens"),
+    )
     if cached is None:
         cached = _token_count(usage.get("cache_read_input_tokens"))  # Anthropic
+    if cached is None:
+        cached = _token_count(usage.get("cached_input_tokens"))
     if cached is None:
         return None, None
     return cached > 0, cached
@@ -523,6 +543,7 @@ class ModelCallRecord(BaseModel):
     model: Optional[str] = None
     dialect: Optional[str] = None
     status_code: Optional[int] = None
+    response_status: Optional[str] = None
     finish_reason: Optional[str] = None
 
     # Wall-clock bounds around the downstream ASGI invocation, as UTC Unix timestamps. These are
@@ -597,6 +618,7 @@ def build_model_call_record(exchange: dict[str, Any], *, call_index: int) -> Mod
         model=model if isinstance(model, str) else None,
         dialect=exchange.get("dialect"),
         status_code=exchange.get("status_code"),
+        response_status=response.get("status") if isinstance(response.get("status"), str) else None,
         finish_reason=finish_reason,
         started_at=exchange.get("started_at"),
         completed_at=exchange.get("completed_at"),
@@ -646,6 +668,7 @@ def _sum(attr: str) -> Optional[float]:
         "tokens_out": _sum("tokens_out"),
         "tokens_reasoning": _sum("tokens_reasoning"),
         "tokens_total": _sum("tokens_total"),
+        "cached_tokens": _sum("cached_tokens"),
         "latency_total_ms": _sum("latency_total_ms"),
         "num_calls": len(calls),
     }

nemo_gym/openai_utils.py

@@ -361,6 +361,26 @@ class NeMoGymResponseUsage(ResponseUsage):
     output_tokens_details: NeMoGymResponseOutputTokensDetails
 
 
+def accumulate_response_usage(
+    total: Optional[NeMoGymResponseUsage], additional: Optional[NeMoGymResponseUsage]
+) -> Optional[NeMoGymResponseUsage]:
+    """Accumulate top-level and detailed response token counts."""
+    if additional is None:
+        return total
+    if total is None:
+        return additional.model_copy(deep=True)
+
+    result = total.model_copy(deep=True)
+    result.input_tokens += additional.input_tokens
+    result.output_tokens += additional.output_tokens
+    result.total_tokens += additional.total_tokens
+    if result.input_tokens_details is not None and additional.input_tokens_details is not None:
+        result.input_tokens_details.cached_tokens += additional.input_tokens_details.cached_tokens
+    if result.output_tokens_details is not None and additional.output_tokens_details is not None:
+        result.output_tokens_details.reasoning_tokens += additional.output_tokens_details.reasoning_tokens
+    return result
+
+
 class NeMoGymResponse(Response):
     output: List[NeMoGymResponseOutputItem]
     usage: Optional[NeMoGymResponseUsage] = None