feat: record per-call model call metadata on v1 traces

[mikasenghaas]

Summary

Closes RES-1085: per-call trace records for platform integration.

• New Trace.calls: list[ModelCall] — one lean typed record per real provider exchange, appended by the interception server on both the non-streaming and streaming (SSE relay) paths:
  • node: index of the assistant MessageNode this call committed (PendingTurn.commit now returns it) — the link into the message graph. The call's conversation is not repeated on the record: it is exactly the linked node's root-to-self path, so linkage holds even under branching (compaction forks, retokenized prefixes)
  • model: the model requested from the provider (equals agent.model by construction of the override; recorded per call because it's cheap and provable)
  • sampling: the call's effective settings, translated by the dialect (Dialect.parse_sampling, next to parse_request/parse_response): a per-dialect whitelist (sampling_fields) of what shapes generation — decoding knobs, budgets/stops, reasoning effort, output contract — with native aliases mapped onto the canonical SamplingConfig knobs (max_output_tokens → max_tokens, reasoning.effort → reasoning_effort) and dialect-specific keys (seed, tool_choice, response_format, parallel_tool_calls, …) riding as extras. Whitelisted, so payload / conversation state / tracking fields can never leak into the record by omission. Captures the eval-imposed knobs plus whatever the harness set that the eval left alone
  • endpoint: the provider path (/chat/completions / /responses / /v1/messages), i.e. which wire dialect the exchange spoke
  • finish_reason + usage: exchange attributes, moved here from MessageNode exclusively. Trace.is_truncated reads the last successful call; Trace.usage aggregates over calls; branches keep their token accounting (Trace.branches attaches each branch's calls in path order, Branch.usage/last_usage/num_* read those), so the trace-level num_*_tokens prime-rl consumes and the platform push payload are unchanged; the v0 bridge records a per-step call
  • error: a failed exchange is recorded too — the error coupled to the call that raised it, with a real traceback for non-provider failures (cancellations included) and the upstream HTTP status on Error.status_code (also populated on rollout-level Trace.errors now)
  • time: wall-clock TimeSpan of the exchange (node timestamps can't provide this: they are all stamped at commit)
• A transient failure the harness SDK retries leaves both records: the errored exchange (error set, no node) and the successful retry (linked to its node). Deliberately not recorded: retries that replay or coalesce onto an earlier attempt (no provider exchange happened), aux routes (count_tokens), and judge calls (outside the agent's graph).
• Raw request/response bodies are deliberately not retained: they grew traces quadratically in turns (83% of a 10-turn tb2 trace was request bodies restating the conversation), and the conversation is already on the graph. With the typed records, calls is linear at ~250 B/call — the same 10-turn trace went 146 KB → 31 KB (calls 121 KB → 2.5 KB, 8%).

Example

Trace.calls from a 10-turn terminal-bench-2 fix-git rollout (first two and last call shown):

[
  {
    "node": 2,
    "model": "deepseek/deepseek-v4-flash",
    "sampling": { "temperature": 0.7, "parallel_tool_calls": false },
    "endpoint": "/chat/completions",
    "finish_reason": "tool_calls",
    "time": { "start": 1784337093.274, "end": 1784337101.452 }
  },
  {
    "node": 4,
    "model": "deepseek/deepseek-v4-flash",
    "sampling": { "temperature": 0.7, "parallel_tool_calls": false },
    "endpoint": "/chat/completions",
    "finish_reason": "tool_calls",
    "time": { "start": 1784337128.306, "end": 1784337132.204 }
  },
  {
    "node": 20,
    "model": "deepseek/deepseek-v4-flash",
    "sampling": { "temperature": 0.7, "parallel_tool_calls": false },
    "endpoint": "/chat/completions",
    "finish_reason": "tool_calls",
    "time": { "start": 1784337382.416, "end": 1784337401.394 }
  }
]

temperature is the eval-injected knob; parallel_tool_calls is rlm's own request field the eval never touched — the per-call sampling captures both.

Verification

Linkage checked per call: successful calls point at a sampled assistant node, every sampled node is covered by exactly one call, finish_reason matches the committed node, model/sampling present.

• gsm8k-v1 (default harness, subprocess, PI deepseek/deepseek-v4-flash): reward 1.0, linkage OK.
• gsm8k-v1 with the codex harness (streaming: Responses dialect over the SSE relay path): reward 1.0, node link correct.
• terminal-bench-2 fix-git (rlm harness, prime VM runtime): 10-turn run — 10 calls, linkage OK, trace 31 KB with calls at 2.5 KB (8%); earlier full runs: no-compaction (16 turns, reward 1.0) and summarize_at_tokens=3000 (40 calls, 87 nodes, 7 branches from compaction forks) — every call links to the right node on its branch.
• Transient-error path checked in-process (flaky client failing once with a 502, then succeeding on the SDK-style retry): both calls recorded — the failure with error (status_code=502) and node=None; the retry linked to its committed node. Non-provider failures record a real traceback (formatted from the exception object — the record is written in a finally, where ambient exception state is gone).
• pytest tests/v1 -m "not e2e" green; test_single_turn (e2e) extended with per-call linkage assertions.

Notes

• OverlongPromptError.status_code now defaults to 400 (the status the interception server surfaces for overlong prompts) instead of inheriting the 502; model_error keeps the provider's real status when the failure carried one.

• MessageNode.finish_reason stays on the node although it is conceptually per-call (now also on ModelCall): removing it would make previously persisted traces unreadable under strict validation — a trace-version-bump follow-up if wanted. Same reasoning keeps usage on the node (it anchors per-branch token accounting and the v0 bridge).

• prime-rl consumes the new field with its next verifiers re-pin; the calls field itself is additive.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches core interception and trace schema (v2 bump) on every model turn; behavior is additive for consumers but changes where usage/truncation are sourced and how errors are recorded.

Overview
Adds Trace.calls — one ModelCall per real provider exchange (non-stream and SSE relay), bumped to TRACE_VERSION = 2. Each record links to the committed assistant node via node, captures model, dialect-whitelisted sampling (parse_sampling on chat/Anthropic/Responses), endpoint, finish_reason, usage, wall-clock time, and coupled error (with status_code on trace errors). The interception server appends calls in record_call from a finally on both paths; PendingTurn.commit now returns the assistant node id.

finish_reason and usage are no longer written on new MessageNode commits; Trace/Branch token accounting, is_truncated, and branch calls all read from calls instead. Dialects gain sampling_fields whitelists and format-specific alias normalization (e.g. max_completion_tokens, Anthropic output_config.effort, Responses reasoning.effort). Anthropic responses use widened ModdedAnthropicMessage for gateway service_tier; OverlongPromptError defaults to 400 with provider status preserved via model_error. The v0 bridge synthesizes minimal ModelCall rows per trajectory step. E2E test_single_turn asserts call↔sampled-node linkage.

^{Reviewed by Cursor Bugbot for commit 5062742. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Summary

Closes RES-1085: per-call trace records for platform integration.

• New Trace.calls: list[ModelCall] — one lean typed record per real provider exchange, appended by the interception server on both the non-streaming and streaming (SSE relay) paths:
  • node: index of the assistant MessageNode this call committed (PendingTurn.commit now returns it) — the link into the message graph. The call's conversation is not repeated on the record: it is exactly the linked node's root-to-self path, so linkage holds even under branching (compaction forks, retokenized prefixes)
  • model: the model requested from the provider (equals agent.model by construction of the override; recorded per call because it's cheap and provable)
  • sampling: the call's effective settings, translated by the dialect (Dialect.parse_sampling, next to parse_request/parse_response): a per-dialect whitelist (sampling_fields) of what shapes generation — decoding knobs, budgets/stops, reasoning effort, output contract — with native aliases mapped onto the canonical SamplingConfig knobs (max_output_tokens → max_tokens, reasoning.effort → reasoning_effort) and dialect-specific keys (seed, tool_choice, response_format, parallel_tool_calls, …) riding as extras. Whitelisted, so payload / conversation state / tracking fields can never leak into the record by omission. Captures the eval-imposed knobs plus whatever the harness set that the eval left alone
  • endpoint: the provider path (/chat/completions / /responses / /v1/messages), i.e. which wire dialect the exchange spoke
  • finish_reason + usage: exchange attributes, moved here from MessageNode exclusively. Trace.is_truncated reads the last successful call; Trace.usage aggregates over calls; branches keep their token accounting (Trace.branches attaches each branch's calls in path order, Branch.usage/last_usage/num_* read those), so the trace-level num_*_tokens prime-rl consumes and the platform push payload are unchanged; the v0 bridge records a per-step call
  • error: a failed exchange is recorded too — the error coupled to the call that raised it, with a real traceback for non-provider failures (cancellations included) and the upstream HTTP status on Error.status_code (also populated on rollout-level Trace.errors now)
  • time: wall-clock TimeSpan of the exchange (node timestamps can't provide this: they are all stamped at commit)
• A transient failure the harness SDK retries leaves both records: the errored exchange (error set, no node) and the successful retry (linked to its node). Deliberately not recorded: retries that replay or coalesce onto an earlier attempt (no provider exchange happened), aux routes (count_tokens), and judge calls (outside the agent's graph).
• Raw request/response bodies are deliberately not retained: they grew traces quadratically in turns (83% of a 10-turn tb2 trace was request bodies restating the conversation), and the conversation is already on the graph. With the typed records, calls is linear at ~250 B/call — the same 10-turn trace went 146 KB → 31 KB (calls 121 KB → 2.5 KB, 8%).

Example

Trace.calls from a 10-turn terminal-bench-2 fix-git rollout (first two and last call shown):

[
  {
    "node": 2,
    "model": "deepseek/deepseek-v4-flash",
    "sampling": { "temperature": 0.7, "parallel_tool_calls": false },
    "endpoint": "/chat/completions",
    "finish_reason": "tool_calls",
    "time": { "start": 1784337093.274, "end": 1784337101.452 }
  },
  {
    "node": 4,
    "model": "deepseek/deepseek-v4-flash",
    "sampling": { "temperature": 0.7, "parallel_tool_calls": false },
    "endpoint": "/chat/completions",
    "finish_reason": "tool_calls",
    "time": { "start": 1784337128.306, "end": 1784337132.204 }
  },
  {
    "node": 20,
    "model": "deepseek/deepseek-v4-flash",
    "sampling": { "temperature": 0.7, "parallel_tool_calls": false },
    "endpoint": "/chat/completions",
    "finish_reason": "tool_calls",
    "time": { "start": 1784337382.416, "end": 1784337401.394 }
  }
]

temperature is the eval-injected knob; parallel_tool_calls is rlm's own request field the eval never touched — the per-call sampling captures both.

Verification

Linkage checked per call: successful calls point at a sampled assistant node, every sampled node is covered by exactly one call, finish_reason matches the committed node, model/sampling present.

• gsm8k-v1 (default harness, subprocess, PI deepseek/deepseek-v4-flash): reward 1.0, linkage OK.
• gsm8k-v1 with the codex harness (streaming: Responses dialect over the SSE relay path): reward 1.0, node link correct.
• terminal-bench-2 fix-git (rlm harness, prime VM runtime): 10-turn run — 10 calls, linkage OK, trace 31 KB with calls at 2.5 KB (8%); earlier full runs: no-compaction (16 turns, reward 1.0) and summarize_at_tokens=3000 (40 calls, 87 nodes, 7 branches from compaction forks) — every call links to the right node on its branch.
• Transient-error path checked in-process (flaky client failing once with a 502, then succeeding on the SDK-style retry): both calls recorded — the failure with error (status_code=502) and node=None; the retry linked to its committed node. Non-provider failures record a real traceback (formatted from the exception object — the record is written in a finally, where ambient exception state is gone).
• pytest tests/v1 -m "not e2e" green; test_single_turn (e2e) extended with per-call linkage assertions.

Notes

• OverlongPromptError.status_code now defaults to 400 (the status the interception server surfaces for overlong prompts) instead of inheriting the 502; model_error keeps the provider's real status when the failure carried one.

• MessageNode.finish_reason stays on the node although it is conceptually per-call (now also on ModelCall): removing it would make previously persisted traces unreadable under strict validation — a trace-version-bump follow-up if wanted. Same reasoning keeps usage on the node (it anchors per-branch token accounting and the v0 bridge).

• prime-rl consumes the new field with its next verifiers re-pin; the calls field itself is additive.

🤖 Generated with Claude Code

---

[!NOTE]
Medium Risk
Touches core interception and trace schema (v2 bump) on every model turn; behavior is additive for consumers but changes where usage/truncation are sourced and how errors are recorded.

Overview
Adds Trace.calls — one ModelCall per real provider exchange (non-stream and SSE relay), bumped to TRACE_VERSION = 2. Each record links to the committed assistant node via node, captures model, dialect-whitelisted sampling (parse_sampling on chat/Anthropic/Responses), endpoint, finish_reason, usage, wall-clock time, and coupled error (with status_code on trace errors). The interception server appends calls in record_call from a finally on both paths; PendingTurn.commit now returns the assistant node id.

finish_reason and usage are no longer written on new MessageNode commits; Trace/Branch token accounting, is_truncated, and branch calls all read from calls instead. Dialects gain sampling_fields whitelists and format-specific alias normalization (e.g. max_completion_tokens, Anthropic output_config.effort, Responses reasoning.effort). Anthropic responses use widened ModdedAnthropicMessage for gateway service_tier; OverlongPromptError defaults to 400 with provider status preserved via model_error. The v0 bridge synthesizes minimal ModelCall rows per trajectory step. E2E test_single_turn asserts call↔sampled-node linkage.

^{Reviewed by Cursor Bugbot for commit 5062742. Bugbot is set up for automated code reviews on this repo. Configure here.}

Changes since #2061 opened

• Moved HTTP status code tracking from ModelCall.status field to Error.status_code field in trace recording [5062742]
• Updated Trace section in v1 overview documentation to specify inclusion of per-call ModelCall records [cf18f49]
• Added inspection checklist item for per-call calls records metadata [a02c7f5]
• Changed the type annotation of the sampling field in the ModelCall model from SamplingConfig | None to Sampling | None within the verifiers.v1.trace module, including adding the corresponding Sampling import from verifiers.v1.types [ce55ead]
• Removed inline documentation from the calls field in the Branch model within the verifiers.v1.trace module [ce55ead]

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new tracing feature with a schema version bump (1→2) and restructures where usage/finish_reason data is stored (from nodes to calls). The schema change and structural reorganization warrant human review to verify compatibility.

^{You can customize Macroscope's approvability policy. Learn more.}

[kennethnym]

the ModelCall is really useful, i have some thoughts below.

some more thoughts:

• model id is stored per rollout under Trace.agent.model. if in the future we support model switching on the fly, then the model id will have to move down a level (probably under ModelCall)
• maybe useful to have ModelCall.id as well, pulled from provider's response?

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cf18f49. Configure here.}