Skip to content

feat: re-pin verifiers for per-call trace records + generation time split - #3082

Merged
mikasenghaas merged 4 commits into
mainfrom
feat/per-call-trace-records
Jul 18, 2026
Merged

feat: re-pin verifiers for per-call trace records + generation time split#3082
mikasenghaas merged 4 commits into
mainfrom
feat/per-call-trace-records

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Jul 18, 2026

Copy link
Copy Markdown
Member

Summary

Companion to the merged verifiers trace work: per-call records (#2061, RES-1085) and the model/harness generation-time split (#2060, RES-1084).

  • Re-pins deps/verifiers to d5320edcb (both merges): Trace.calls per-call ModelCall records (node, model, sampling, endpoint, finish_reason, usage, time, error; upstream status on Error.status_code), with finish_reason/usage moved off MessageNode onto ModelCall (trace schema v2); and Timing.generation becoming a GenerationSpan with model/harness TimeSplits.
  • tests/unit/orchestrator/test_advantage.py: the rollout builder attaches provider usage via a per-call vf.ModelCall instead of the removed node field.
  • Exports the generation split to wandb: TimingMetrics gains generation/model and generation/harness (per-rollout timing.generation.model.duration / .harness.duration), emitted as {prefix}/{subset}/timing/generation/model/* and .../generation/harness/* alongside the existing timing/generation/*. The env server stamps split_generation in Rollout.run, so the split rides the wire already populated.

No other prime-rl behavior change: GRPO length penalties and token metrics consume the trace-level num_*_tokens, which verifiers still derives (branches read usage off their attached calls); calls and the generation split ride the existing records additively.

Breaking

  • Inherited from the verifiers pin: trace schema v2 — previously persisted traces (rollouts/step_N/**/traces.jsonl from older runs) no longer validate under the new verifiers, since their nodes still carry finish_reason/usage. Read old dumps with the old verifiers.

Verification

  • tests/unit/orchestrator + tests/unit/utils green at the new pin (159 passed); test_metrics extended to assert the timing/generation/model + timing/generation/harness wandb keys.
  • Reverse-text RL smoke on this box (2x RTX PRO 6000 Blackwell, subprocess env server, 3 steps): trains end-to-end (reward 0.12 -> 0.17, 0% error). Across the 552 saved rollouts the split is populated on every one - timing/generation/model/mean = 0.775, timing/generation/harness/mean = 3.209, timing/generation/mean = 3.984 (model + harness == generation exactly).

🤖 Generated with Claude Code

mikasenghaas and others added 3 commits July 18, 2026 01:46
Companion to PrimeIntellect-ai/verifiers#2061: Trace.calls per-call records,
with finish_reason and usage moved off MessageNode onto ModelCall (trace
schema v2). test_advantage's rollout builder attaches usage via a per-call
record instead of the removed node field.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mikasenghaas
mikasenghaas marked this pull request as ready for review July 18, 2026 02:22
Bumps the verifiers pin to d5320edcb (per-call ModelCall records #2061 +
model/harness generation-time split #2060). TimingMetrics gains
generation/model and generation/harness (from timing.generation.{model,
harness}.duration, stamped server-side by Rollout.split_generation and
carried on the wire), emitted as timing/generation/{model,harness}/*.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mikasenghaas mikasenghaas changed the title feat: re-pin verifiers for per-call trace records feat: re-pin verifiers for per-call trace records + generation time split Jul 18, 2026
@mikasenghaas
mikasenghaas requested a review from samsja July 18, 2026 03:02
@mikasenghaas
mikasenghaas merged commit d085605 into main Jul 18, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants