feat: re-pin verifiers for per-call trace records + generation time split - #3082
Merged
Conversation
Companion to PrimeIntellect-ai/verifiers#2061: Trace.calls per-call records, with finish_reason and usage moved off MessageNode onto ModelCall (trace schema v2). test_advantage's rollout builder attaches usage via a per-call record instead of the removed node field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas
marked this pull request as ready for review
July 18, 2026 02:22
Bumps the verifiers pin to d5320edcb (per-call ModelCall records #2061 + model/harness generation-time split #2060). TimingMetrics gains generation/model and generation/harness (from timing.generation.{model, harness}.duration, stamped server-side by Rollout.split_generation and carried on the wire), emitted as timing/generation/{model,harness}/*. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
samsja
approved these changes
Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Companion to the merged verifiers trace work: per-call records (#2061, RES-1085) and the model/harness generation-time split (#2060, RES-1084).
deps/verifierstod5320edcb(both merges):Trace.callsper-callModelCallrecords (node,model,sampling,endpoint,finish_reason,usage,time,error; upstream status onError.status_code), withfinish_reason/usagemoved offMessageNodeontoModelCall(trace schema v2); andTiming.generationbecoming aGenerationSpanwithmodel/harnessTimeSplits.tests/unit/orchestrator/test_advantage.py: the rollout builder attaches provider usage via a per-callvf.ModelCallinstead of the removed node field.TimingMetricsgainsgeneration/modelandgeneration/harness(per-rollouttiming.generation.model.duration/.harness.duration), emitted as{prefix}/{subset}/timing/generation/model/*and.../generation/harness/*alongside the existingtiming/generation/*. The env server stampssplit_generationinRollout.run, so the split rides the wire already populated.No other prime-rl behavior change: GRPO length penalties and token metrics consume the trace-level
num_*_tokens, which verifiers still derives (branches read usage off their attached calls);callsand the generation split ride the existing records additively.Breaking
rollouts/step_N/**/traces.jsonlfrom older runs) no longer validate under the new verifiers, since their nodes still carryfinish_reason/usage. Read old dumps with the old verifiers.Verification
tests/unit/orchestrator+tests/unit/utilsgreen at the new pin (159 passed);test_metricsextended to assert thetiming/generation/model+timing/generation/harnesswandb keys.timing/generation/model/mean = 0.775,timing/generation/harness/mean = 3.209,timing/generation/mean = 3.984(model + harness == generation exactly).🤖 Generated with Claude Code