feat(v1): seat replay — pin any env agent to a finished run's traces

[mikasenghaas]

Summary

Seat replay: pin any env agent seat to a finished run's traces, so the other seats iterate against fixed work. Stacked on #2109 (uses its agentic-judge env for the guard and the motivating workflow); retarget to main once that merges.

• AgentConfig.replay: Path | None — a run dir (or its traces.jsonl). Each task returns that run's saved trace for this seat instead of sampling: _ReplayEpisodeAgent revalidates the record (WireTrace) and re-stamps it with the seat's standing (name, trainable), so downstream env logic (episode.traces, finalize) sees it exactly like a live rollout. A single-seat source run matches any requested seat name (a plain single-agent baseline replays into a multi-agent env's solver); tasks are matched by task name (fallback idx), with a clear error when the selection doesn't overlap the saved run.
• Constraint, stated not hidden — a replayed seat runs no model and leaves no runtime state. Envs whose downstream agents need the replayed seat's box can't replay it: provision() on a replayed seat raises, and agentic-judge refuses shared_runtime + a replayed solver at construction with the fix in the message (--env.shared-runtime false). Env.run stays arbitrary user-space — chained/parallel/interleaved seats all work off the trace.
• Why — quick iteration on judging: re-grade saved solver rollouts through the real env path with uv run eval @ judges/scaleswe.toml --env.solver.replay <run dir> (same task selection as the source run), instead of a bespoke re-judge script.

Verification

• Docker smoke (gsm8k through agentic-judge, --env.solver.replay <prior run> --env.shared-runtime false): the replayed solver trace keeps the source run's trace id verbatim while a live judge grades it in its own box; verdict lands as judge/solved + the judge reward.
• Used for the scaleswe re-judge experiment: 64 saved baseline solver rollouts replayed through judges/scaleswe.toml with a live glm-5.2 judge on prime sandboxes.
• ruff check, repo ty check (diagnostic count unchanged vs main), full non-e2e tests/v1 green.

Note

Add seat replay to pin any env agent to traces from a finished run

• Adds an optional replay path field to AgentConfig that, when set, sources traces from a prior run's traces.jsonl instead of sampling a model.
• Introduces ReplayStore in agent.py to index saved traces by seat and task key, with early errors for ambiguous or missing lookups.
• Adds _ReplayEpisodeAgent, which returns the stored trace and fires callbacks without performing inference; calling provision() on it raises RuntimeError.
• Env.agents in env.py selects _ReplayEpisodeAgent when AgentConfig.replay is set, caching ReplayStore instances per path.
• AgenticJudgeEnv now raises ValueError at construction when shared_runtime=true and the solver is configured with replay, since no box exists to share.

📊 Macroscope summarized 9f38cd1. 3 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[macroscopeapp]

🟡 Medium v1/agent.py:435

ReplayStore keys each seat's saved traces by task name/idx only, so when the source run produced multiple rollouts for the same task (e.g. num_rollouts > 1), line 455 silently overwrites every earlier trace for that key with the last one. Each replay episode for that task then receives the same single last trace regardless of which source rollout it should correspond to, duplicating one sample and discarding the rest — biasing re-grading and rollout-group statistics. Consider keying by a per-rollout discriminator (e.g. an episode index) in addition to task name/idx, or preserving all traces per key and selecting by episode.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/agent.py around line 435:

`ReplayStore` keys each seat's saved traces by task name/`idx` only, so when the source run produced multiple rollouts for the same task (e.g. `num_rollouts > 1`), line 455 silently overwrites every earlier trace for that key with the last one. Each replay episode for that task then receives the same single last trace regardless of which source rollout it should correspond to, duplicating one sample and discarding the rest — biasing re-grading and rollout-group statistics. Consider keying by a per-rollout discriminator (e.g. an episode index) in addition to task name/`idx`, or preserving all traces per key and selecting by episode.

[mikasenghaas]

Parking this for now — focus is on the agentic judge (#2109); branch stays for later pickup.

[macroscopeapp]

🟡 Medium v1/agent.py:717

When two tasks in the dataset share the same non-empty name but have different idx values, ReplayStore indexes traces by name alone, so the second task's trace overwrites the first's in by_seat. Both tasks then replay the second task's trace — the first task's saved trace is silently lost. The lookup key should include idx (or another unique identifier) so tasks with the same name don't collide.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/agent.py around line 717:

When two tasks in the dataset share the same non-empty `name` but have different `idx` values, `ReplayStore` indexes traces by `name` alone, so the second task's trace overwrites the first's in `by_seat`. Both tasks then replay the second task's trace — the first task's saved trace is silently lost. The lookup key should include `idx` (or another unique identifier) so tasks with the same `name` don't collide.