feat(orchestrator): multi-agent native metrics

[mikasenghaas]

Summary

Rollout metrics now mirror the episode/trace hierarchy that vf.Episode exposes, so a multi-agent env's seats never mix into one distribution.

Every metric key lives at exactly one of two levels:

• Episode level ({scope}/{subset}/...) — facts about the whole episode: the token/turn/branch counts, summed across the episode's traces (the same aggregation as Episode.num_turns / num_*_tokens).
• Agent level ({scope}/{subset}/<agent>/...) — everything trace-scoped, grouped by agent name (mirroring Episode.by_agent).

scope is train/agg (all train envs) or train/<env> / eval/<env>; subset is all or effective; <stat> is {mean,max,min,p10,p90}.

Before

Every metric sat at one flat level, pooling all of an episode's agents into a single distribution:

{train,eval}/{agg,<env>}/{all,effective}/
├── reward/<stat>                                  # train (eval: avg@k)
├── num_{total,input,output}_tokens/<stat>         # flat over traces
├── num_turns/<stat>  ·  num_branches/<stat>       # flat over traces
├── is_truncated/mean  ·  is_completed/mean
├── has_error/mean  ·  error/<type>                # `all` only
├── stop_condition/<name>
├── timing/{setup,agent,finalize,scoring,total}/<stat>
├── timing/agent/{model,harness}/<stat>
├── metrics/<name>/<stat>  ·  rewards/<name>/<stat>
├── solved_{all,none,some}
├── is_trainable/mean  ·  is_filtered/mean  ·  filters/<name>/mean   # train only
└── avg@k  ·  pass@k  ·  pass^k                                      # eval only

For a proposer-solver env that meant .../reward/mean averaged the proposer's reward together with its solvers', and .../num_turns/mean mixed a 1-turn proposer with 8-turn solvers — neither number describing anything real.

After

{train,eval}/{agg,<env>}/{all,effective}/
├── num_{total,input,output}_tokens/<stat>         # per-EPISODE sums
├── num_turns/<stat>  ·  num_branches/<stat>       # per-EPISODE sums
└── <agent>/                                       # one subtree per agent name
    ├── reward/<stat>
    ├── num_{total,input,output}_tokens/<stat>
    ├── num_turns/<stat>  ·  num_branches/<stat>
    ├── is_truncated/mean  ·  is_completed/mean
    ├── has_error/mean  ·  error/<type>            # `all` only
    ├── stop_condition/<name>
    ├── timing/{setup,agent,finalize,scoring,total}/<stat>
    ├── timing/agent/{model,harness}/<stat>
    ├── metrics/<name>/<stat>  ·  rewards/<name>/<stat>
    ├── solved_{all,none,some}
    ├── is_trainable/mean  ·  is_filtered/mean  ·  filters/<name>/mean   # train only
    └── avg@k  ·  pass@k  ·  pass^k                                      # eval only

A single-agent env has one subtree, named for its one agent (usually agent):

train/reverse-text/effective/num_turns/mean               # episode
train/reverse-text/effective/agent/reward/mean            # trace
train/reverse-text/effective/agent/timing/agent/model/mean
eval/reverse-text/effective/agent/avg@16

A proposer-solver env separates the seats, and the episode level sums them:

train/proposer-solver+gsm8k-v1/effective/num_turns/mean            # = proposer + solvers
train/proposer-solver+gsm8k-v1/effective/proposer/num_turns/mean
train/proposer-solver+gsm8k-v1/effective/solver/num_turns/mean     # over the solver traces
train/proposer-solver+gsm8k-v1/effective/solver/reward/mean
train/proposer-solver+gsm8k-v1/all/judge/is_trainable/mean         # 0.0 — the judge never trains

Aggregation rules

One aggregation per level, so a metric's unit is never ambiguous:

• Episode level — one value per episode, summing its traces. Rollouts without an episode_id (legacy envs, synthesized error markers) count as single-trace episodes.
• Agent level — one value per trace, flat over that agent's rollouts. An in-episode fan-out (n solvers) simply contributes n samples; weighing whole episodes against each other is the episode level's job, and the trace is also what the advantage computation samples.
• Rates (is_truncated, is_completed, has_error, is_trainable, is_filtered, filters/<name>) emit /mean only — a 0/1 distribution's other stats carry no information.
• avg@k's k is the largest number of traces one example drew, so it matches the sample count the value averages.

Also

• RolloutMetrics is renamed EpisodeMetrics; the per-agent view is TraceMetrics.
• The W&B overview reward/error/truncation/avg@k panels match the per-agent keys by regex (one panel per agent, env names regex-escaped); the changed panel set rolls the saved view via view_signature.
• The monitor-run skill documents both levels.
• Episode wording cleanups in envs.py / dispatcher.py (no more "env-rollout").

Breaking

Trace-level metrics move under the agent subtree. These keys no longer exist at {scope}/{subset}/:

reward/<stat>            → {scope}/{subset}/<agent>/reward/<stat>
is_truncated/mean        → {scope}/{subset}/<agent>/is_truncated/mean
is_completed/mean        → {scope}/{subset}/<agent>/is_completed/mean
has_error/mean           → {scope}/{subset}/<agent>/has_error/mean
error/<type>             → {scope}/{subset}/<agent>/error/<type>
stop_condition/<name>    → {scope}/{subset}/<agent>/stop_condition/<name>
timing/<phase>/<stat>    → {scope}/{subset}/<agent>/timing/<phase>/<stat>
metrics/<name>/<stat>    → {scope}/{subset}/<agent>/metrics/<name>/<stat>
rewards/<name>/<stat>    → {scope}/{subset}/<agent>/rewards/<name>/<stat>
solved_{all,none,some}   → {scope}/{subset}/<agent>/solved_{all,none,some}
is_trainable/mean        → {scope}/{subset}/<agent>/is_trainable/mean
is_filtered/mean         → {scope}/{subset}/<agent>/is_filtered/mean
filters/<name>/mean      → {scope}/{subset}/<agent>/filters/<name>/mean
avg@k                    → {scope}/{subset}/<agent>/avg@k
pass@k  ·  pass^k        → {scope}/{subset}/<agent>/pass@k  ·  .../pass^k

Single-agent envs read them under their one agent name (usually agent).

One further break: the count metrics at {scope}/{subset} are distributions over episodes (per-episode sums), no longer over loose traces. Numerically unchanged for single-agent envs; multi-agent envs previously mixed every agent's traces into one distribution.

Verification

• uv run pytest tests/unit/orchestrator tests/unit/utils (159 passed), including a proposer-solver fan-out test asserting episode sums, per-agent means, and the agent-only key set.

• ruff format --check / ruff check on the touched files.

• Live runs in wandb: episode-agent-metrics, one per shape. The env level is the same 50 keys in both — it does not grow with agent count — while the agent level fans out:

  run	keys	env-level	agent-level
  single-agent-reverse-text	215	50	165 under agent/
  two-agent-agentic-judge	313	50	263 under solver/ + judge/

  The two-agent run shows the split the flat layout could not express, and the episode level summing it:

  all/num_turns/mean            = 8.889   # the episode: solver + judge
  all/solver/num_turns/mean     = 1.000
  all/judge/num_turns/mean      = 7.889
  all/solver/is_trainable/mean  = 0.139
  all/judge/is_trainable/mean   = 0.000   # a frozen seat never trains
  

  (That judge run also carries a 75% rollout error rate — its max_turns is set too low in a local-only debug config, unrelated to this change. Errors are reported faithfully as has_error/mean on both subtrees.)

• A longer 20-step reverse-text run on an earlier revision of this branch: https://wandb.ai/primeintellect/reverse-text/runs/0988528e8b62439097ad6f1557afd571

🤖 Generated with Claude Code

---

Note

Medium Risk
Breaking W&B metric key paths affect dashboards and alerts; orchestrator logging behavior changes but training logic is unchanged—risk is mainly observability and downstream consumers of old metric names.

Overview
Rollout metrics now follow the episode/trace hierarchy instead of one flat distribution over all traces. Episode-level keys ({scope}/{subset}/num_*, num_turns, num_branches) aggregate per episode by summing traces; trace-level metrics (reward, truncation, errors, timing, filters, eval scores) emit under {scope}/{subset}/<agent>/..., grouped by agent.name with one sample per trace.

RolloutMetrics becomes EpisodeMetrics plus TraceMetrics per agent; train/eval to_wandb and pass_at_k are scoped per agent. W&B overview panels use regexes for per-agent reward, has_error, truncation, and avg@k; monitor-run documents the two-level key layout. Minor episode wording in dispatcher.py / envs.py.

Breaking: flat keys like {scope}/{subset}/reward/mean move to {scope}/{subset}/<agent>/reward/mean (single-agent runs typically use agent). Top-level token/turn stats are over episodes, not loose traces (unchanged for single-agent, different for multi-agent).

^{Reviewed by Cursor Bugbot for commit 46fbe0d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 46fbe0d. Configure here.}

[cursor]

Synthetic errors mis-attributed by agent

Medium Severity

by_agent groups every rollout by agent.name, including dispatcher-synthesized error and cancel markers that use a default AgentInfo. In multi-agent envs those failures land under a default seat (typically agent) instead of the real seats, so per-agent has_error and related rates miss complete episode failures and cancellations.

 

^{Reviewed by Cursor Bugbot for commit 46fbe0d. Configure here.}

[mikasenghaas]

ouf ya, this is annoying. externally failed episodes should get a separate type (not try to imitate the internal data type)

[mikasenghaas]

will fix this in follow up

[mikasenghaas]

Fixed in #3183 — cancellations and task failures now arrive as a first-class EpisodeFailure on an EpisodeResult envelope rather than a stand-in Rollout, so they are counted at the episode level and can no longer land in a phantom agent subtree.