Skip to content

feat(orchestrator): task sampler skeleton - #3262

Merged
hallerite merged 2 commits into
chore/rename-rollout-sourcefrom
feat/task-sampler-skeleton
Aug 13, 2026
Merged

feat(orchestrator): task sampler skeleton#3262
hallerite merged 2 commits into
chore/rename-rollout-sourcefrom
feat/task-sampler-skeleton

Conversation

@hallerite

@hallerite hallerite commented Aug 13, 2026

Copy link
Copy Markdown
Member

The TaskSampler foundation in one PR, three commits: free the name, build the memory, retire the filter pipeline. Picks are byte-for-byte identical to main by default — this is the observability half of the curriculum design (design/task-sampler.md): estimates get warm and visible before anything is allowed to steer sampling.

1. chore(orchestrator): rename Sampler to RolloutSource — pure rename. The class is one env's rollout source (which model generates its train rollouts); its docstring already said so. Frees "sampler" for the task-sampling layer.

2. feat(orchestrator): TaskSampler skeleton with per-task outcome statsTrainSource becomes TaskSampler: same pick contract to the dispatcher, plus a memory.

  • task_stats.py: discounted success/failure counts (Beta posterior with forgetting) keyed by (env, content-hash task key, agent role), plus reward mean/std EMAs. Keys recompute from the trace echo, so stats survive dataset reordering and resumes. Nothing is evicted — estimates decay back toward the prior.
  • TrainSink.on_group_finalized: one callback per finalized train group (all-errored ones included). Sink stays sampler-blind; sampler stays token-blind.
  • sampler/{env}/* metrics: pool occupancy (learnable/saturated/hopeless/unseen), coverage, realized_signal_rate, wasted_token_frac (inference tokens in zero-signal groups — the number later enforcement has to beat).
  • Checkpointing: stats ride inside the existing "train_source" key; older checkpoints load with empty stats and warm back up.

3. feat(orchestrator): retire the filter pipeline — the filter subsystem served three detectors; each gets a proper home and the pipeline (both config slots, double apply_filters pass, RolloutFilter protocol) is deleted. Group finalization becomes the pipeline's single decision point.

  • Zero-advantage splits into its three roles: evidence (groups feed the sampler), admission ([orchestrator.sampler] drop_degenerate_groups = true drops zero-signal groups so the batch backfills — the old pre-batch enforce + oversampling recipe), and the ship rule (Algorithm.trains_on_zero_advantage, default False, echo True — MaxRL's all-zero sentinel stops depending on a filter being configured; advantages=None always ships).
  • Gibberish/repetition demote to rollout-granularity detections ([[orchestrator.detections]]), evaluated at group finalization. An enforced detection ships no samples and never occupies a batch slot; its reward still counts toward the group baseline.
  • Rollout.filter_results/is_filtereddetections/is_excluded; pre_filters/* metrics → detections/*; five tracked configs migrated, no shim. Defaults map 1:1. One deliberate metrics change: zero-advantage rollouts stay in the effective subset (excluding all-correct groups clipped reward stats).

Overlaps in intent with #3254 (independently converged on trains_on_zero_advantage, killing the filter slots, and drop-with-backfill); diverges on configurability, drop granularity, and the evidence wire — reconciliation discussed there.

Verified: full non-GPU unit suite passes at each commit (143 orchestrator+config tests at the tip, incl. 5 new task_stats tests and the rewritten test_detections.py); known-environmental failures (qwen3_vl_e2e, test_load_configs, reverse_text) unaffected.

🤖 Generated with Claude Code

TrainSource becomes TaskSampler: same pick contract to the dispatcher
(weighted-RR env, epoch-shuffled cursor — behavior identical), plus a memory.
Every finalized train group now flows back through TrainSink's new
on_group_finalized hook into TaskStats: discounted success/failure
pseudo-counts (a Beta posterior with forgetting) keyed by
(env, content-hash task key, agent role), with reward EMAs alongside.

Always-on observability, nothing enforces: new sampler/{env}/* metrics report
pool occupancy (learnable/saturated/hopeless/unseen from the posterior),
coverage, realized signal rate, and the wasted-token fraction — the evidence
base for the upcoming weighted-sampling stage. Stats ride inside the existing
"train_source" checkpoint key; older checkpoints load with empty stats and
warm back up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hallerite
hallerite force-pushed the feat/task-sampler-skeleton branch from b1107dc to 04b5cfe Compare August 13, 2026 13:32
The filter subsystem served three detectors; each now has a proper home and
the pipeline itself (pre/post batch slots, double apply pass, RolloutFilter)
is gone:

- Zero-advantage splits into its three roles. Evidence: every finalized group
  already feeds the task sampler. Admission: [orchestrator.sampler]
  drop_degenerate_groups drops zero-signal groups at group finalization so
  the batch backfills (the old pre-batch enforce + oversampling recipe).
  Ship rule: Algorithm.trains_on_zero_advantage — a class declaration, not a
  run-level knob. MaxRL's all-zero sentinel stops depending on a filter being
  configured; echo's exception (observation CE trains through collapsed
  advantages) becomes automatic and per-env instead of a global enforce=false.
- Gibberish/repetition demote to arrival-time detections ([[orchestrator.detections]]):
  each reads only the rollout's own branches, so one pass at tokenization
  replaces both slots. One drop semantic replaces pre-vs-post: an enforced
  detection ships no samples and never occupies a batch slot (backfill), while
  the reward still counts toward the group baseline.

Rollout.filter_results/is_filtered become detections/is_excluded;
pre_filters/* metrics become detections/*; degenerate-group drops surface as
sampler/{env}/dropped_degenerate_groups. Default behavior is preserved: the
old post-batch enforcing zero_advantage maps 1:1 onto the ship rule, and
gibberish/repetition still default to monitoring. Tracked configs migrated in
place, no shim. One deliberate metrics change: zero-advantage rollouts now
stay in the effective subset — an all-correct group is a real outcome, and
excluding it clipped reward stats.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hallerite hallerite changed the title feat(orchestrator): TaskSampler skeleton with per-task outcome stats feat(orchestrator): task sampler skeleton Aug 13, 2026
@hallerite
hallerite deleted the branch chore/rename-rollout-source August 13, 2026 15:52
@hallerite hallerite closed this Aug 13, 2026
@hallerite hallerite reopened this Aug 13, 2026
@hallerite
hallerite merged commit c1053d5 into main Aug 13, 2026
20 checks passed
@hallerite
hallerite deleted the feat/task-sampler-skeleton branch August 13, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant