Repository navigation
RFC 013: Task sampling and in-training evaluation - #1256
Matheus-F-Scatolin wants to merge 6 commits into
Conversation
Proposes a TaskSampler protocol (sequential, uniform, cost-aware curriculum), a generic TaskAPIClient, an EnvEvalHarness built on the collect pipeline, EvalSchedule, and an `openenv eval` command. Includes preliminary results from a single-seed pilot (Llama-3.2-1B, GRPO + LoRA, reasoning_gym_env) and lists the index entry in rfcs/README.md. Co-Authored-By: Tiago Perrupato Antunes <tiagoperrupato@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a95de40. Configure here.
| curriculum.set_attr_level("num_terms", terms) | ||
| curriculum.set_attr_level("num_digits", digits) | ||
| return curriculum.generate_configuration( | ||
| context=DefaultCurriculumContext(mode=RangeAttributeMode.UPPER_BOUND)) |
There was a problem hiding this comment.
Example tiers nest easier difficulties
Medium Severity
tier_config builds each tier with RangeAttributeMode.UPPER_BOUND, so harder keys only raise max_terms/max_digits and keep the easy minima. Harder tiers then mix in easy questions, which conflicts with the discrete 2x1…7x5 mapping and the monotonically falling per-tier scores in the same section.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a95de40. Configure here.
There was a problem hiding this comment.
Each tier is a single difficulty here. With RangeAttributeMode.UPPER_BOUND and ensure_interval left at its default (False), reasoning-gym sets the lower bound to the same level as the upper one (lo_index = max(0, hi_index) in coaching/base_curriculum.py). With reasoning-gym 0.1.25, tier_config(t) gives min_terms == max_terms and min_digits == max_digits on all ten tiers, from 2 terms of 1 digit (tier 0) to 7 terms of 5 digits (tier 9), so a harder tier does not mix in easier questions.
|
We pushed a revision of the RFC with results on three more seeds; the list of changes is in the description. The pilot's per-token result did not repeat. Difference to uniform sampling, in points of test success (500 questions), as the mean of the evaluations at 4 to 5 million generated tokens ("per token") and at steps 100 to 700 ("per episode"):
Per episode, seed 2 is read at steps 100 to 700 for each pair of runs that reached step 700. The rule we wrote before the run read every run of a seed only up to the step all of them reached; one run of seed 2 stopped at step 223, and that rule gives +3.9 for the curriculum, with the same conclusion. Three new seeds are a pattern, not a significance test. What changed in the proposal:
Three questions for the maintainers are in #1445. The trial's code and logs are at Matheus-F-Scatolin/OpenEnv@rfc-013-trial. |
There was a problem hiding this comment.
Thanks @Matheus-F-Scatolin @tiagoperrupato for the RFC and the trial data. The TaskSampler protocol, TaskAPIClient, the eval harness and its reproducibility checks look good to me. I'm requesting changes on the cost path only, with details inline:
- Keep cost out of the sampling weights. I found no lab report or paper that divides a task's sampling weight by its cost. Labs that handle cost keep per-source shares and give slow sources more rollout capacity (MiMo-V2.6, LongCat-Flash-Thinking-2601, MiniMax M2), and budget-aware methods move rollouts toward hard tasks (Knapsack RL). With dollar costs from an environment (#1382), dividing would starve the expensive, hard tasks. Costs should be reported, and used for budgets and concurrency.
- State the trainer hooks a sampler needs. It needs to draw prompts from
sample(n)and to callupdateon every scored group before dynamic-sampling filters such as huggingface/trl#7609 drop the all-pass and all-fail groups. Aligning with prime-rl'sTaskSamplerwould help, andCurriculumSamplercan stay an example (open question 9). - Cite the trainer-side pieces rather than define them. Post-generation filtering and per-environment mixture shares are enforced by the trainer's rollout scheduler, and the
groupkey already carries the metadata those shares need.
| p̂_k = (s_k + 1) / (s_k + f_k + 2) # Beta(1, 1) prior | ||
| ``` | ||
|
|
||
| A rollout counts as a success when its reward is at least `success_threshold` (default `1.0` for binary rewards; configurable for graded rewards). The decay `γ` (default `0.9` per update) lets the estimate follow a policy that is improving. |
There was a problem hiding this comment.
For graded rewards, within-group reward variance (the same quantity a zero-variance filter computes) avoids choosing a success_threshold, and would answer open question 5.
There was a problem hiding this comment.
Agreed as a direction, and our logs support it for a second reason: the formula over-predicts the share of groups with signal by about 2× on tiers 2 to 4. The outcomes inside a group are correlated, in large part, we think, because the questions of one tier differ (section 4 now gives the numbers and the caveat; the tables are in notes/results/calibration.md on the trial branch). A decayed share of groups whose rewards vary would be calibrated by construction and would need no success_threshold. Two caveats, both now in open question 5: it trades the threshold for a tolerance on the spread of the rewards, and with continuous rewards almost every group varies a little, so the tolerance or the weight would have to follow the reward scale. We could not test the graded case: about 0.1% of chain_sum's training answers got partial credit. On binary rewards it would change the weights we measured, mostly by lowering those of the solved tiers, so 2a5bbf6 records it as the proposed direction for open question 5 and keeps the measured sampler for the reference experiment.
| The `cost` argument selects the unit in which the sampler counts and reports cost: | ||
|
|
||
| - `"episodes"` (default): every rollout costs 1. The weights are the ones above. | ||
| - `"tokens"`: the trainer passes the number of generated tokens per rollout. The sampler keeps a decayed mean cost per rollout, `ĉ_k ← γ · ĉ_k + (1 - γ) · mean(costs)`; a key with no cost yet uses the mean of the keys that have one, or 1 before any has. It weights by signal per token, in this form: |
There was a problem hiding this comment.
Suggest dropping cost="tokens" as a weighting mode, along with the corrected-form paragraphs below, and keeping cost for counting and reporting only. I found no lab report or paper that divides a task's sampling weight by its cost. The ones that model cost do the opposite:
- MiMo-V2.6 keeps per-source shares fixed and gives slower sources more concurrent rollouts: https://arxiv.org/abs/2610.11959
- LongCat-Flash-Thinking-2601 raises the rollout quota for low-throughput domains: https://arxiv.org/abs/2601.16725
- MiniMax M2 found that batches tilted toward short, easy tasks destabilised training: https://arxiv.org/abs/2605.26494
- Knapsack RL uses this same signal term to allocate rollouts per task under a budget, which moves rollouts toward hard tasks: https://arxiv.org/abs/2509.25849
The trial agrees: the corrected form only matched "episodes". With dollar costs from an environment, dividing would starve the expensive, hard tasks.
There was a problem hiding this comment.
Done in 2a5bbf6: cost="tokens" and the corrected-form paragraphs are gone, and the cost argument with them. The sampler counts the costs it is given and reports signal per unit of cost; the measured forms move to Alternatives Considered with their numbers.
The trial agrees, as you say. Per token the corrected form was neither better nor worse than the signal-only weights (+0.1, −3.1 and +1.6 points) by the thresholds we wrote before the trial, though that comparison was exploratory, and it got fewer groups with signal per generated token on two of three seeds: the policy changes its answer length, so a decayed mean of past lengths does not control what the next answers cost. The trial also shows the shift you expect: both token forms moved episodes away from the two hardest tiers (25% to 34% of the episodes under the first form and 47% to 56% under the corrected one, against 52% to 60% under the signal-only weights). Notes on the sources are in the summary comment.
|
|
||
| The first version of this RFC divided `(signal + ε)` by `ĉ_k`, with the fixed prior above. Measured, that sends many episodes to keys the policy already solves: their completions are the shortest, the fixed prior leaves them a signal of about 0.09 that never reaches 0, and `ε` divided by a small cost is large. In a 100-step probe it gave the solved tier 28.5% of the episodes; in full runs, 14% to 22%, against 7% for the episode weights. The form above keeps `ε` out of the division and lets the prior fade; it gave that tier 5% to 8%. The floor alone is not enough: with the fixed prior it still gave the solved tier 21.2% in its probe. In the trial, the corrected token weights matched the episode weights per generated token, without beating them (Preliminary Results). They stay available for environments where completion length varies much more across tasks than in that experiment. | ||
|
|
||
| Costs come from the trainer, which knows them: the number of tokens each completion generated (the length of its generated ids, never a count made by tokenizing the text again), the token records that RFC 006's harness already carries, or a spend that the environment reports with each episode, as issue #1382 proposes (`metadata["service_spend"]`). The sampler never sees rollouts, so it takes no cost function. |
There was a problem hiding this comment.
| Costs come from the trainer, which knows them: the number of tokens each completion generated (the length of its generated ids, never a count made by tokenizing the text again), the token records that RFC 006's harness already carries, or a spend that the environment reports with each episode, as issue #1382 proposes (`metadata["service_spend"]`). The sampler never sees rollouts, so it takes no cost function. | |
| Costs come from the trainer, which knows them: the number of tokens each completion generated (the length of its generated ids, never a count made by tokenizing the text again), the token records that RFC 006's harness already carries, or a spend that the environment reports with each episode, as issue #1382 proposes (`metadata["service_spend"]`). The sampler counts and reports costs but never weights by them; trainers use them for budgets, concurrency and rollout allocation. The sampler never sees rollouts, so it takes no cost function. |
There was a problem hiding this comment.
Applied in 2a5bbf6, moved before "Costs come from the trainer", with two edits. It says "The CurriculumSampler counts and reports costs but does not weight by them" instead of "The sampler … never weights by them", because the RFC leaves room for another TaskSampler that accounts for cost (D2, Alternatives Considered), although it proposes none. And one added sentence points to the measured forms in Alternatives Considered. It now also says that a trainer reports costs for every group or for none.
|
|
||
| A plain Python loop, an MLX loop, or a TRL callback calls `maybe_run(step)` after each optimizer step. The policy under evaluation is served through an OpenAI-compatible endpoint (vLLM, `mlx_lm.server`, or any other server), so the harness does not depend on how the trainer holds the weights. `on_result` receives the step, and the trainer should log the sampler's `episodes` and `cost` counters next to it, so learning curves can be drawn against episodes and against cost, not only against steps or wall-clock time. | ||
|
|
||
| With TRL's `GRPOTrainer`, the sampler sits where the training prompts come from, and the function that returns the environment's rewards to the trainer also passes them, with the completions' token counts, to `update`. Rollout step 6 adds a TRL example that shows this end to end. |
There was a problem hiding this comment.
GRPOTrainer has no place for a sampler today. Prompts come from RepeatSampler, and async GRPO reads the dataset in order. TRL's open dynamic-sampling PR (huggingface/trl#7609) drops all-pass and all-fail groups in the rollout worker before anything else sees them, and those are exactly the groups that tell a curriculum a task is solved or out of reach.
A sampler needs two trainer hooks:
- drawing prompts from
sample(n); - calling
updateon every scored group, before that filter.
Suggest stating this here, and aligning the protocol with prime-rl's TaskSampler (__next__, observe(group), state_dict, metrics): https://github.com/PrimeIntellect-ai/prime-rl/blob/855a2b815b80/src/prime_rl/orchestrator/curriculum/samplers/base.py
That also answers open question 9: keep CurriculumSampler as an example.
There was a problem hiding this comment.
Agreed on the two hooks. 2a5bbf6 states them in Trainer integration, with prime-rl's Curriculum.on_result and huggingface/trl#7609 as examples, and says that the trial's trainer calls update on every group before it drops the groups without signal.
On TRL: it has no sampler API, but from reading its code, a datasets.IterableDataset that draws its rows from sample(n) and a reward function that calls update would connect one, since the reward functions receive the dataset columns and the generated ids. The RFC lists the limits we found: the prompts are drawn one or two rounds before they are generated, so an update first affects the draws one or two rounds later; shuffle_dataset must be off; on several processes the copies of the sampler must draw the same tasks while each reward function sees only its own process's rollouts, so the path works on one process; and the asynchronous trainer reads the dataset in a separate rollout process, where the sampler would have to live. We have not run it, so a TRL example of the sampler is left for later; rollout step 6 keeps a TRL callback for EvalSchedule.
On prime-rl: 2a5bbf6 adds metrics() and says the protocol has the shape of its TaskSampler, with two differences. sample(n) draws the tasks of a step at once, which is how a synchronous trainer asks for them, and a trainer that opens one group at a time calls sample(1), as the __next__ of the wrapper the RFC sketches for prime-rl would (for CurriculumSampler the two give the same distribution but a different sequence for a seed, and the RFC's results used sample(n)). And update takes rewards and costs instead of verifiers episode objects, so core depends on no training framework. Wrapping one of our samplers for prime-rl is a short class plus a change in prime-rl (a config class and a branch in Curriculum.__init__), since its sampler configs are a closed set; the other direction needs verifiers episode objects.
Open question 9: agreed. The RFC offers CurriculumSampler as an example, and #1445 asks the maintainers to confirm.
|
|
||
| - **Chosen approach**: Samplers run in orchestration code and hand each task to the session factory, which turns it into what the environment needs: `reset(split=..., index=...)` for an environment that follows the Task API guide. Environments only publish task metadata. | ||
| - **Rationale**: This keeps the invariants intact. Rewards are still computed inside the environment; the sampler only reads them. The agent has no path to influence which task it gets, because selection goes through the session factory and `reset()`, which stay on the infrastructure side of the dual API boundary. The environment server stays free of cross-episode state, which also keeps "one env = one trajectory" and stacking of environment instances simple. It is also where task selection already happens today: the Harbor harness picks `indices` on the trainer side. | ||
| - **Trade-offs**: Each trainer process holds its own statistics. A distributed trainer with many rollout workers must aggregate them; `state_dict` gives a merge point, but a shared sampler service is out of scope for this RFC. |
There was a problem hiding this comment.
Two trainer-side limits worth naming here:
- Mixture shares: labs set per-environment shares and enforce them in the rollout scheduler, not the sampler. MiMo-V2.6 raises concurrency for slow sources (https://arxiv.org/abs/2610.11959); INTELLECT-3 uses a static per-environment ratio (https://arxiv.org/abs/2512.16144).
- Async bias: fast tasks finish first, so per-key estimates lean toward them. MiniMax waits for slow tasks with a windowed FIFO (https://arxiv.org/abs/2605.26494).
The group key is already the metadata trainers need, so I'd cite these rather than define shares in OpenEnv.
There was a problem hiding this comment.
Agreed. 2a5bbf6 cites them in Trainer integration instead of defining shares: shares across sources stay with the trainer and its rollout scheduler (MiMo-V2.6's concurrency per source, LongCat's rollout quota, prime-rl's draw of the source before the task; INTELLECT-3's paper says only that its mix was tuned, so the RFC names prime-rl's fixed ratio instead), and a sampler works inside one source's share. On the group key: in the RFC it names a family of tasks inside one environment, so shares across environments still need the environment's identity, which the trainer has. The async bias is now a stated limit: updates arrive late and out of order, and a seed reproduces a run only in a synchronous trainer such as the trial's.
|
|
||
| #### D2. The curriculum targets GRPO signal, and reports it per unit of cost | ||
|
|
||
| - **Chosen approach**: Weight keys by the probability that a group of size `G` has non-zero reward variance. Count cost in a unit the trainer chooses and report signal per unit of cost. Weight by signal per token only with `cost="tokens"`, in the corrected form of section 4. |
There was a problem hiding this comment.
With cost out of the weights, the title can become "The curriculum targets GRPO signal, and reports cost".
| - **Chosen approach**: Weight keys by the probability that a group of size `G` has non-zero reward variance. Count cost in a unit the trainer chooses and report signal per unit of cost. Weight by signal per token only with `cost="tokens"`, in the corrected form of section 4. | |
| - **Chosen approach**: Weight keys by the probability that a group of size `G` has non-zero reward variance. Count cost in a unit the trainer chooses and report signal per unit of cost, without weighting by it. |
There was a problem hiding this comment.
Applied in 2a5bbf6, with the title you proposed.
| ## Alternatives Considered | ||
|
|
||
| - **Curriculum inside the environment.** The server would track success rates and pick the next task itself. Rejected: it makes the environment stateful across episodes, mixes orchestration into the environment boundary, and breaks when many trainer workers share or stack environment instances. | ||
| - **Only filtering zero-signal groups after generation (DAPO-style).** This is complementary, not a replacement: it still pays for generating the discarded groups. A trainer can use both. |
There was a problem hiding this comment.
Suggest citing the trainer implementations: TRL's async GRPO filter (huggingface/trl#7609) and prime-rl. The TRL filter doesn't refill, so groups_per_step shrinks. That makes the sampler the piece that avoids generating the dropped groups, which is the strongest argument for it.
There was a problem hiding this comment.
Done in 2a5bbf6, citing huggingface/trl#7609 and prime-rl's gate. 2a5bbf6 also changes the Motivation, which said a sampler avoids generating these groups, to say that it generates fewer of them: with the curriculum, about two thirds of the groups still had no signal in our trial. Our trial's trainer does not replace dropped groups, and uniform sampling made no update at all on 19% to 21% of its steps.
| 3. Where should shared sampler state live for distributed trainers with many rollout workers? | ||
| 4. Should `openenv eval` results be pushable to the Hub in a schema that the environments datasets RFC (#795) can reference? | ||
| 5. What is the right default for `success_threshold` on environments whose rewards are graded rather than binary? | ||
| 6. Should `cost` stay `"episodes"` by default, or should environments declare their dominant cost (for example in their manifest) so trainers can pick a sensible unit? A spend that the environment reports with each episode, as #1382 proposes, would be one such declaration. |
| - Environments datasets RFC (#795): possible home for published evaluation results. | ||
| - Task API guide (`docs/source/guides/task-api.md`) and the Harbor harness (`envs/harbor_env/harness.py`): the current Task API contract and the manual task selection this RFC generalizes. | ||
| - Training guide (`docs/source/guides/training.md`): training stays in the user's framework, and core depends on no trainer; the samplers and `EvalSchedule` follow that. | ||
| - Issue #1382: a proposal for environments that report a per-episode spend, which a trainer can forward to `update` as costs. |
There was a problem hiding this comment.
| - Issue #1382: a proposal for environments that report a per-episode spend, which a trainer can forward to `update` as costs. | |
| - Issue #1382: environments that report a per-episode spend, which a trainer can count as cost and use for budgets and concurrency. |
Dividing a task's sampling weight by its cost starves expensive, hard tasks, and no lab report or paper found does it. service_spend is for accounting and for planning budgets and concurrency; a trainer can still report it to RFC 013's update() as costs. Refs openenv#1382, openenv#1256
|
Thanks @rycerzes for the careful review. 2a5bbf6 revises the RFC:
On the sources: we read the five reports, huggingface/trl#7609 and prime-rl's sampler. For lab practice they support what you describe: none of them divides a task's sampling weight by its cost. Outside lab reports the idea does exist. Graves et al. (2017) divide a task's learning progress by the time a sample of it takes to process (the length of its longest input sequence), and two 2026 preprints that apply the idea to GRPO, DUET and Cost-Aware Learning, divide a usefulness term by the square root of the cost, to allocate rollouts within a batch and to sample update minibatches. Cost-aware Bayesian optimization also documents the failure you expect, in best-point search: a plain value/cost ratio keeps choosing cheap, low-value points (Lee et al., 2020; Astudillo et al., 2021). The RFC cites Graves et al., DUET and Cost-Aware Learning in Alternatives Considered; the last two allocate rollouts or minibatches, not tasks. One nuance on MiniMax-M2: it gives the instability of batches tilted toward short, easy tasks as the reason for its windowed FIFO, without a measurement, and that tilt comes from completion order in an asynchronous pipeline, not from a constant reweighting such as dividing by cost. |
rycerzes
left a comment
rycerzes
left a comment
There was a problem hiding this comment.
Thanks for addressing the issues @tiagoperrupato @Matheus-F-Scatolin
you can resolve the review conversation since they are outdated now
One suggestion: keep the RFC to OpenEnv's side. The prime-rl wrapper sketch (L155) and the TRL IterableDataset path (L257) describe those trainers' internals and would fit better in their repos or in the examples/ of rollout step 6. The two hooks (L255) and the notes on what stays with the trainer (L259, L261) are enough here.
Summary
Adds RFC 013, which proposes trainer-side task sampling and in-training evaluation on top of the existing Task API: a
TaskSamplerprotocol with sequential and uniform samplers in core, a curriculum sampler offered as an example, a genericTaskAPIClient, anEnvEvalHarnessbuilt from theopenenv collectpipeline's pieces, anEvalSchedule, and anopenenv evalcommand. It targets the "simple evals that can be scored by a script" step of RFC 000's phase three. This PR only adds the RFC and its index entry; no code changes.The RFC includes results measured on a laptop (Llama-3.2-1B, GRPO + LoRA in MLX,
reasoning_gym_envchain_sumin five difficulty tiers): a single-seed pilot, then a trial on three more seeds, with predictions and the rules for reading the results written down before it ran. A curriculum that targets GRPO signal learned more than uniform sampling per environment episode on all four seeds, and per generated token on the three new ones (+5.5, +18.3 and +14.4 points); on the pilot's seed, uniform sampling was ahead per token by 7.7 points. Three new seeds are a pattern, not a significance test. Weighting the curriculum by token cost with the formula of the first version did worse than the signal-only curriculum, and a corrected form matched it per generated token without beating it, so theCurriculumSamplerweights by signal only and the samplers proposed here report signal per episode and per unit of the cost the trainer reports.Pilot code and logs: Matheus-F-Scatolin/OpenEnv@rfc-013-pilot. Trial code and logs: Matheus-F-Scatolin/OpenEnv@rfc-013-trial.
Questions for the maintainers are in #1445.
Changes since 2026-09-27
costargument from the curriculum, stated the two hooks a sampler needs in a trainer and their limits in TRL, addedmetrics()to the protocol, left shares across sources to the trainer, reported how far the signal formula over-predicts, cited prior work on rollout allocation and filtering, narrowed the reference experiment to uniform sampling against the curriculum on five new seeds, answered open question 6 and proposed a direction for open question 5.Type of Change
Alignment Checklist
Before submitting, verify:
PRINCIPLES.md) and this PR aligns with our principlesINVARIANTS.md) and no invariants are violated (one tension is flagged below)lint.shand tests) and addressed all issues. Not run: the PR changes only Markdown inrfcs/, whichlint.shand the tests do not cover, so it adds no new finding.RFC Status
Test Plan
Docs only. To rerun the pilot (Apple Silicon, MLX needs the Metal GPU):
The pilot's script writes today's date into every prompt, through the model's chat template, and stops after 150 minutes of training, so a rerun on another day or another machine gives other numbers than the stored logs. The trial's trainer fixes the date and stops on generated tokens; run again on seed 0, it reproduced the pilot's stored logs step for step for the first 370 steps of uniform sampling and 158 of the curriculum. To regenerate every table and figure of the trial from its stored logs, without a GPU:
The README of that branch gives the commands that ran the trial itself.
Alignment Review
Alignment review done by hand against this diff, following the project's alignment-review checklist.
Automated Checks
rfcs/Open RFCs Context
update.reasoning_gym_envas a partial answer to its open question 2 (cheap, but single-turn with no tool use).openenv evalresults (open question 4).update.Tier 1: Fixes Required
None: the PR changes only Markdown.
Tier 2: Alignment Discussion
Principle Conflicts
ALIGNMENT FLAG: Task metadata over HTTP
/reset,/step,/state#1156)TaskAPIClientconsumes the existing HTTP-only Task API routes, as documented in the Task API guide. The RFC adds no routes, but it adds a core consumer of them. D5 isolates this in one class so a later WebSocket move changes onlyTaskAPIClient.ALIGNMENT FLAG: Evaluation follows the collect pipeline
openenv collectresolves onlyopenspiel:andreasoning_gym:specs,CollectRunnerpasses no seed and records no failed episode (rollout step 5 adds both), and Harbor, which runs a whole agent per call, is not covered by the first version (D3, open question 8).RFC Conflicts
ALIGNMENT FLAG: Task API helpers in core
EnvClient,SyncEnvClient) do not ship Task API methods, because task specs are environment-specific")TaskAPIClienttoopenenv.core. D6 argues it keeps specs untyped and adds nothing toEnvClient, so the guide's reasoning still holds, but this reverses a documented choice and should be agreed explicitly.No conflicts found with the dual API boundary, rewards-in-environment, client-server separation, or "one env = one trajectory": samplers run on the orchestration side and only read rewards the environment computes, and a reward the environment did not give is left out, never counted as zero.
Summary
Disclosure
#1250 and #1252, cited in the RFC, are by one of the authors. This PR was prepared with AI assistance.
Note
Low Risk
Markdown-only RFC and index entry; no production code, behavior, or dependency changes.
Overview
Adds RFC 013 (documentation only) and lists it under Reward & Evaluation in
rfcs/README.md. No runtime or API code ships in this PR.The RFC proposes trainer-side task sampling and in-training evaluation on top of the Task API: a
TaskSamplerprotocol (SequentialSampler,UniformSamplerin core; optional exampleCurriculumSampler), genericTaskAPIClient,EnvEvalHarnessextending existingEvalHarness,EvalSchedule, and anopenenv evalCLI—aligned with RFC 000 phase-three “simple script-scored evals.”It documents design choices (sampler on orchestration side, eval reusing collect pipeline pieces, reproducibility probes, HTTP Task API for now) and a six-step rollout plan for follow-up PRs. Empirical GRPO/MLX results on
reasoning_gym_envjustify a signal-targeting curriculum over uniform sampling and drop cost from curriculum weights while still logging cost from the trainer. Open questions flag HTTP vs WebSocket metadata, Harbor-style evals, and whetherCurriculumSamplerbelongs in core.Reviewed by Cursor Bugbot for commit f3c0589. Bugbot is set up for automated code reviews on this repo. Configure here.