Conversation
Signed-off-by: Sami Jaghouar <sami.jaghouar@gmail.com>
samsja
force-pushed
the
grpo-loss
branch
2 times, most recently
from
February 20, 2025 02:18
119c0da to
2befb70
Compare
Signed-off-by: sami jaghouar <sami.jaghouar@gmail.com>
Signed-off-by: Sami Jaghouar <sami.jaghouar@gmail.com>
Signed-off-by: Sami Jaghouar <sami.jaghouar@gmail.com>
kalomaze
added a commit
that referenced
this pull request
Jun 12, 2025
Co-authored-by: samsja <55492238+samsja@users.noreply.github.com>
samsja
added a commit
that referenced
this pull request
Jun 13, 2025
* formatask init * add task def * formatask configs * fix formatting?? * fix + overhaul difflib reward * complete reward function definition * autoclean rollouts #1 Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> * autoclean rollouts #2 Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> * expanded multiepoch dataset * ruff fix --------- Co-authored-by: kalomaze <66376113+kalomaze@users.noreply.github.com> Co-authored-by: kalomaze <juicebox22245@gmail.com>
samsja
added a commit
that referenced
this pull request
Nov 7, 2025
leonardtang
pushed a commit
to haizelabs/prime-rl
that referenced
this pull request
Jan 1, 2026
[fix] grading to take grading_config
samsja
added a commit
that referenced
this pull request
Mar 30, 2026
* bump verifier * add mika new env * add all envs * update envs * reduce sandbox resources and change sandbox label for `mini-swe-agent-plus` (#2) * update envs * bump prime-rl to main * update prime-rl * update prime-rl * update env --------- Co-authored-by: Mika Senghaas <mail@mikasenghaas.de> Co-authored-by: rasdani <73563550+rasdani@users.noreply.github.com>
S1ro1
added a commit
that referenced
this pull request
Apr 21, 2026
Hypothesis: orchestrator's /pause returns before inference CUDA work finishes draining. Trainer starts RDMA-writing while inference is still using that memory, causing non-deterministic partial corruption that compounds over steps. Fix: - Inference: cuda.sync before SPG barrier (drain in-flight work). - SPG barrier #1: signal "inference is quiescent, trainer can write". - Trainer: blocks at matching barrier, THEN posts writes + drains. - SPG barrier #2: signal "writes done" (existing). - Inference: cuda.sync after #2, then MLA absorb. Reverted iter14 knobs (flush_every=1, enforce_eager) to isolate the effect of the pre-write barrier alone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
seanbell
added a commit
to clouddatalabs/scalerl-prime-rl
that referenced
this pull request
Apr 25, 2026
snowflake_poc_critique.md findings PrimeIntellect-ai#2 and PrimeIntellect-ai#3 against arXiv:2510.13786: - Terminal-Bench handoff config now matches the paper's batch geometry: batch_size = 768 (= 48 prompts × 16 generations/step). The math smoke config keeps batch_size = 256 (16 prompts × 16) — already documented inline as a smaller-than-paper smoke choice; that comment stays. - Both configs now use trainer.scheduler.type = "linear" with warmup_steps = 100 and decay_steps = 0 to match the paper's 100-step linear warmup into the constant LR phase. torch.optim's LinearLR caps at end_factor=1.0 after total_iters, so warmup_steps=100 + decay_steps=0 is effectively warmup→constant. Comment in the math config explains the mechanism so future readers don't re-flatten to pure constant. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
snimu
added a commit
that referenced
this pull request
Jun 4, 2026
…#2/#5) - enabled_losses can now disable the primary loss for an env: when the term matching training_mode isn't enabled, train_sink zeroes that env's samples' completion_mask (after the decode/prefill metric), so the primary core trains nothing while echo (its own mask) still applies. [#2b] - setup_loss_fns no longer fabricates a default rl core when no rl/custom term exists; the rl core raises if an rl-mode batch is applied. [#2a, trainer-side] - OrchestratorConfig.validate_primary_loss: training_mode must have a matching term (rl/custom | sft | opd) in `losses`. [#2a, config-time] - LossList rejects more than one primary (rl/custom) term. [#5] - tests for the validators + the no-rl-term rl-core error. Not run locally; ruff + py_compile clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
snimu
added a commit
that referenced
this pull request
Jun 4, 2026
…sft warning - Drop sft/opd presets from the losses DSL: they dispatch to fixed cores by training_mode and are not loss-list terms, so listing them (or naming them in enabled_losses) was a silent no-op footgun. `losses` now holds rl/custom + echo only. [review #2] - Gate the zero-advantage filter on the env's rl primary being active (ZeroAdvantageFilter.primary_active, wired to train_sink._primary_enabled), so echo-only / rl-disabled envs (where advantage is irrelevant — e.g. group_size=1) aren't dropped on zero advantage. Default always-active → unchanged without echo. [review #1] - The MITO prompt-role-echo warning also fires for training_mode='sft' (which forces renderer=None after the warning's original check). [review #7] - tests for sft-type rejection + the filter gate. Left as-is: trainer.loss migration (intentional break), backfill prompt-role echo (best-effort, known), docs DPPO clipping (pre-existing main doc bug). Not run locally; ruff + py_compile clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
snimu
added a commit
that referenced
this pull request
Jun 5, 2026
…lters - #2 loss_overrides may only override fields the orchestrator applies per-env: the role/custom `filters` and a *constant* weight's `alpha`. Overriding `weight` type/tau/custom, `loss` (core), or `name` (all resolved globally) is now rejected at config time instead of silently ignored. - #3 filters chain by AND (intersection): to_echo_config intersects role filters' role sets (and tool_names) and keeps *all* custom filters; build_echo_annotations intersects their per-step masks (_apply_echo_filters). EchoLossConfig.filter -> filters (list). validate_supported rejects role filters that intersect to no roles. Multiple role/custom filters now narrow rather than union/last-win. Tests: loss-override unsupported-field rejection; disjoint-role-filter rejection; multiple-filter acceptance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
snimu
added a commit
that referenced
this pull request
Jun 10, 2026
…s to rl mode Two related bugs in primary λ/reduce/hooks handling (bug review #1 + #2): #1 — a custom-core primary (grpo advantage + `loss = custom`) is dispatched under "rl" via is_primary, but train.py picked the λ/reduce/hooks term by `loss.type == "dppo_kl"`, so a custom primary silently lost its lambda_weight / reduce / hooks. Now the tuning term is selected by is_primary (the dppo_kl lookup stays only for token-export's DPPO threshold annotations, which need that core). #2 — compute_loss applied primary_lambda/reduce/hooks to whatever core training_mode dispatched, including the fixed sft/opd cores, so a configured rl term's knobs could leak into sft/opd training. Gate them to training_mode == "rl" (sft/opd: λ=1, global-mean reduce, no hooks) — matching the "sft/opd cores are fixed" design. Default configs are unaffected (the leak only bit non-default rl tuning under sft/opd). Test: sft ignores a non-unit primary_lambda + a loss-zeroing primary hook. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.