Repository navigation
refactor(configs): one trainer-common config base for TrainerConfig and SFTConfig - #3849
Merged
Merged
Conversation
faresobeid
marked this pull request as ready for review
October 4, 2026 12:32
This was referenced Oct 4, 2026
faresobeid
force-pushed
the
refactor/trainer-config-base
branch
from
October 8, 2026 14:13
558057d to
806a781
Compare
This was referenced Oct 8, 2026
samsja
added a commit
that referenced
this pull request
Oct 10, 2026
Replaces the three switches `model.fsdp_cpu_offload`, `model.optim_cpu_offload` and `model.full_offload` with one field: ```toml [model] offload = "optimizer" # default; also "full", "none" ``` This applies to both the RL trainer and SFT. The `fsdp` switch is not migrated: the mode is removed outright (rationale below). The `[model.full_offload] numa_bind` knob is dropped: no recipe, deployment, template or doc sets it. Full offload now always pins each rank's CPUs to its GPU's NUMA node. That was already the default. Ranks whose GPU reports no NUMA node still skip binding with a warning. On hosts without GPU sysfs topology (`/sys/bus/pci/devices/<bus>/numa_node`), binding is no longer optional: full offload fails there instead of being switched off. Why: the three switches could only be used one at a time, enforced by `cpu_offload_mutual_exclusion`. Because `optim_cpu_offload` defaulted to `true`, enabling full offload also meant writing `optim_cpu_offload = false`, otherwise validation failed (see the old `configs/advanced/deepseek-v4-flash/sft.toml` comment). With one value the conflicting combinations cannot be expressed, so that validator is deleted. The default behaviour does not change: optimizer-state offload stays on. Why drop `fsdp`: the mode was the stock torch FSDP2 `CPUOffloadPolicy` with no in-tree config, example, or test user; Muon rejected it and it broke NCCL weight broadcast. It was kept only for long-context runs whose fp32 masters/moments no longer fit on GPU (the reason it was restored in #3052); the `full` mode added in #3234 keeps FP32 masters, moments, and gradients in host RAM, covering that memory pressure instead. Gone with it: the `CPUOffloadPolicy` selection in `setup_fsdp`, CPU-side model init in `setup_model`/`load_dcp_from_hf`, `_move_buffers_to_cuda`, and the fsdp leg of the Gloo special case. Changes: - `ModelConfig.offload: Literal["none", "optimizer", "full"] = "optimizer"` replaces the three fields. `OptimizerInBackwardOffloadConfig` (and with it `numa_bind`) and its bool-normalizing `BeforeValidator` are gone. - Validators in `TrainerConfig` and `SFTConfig`: - full offload accepts only AdamW or SignSGD and disables clipping, now reading `offload == "full"`; - the `Muon optimizer does not support FSDP CPU offload` validators are deleted with the mode. - Runtime: - `setup_optimizer(..., offload=...)` replaces `cpu_offload` + `full_offload_config`, and its "both enabled" check is gone. - `FullCPUOffloadOptimizer` drops its unused `offload_config` argument. - `setup_full_cpu_optimizer_offload()` takes no arguments and always binds. - `model.py`: FSDP always uses the default `OffloadPolicy()`; CPU-side model init in `setup_model`/`load_dcp_from_hf` and `_move_buffers_to_cuda` are gone. - Tomls: `glm-4.5-air/swe-budget.toml` and `ci/integration/reverse-text-sft/full-offload-resume.toml` → `offload = "full"`; `glm-5.3/sft/h200/16k-12-node.toml` → `offload = "none"`; `glm-5.3/sft/h200/base.toml` → `offload = "optimizer"`. Also updated `docs/scaling.md`, the `deepseek-v4-flash/sft.toml` comment, and `skills/training/start-run/SKILL.md`, which documented the removed fields. Breaking: the old keys now fail with `extra_forbidden`, and `offload = "fsdp"` fails the `Literal`. There is no shim. Interaction with #3849 (one trainer-common config base): #3849 moves the full-offload and Muon/FSDP validators from `TrainerConfig`/`SFTConfig` onto `BaseTrainerConfig`. This PR edits the full-offload validator's condition and deletes the Muon/FSDP one, so the conflict is textual only: keep #3849's single copy of the full-offload validators with `self.model.offload == "full"` and drop its Muon/FSDP copy. Whichever lands second takes a small rebase. Other open PRs that set or read the old keys need a rebase: - tomls: #3883, #3879, #3766, #3765, #3729, #3629, #3627, #3616, #3544 - code: #3885 (`optim_cpu_offload`), #3739 (`full_offload` validators), #3724, #3719, #3699, #3610, #3487 (`fsdp_cpu_offload` / `enable_gloo`) Test plan - `pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/train/test_cpu_adam.py tests/unit/train/test_state_offload.py -m "not gpu"`: 336 passed, 1 skipped (Qwen3-VL snapshot not cached on this host), 4 deselected (GPU; this host has no GPU). The existing offload config tests are updated to the new field. `test_load_configs` covers the 4 edited tomls. - `cli(TrainerConfig|SFTConfig, ...)`: `--model.offload full|none` and the default (`optimizer`) resolve as expected; `optim_cpu_offload` and `offload = "fsdp"` are rejected. - `import prime_rl.trainer.rl.train, prime_rl.trainer.sft.train` works. - `ruff check` and `ruff format --check` on the changed Python files: clean. - Not run: GPU or multi-node training with `offload = "full"`. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Breaking TOML/API surface touches distributed training memory paths (FSDP offload, optimizer-in-backward, Gloo); behavior should match prior modes when mapped correctly, but mis-migrated configs will fail at parse time. > > **Overview** > **Breaking config change:** `model.fsdp_cpu_offload`, `model.optim_cpu_offload`, and `model.full_offload` are replaced by a single **`model.offload`** setting (`"optimizer"` default, or `"full"`, `"fsdp"`, `"none"`). RL and SFT share the same field; example TOMLs and scaling docs are updated accordingly. > > The mutual-exclusion validator between the old flags is removed because only one mode can be expressed. **`[model.full_offload]` / `numa_bind` is dropped** — `"full"` always runs NUMA binding and CPU optimizer thread setup via parameterless `setup_full_cpu_optimizer_offload()`. Runtime paths (`setup_optimizer`, FSDP CPU offload in `model.py`, Gloo when offload is `"full"` or `"fsdp"`) key off the new enum; `FullCPUOffloadOptimizer` no longer takes an offload config object. > > Default training behavior is unchanged: optimizer-state CPU offload remains the default (`"optimizer"`). Old TOML keys are not aliased and should fail validation. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 35da73a. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: sami jaghouar <sami.jaghouar@gmail.com>
faresobeid
force-pushed
the
refactor/trainer-config-base
branch
from
October 11, 2026 12:59
806a781 to
04d7143
Compare
…nd SFTConfig BaseTrainerConfig holds the fields and validators shared by the RL trainer and SFT. The three 'auto' defaults that differ (moe_router_dtype, freeze_moe_router, weight_decay) are class variables resolved by one validator.
faresobeid
force-pushed
the
refactor/trainer-config-base
branch
from
October 12, 2026 18:35
04d7143 to
6061eba
Compare
…ig-base # Conflicts: # packages/prime-rl-configs/src/prime_rl/configs/sft.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SFTConfigduplicated most ofTrainerConfig. This PR addsBaseTrainerConfiginconfigs/trainer.pywith the shared fields and validators.TrainerConfigandSFTConfigboth subclass it.Shared fields:
model,tokenizer,optim,scheduler,ckpt,log,matmul_precision,max_steps,memory_profiler_path,gc,trace_path,dist_timeout_seconds,heartbeat,control,env_vars.Shared validators:
deepep_disables_grad_clipping,full_optimizer_offload_requires_supported_optimizer,full_optimizer_offload_disables_grad_clipping,vlm_freeze_incompatible_with_lora,dont_do_massive_traces,validate_scheduler_steps,auto_setup_tokenizer.The three
"auto"settings that resolve differently are class variables, resolved by oneresolve_auto_defaultsvalidator (replaces three validators per class):TrainerConfig)SFTConfig)auto_moe_router_dtypefloat32bfloat16auto_freeze_moe_routerTrueFalseauto_weight_decay0.00.01resume,monitors,output_dirandweight_broadcaststay per class: their types or meanings differ.Behavior
No runtime change. Defaults, resolved values, warnings and validation errors are the same.
Visible differences:
max_steps("Maximum number of training steps.") andenv_vars("...for the trainer process(es)...").model_fields,--helpand resolved-config dumps. The set of fields and their values do not change.Proof
mainat the PR's original base: every TOML underconfigs/andexamples/(78 files; there is nok8s/TOML), standalone and composed with each sibling TOML, plus 18 CLI override cases (auto overrides, trace/max_steps, full offload, Muon + FSDP offload, cosine scheduler, DeepEP, LoRA, tokenizer). Each case was parsed with all 7 config classes, withrun.nameuuids pinned. That is 2310 cases, 245 of which parse.model_dump, error messages and warnings are identical.model_json_schema()is identical for all 7 config classes.--help:TrainerConfigandRLConfighave the same lines (order only).SFTConfigdiffers only in the two descriptions above.main(52455b4): everyTrainerConfig/SFTConfigfield has the same default and type as onmain(includingdist_timeout_seconds = 900from feat(trainer): 900 s collective timeout, flight recorder dumps, master-only waits off NCCL #4019).pytest tests/unit/test_configs.py tests/unit/orchestrator -m "not gpu": 336 passed.ruff checkandruff format --checkare clean.Note
Low Risk
Refactor-only config hierarchy with parity testing claimed; no training logic changes, only consolidation and validator ordering.
Overview
Introduces
BaseTrainerConfigintrainer.pyso RLTrainerConfigandSFTConfigshare one definition of model/tokenizer/optim/scheduler, logging, profiling, distributed timeout, and the common cross-cutting validators (DeepEP vs grad clip, full optimizer offload, VLM+LoRA, trace limits, scheduler steps, tokenizer auto-setup).SFTConfignow subclasses the base instead of re-declaring those fields and copy-pasted validators. RL- vs SFT-specific behavior for"auto"settings is expressed as subclassClassVardefaults (auto_moe_router_dtype,auto_freeze_moe_router,auto_weight_decay) resolved in a singleresolve_auto_defaultsvalidator on the base.RL-only and SFT-only pieces stay on the respective subclasses (e.g. RL
loss/rollout_transport; SFTrenderer, deployment, online eval wiring). Intended runtime behavior is unchanged; help text and field/validator ordering may differ slightly.Reviewed by Cursor Bugbot for commit 1b459c1. Bugbot is set up for automated code reviews on this repo. Configure here.