feat(deepseek-v4): enable bf16 RL, bump vLLM to 0.29.0 and prime-envs - #3543
Draft
garrett361 wants to merge 9 commits into
Draft
garrett361 wants to merge 9 commits into
garrett361 wants to merge 9 commits into
Conversation
garrett361
force-pushed
the
feat/ds-v4-bf16-rl
branch
from
September 17, 2026 20:34
3c937fb to
7f14af7
Compare
garrett361
force-pushed
the
feat/ds-v4-bf16-rl
branch
from
September 21, 2026 20:42
11c6937 to
bd3106e
Compare
garrett361
force-pushed
the
feat/ds-v4-bf16-rl
branch
from
September 22, 2026 13:11
bd3106e to
0c319cf
Compare
…okenize-params patch The patched _text_len_check and get_encode_kwargs are full-body copies of upstream, and 0.29 added truncation-side overrides to both originals: encode must skip tokenizer-level truncation (its default side may differ from the request) and _text_len_check pre-trims characters from the requested side. Restore both branches, keyed on max_total_tokens as the copies already are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ight loaders DeepseekV4Model.load_weights writes the attention sinks with a bare params_dict[name][:n].copy_(narrow_weight) instead of going through param.weight_loader. Layerwise reload moves a layer's tensors to meta and wraps each loader to buffer the incoming tensor, so that copy lands in a meta tensor and is discarded silently: load_numel stays 0, finalize restores the boot value with only a warning, and the loader still does loaded_params.add(name), so a named_parameters() - loaded_params diff cannot see the loss either. Attention sinks are trainable, so every weight update keeps serving the sinks the server booted with. Padding this rank's heads back up with -inf, the parameter's own init value, makes it an ordinary full-parameter load through the existing loader contract. Still required on vLLM 0.29.0 (model.py:1546). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Writes a bf16 HF-layout export of a prime-rl conversion cache, restoring the published key names, splitting the fused expert tensors back per expert, and emitting a config.json with the quantization fields removed. DeepSeek V4 Flash cannot otherwise be served against a bf16 weight broadcast: the published repo's config.json carries quantization_config and expert_dtype = "fp4", so vLLM allocates fp8 linears and MXFP4 experts that bf16 weights cannot fill. The routed experts then fail loudly in _load_w13 while the fp8 linears fail silently, since a bf16 tensor matches the fp8 parameter's shape and copy_ casts it with no scale applied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rl.toml trains deepseek-ai/DeepSeek-V4-Flash-0731 at full depth over a bf16 NCCL weight broadcast, 4 train nodes plus 1 inference node, with router replay on. As written it is a fidelity measurement rather than a training run: lr = 0 holds the policy fixed so the trainer-vs-inference mismatch KL is measurable without policy drift. inference_bf16.toml is the serving half on its own, a standalone pre-flight for whether this cluster's vLLM allocates bf16 parameters for this checkpoint. Both serve the bf16 export built by tools/convert_prime_to_hf.py, not the published repo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bf16 export is now published as PrimeIntellect/DeepSeek-V4-Flash-0731-bf16, so the configs name a hub id instead of an absolute path under /home/hf-cache/exports. That path existed on exactly one cluster, which made the shipped configs unrunnable anywhere else. Publishing the artifact also retires tools/convert_prime_to_hf.py from this PR. It was included so others could rebuild the export, and a published checkpoint needs no rebuild. It remains in git as ebadd7589 on exp/ds-v4-router-replay. The trainer keeps loading deepseek-ai/DeepSeek-V4-Flash-0731 and dequantizing it through its own conversion cache, so only the serving side moves. validate_shared_model_name requires the orchestrator and vLLM model names to be identical, so those two move together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
propagate_shared_fields fans a top-level [model] name out to trainer, orchestrator and inference, so the config names the bf16 repo once instead of three times, matching configs/advanced/minimax-m2.5/swe.toml. The trainer previously loaded the published fp8 checkpoint and dequantized it on the way in. That yields the same weights by construction, since the bf16 export was written from the trainer's own conversion cache and the index's metadata.total_size is byte-identical to it, but naming a single source removes a link from the chain the mismatch-KL evidence rests on. This costs disk rather than saving it: trainer/model.py:663 builds a prime conversion cache for any HF-format snapshot, and both repos are HF-format, so bf16 means a 530 GB download plus a 530 GB cache where fp8 meant 167 GB plus the same cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restores the math environment's default solver network access. prime-envs #780 had set `network_allow = []` on `MathData`, which makes verifiers treat every math task as requiring a framework-aware network policy and refuse the `subprocess` runtime that every config in this repo uses. prime-envs #795 reverts that default-deny, so the KL merge-bar measurement on `math` no longer needs a wrapper taskset to restore the wildcard. The bump spans ten commits, so it also brings fair-use solver prompts to the SWE and terminal training tasksets, moves MCP Atlas and Enterprise Ops Gym into VMs, and adds OfficeQA Pro V2, along with a large `registry.json` regeneration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
garrett361
force-pushed
the
feat/ds-v4-bf16-rl
branch
from
September 22, 2026 13:34
0c319cf to
3e6d699
Compare
vllm-project/vllm#54955 is a "[Preview][DO NOT MERGE]" branch, so it can never be the event that lets us delete this patch. Cite #57798 instead, which fixes the DeepSeek V4 target models on top of the shared padded-sink helper in #57797. Verified against upstream main d90f0eade5: the bare params_dict[name][:n].copy_() is still there, so the patch is still needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DeepSeek V4 Flash passes the KL mismatch merge bar for bf16 RL. The blocker was a vLLM weight-reload bug that left captured CUDA graphs reading a freed pointer, fixed upstream in vllm#52626 and picked up here by pinning vLLM 0.29.0.
mathenvironment,batch_size = 64, 20 steps,lr = 0, full depth, 4 trainer nodes + 1 inference node:All 40 entries under the 0.015 bar: max 0.00040 with replay on, 0.00110 with replay off.
reverse-text
The same model and topology on
reverse-textatbatch_size = 32, over 20 steps:The
vLLM 0.28row is a 2026-09-14 measurement carried forward. The twothis PRrows werere-measured on the current branch, so rerunning the baseline would compare against a different
mainthan the one it was originally a baseline for.mathtable was measured before this branch bumped prime-envs. At the old pin1f1e050ab,environments/math/setnetwork_allow = [], which makes verifiers refuse thesubprocessruntime every config in this repo uses, so the run needed a scratch wrapper taskset restoring the wildcard. The bump toa6fc0c66fpicks up prime-envs Simple implementation of different loss scaling #795, which reverts that default-deny, so the environment now runs unwrapped. The other two measurement-only settings,constant_trainer_batch_size = falseand an inert length-penalty tiebreaker so frozen-policy groups ship, are still not committed.config.jsoncarriesquantization_configandexpert_dtype = "fp4". These configs usePrimeIntellect/DeepSeek-V4-Flash-0731-bf16instead, currently private.