Repository navigation
chore(models)!: remove GPT-OSS support - #3866
Closed
faresobeid wants to merge 1 commit into
Closed
faresobeid wants to merge 1 commit into
faresobeid wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes GPT-OSS (
gpt_oss) support. No back-compat shims: a GPT-OSS checkpoint now fails at trainer setup like any other unsupported architecture.Removed
trainer/models/gpt_oss/(model, its own FA4 attention with learned sinks, and its own ring/Ulysses CP paths),tests/unit/train/models/test_gpt_oss.py.inference/vllm/gpt_oss_weight_loading.pyand its call ininference/patches.py.docs/advanced.mdrows and the GPT-OSS FA4/sinks note.openai/gpt-osstool-parser auto-detection, plus its parser test rows.Shared code that only existed for GPT-OSS
layers/attn.py,layers/ulysses_attn.py: drop the GPT-OSS CP substitution hooks. Also drop thelearnable_sinkandsoftmax_scaleargs ofulysses_flash_attn_varlen_func, because GPT-OSS was their only caller. DeepSeek-V4's sinks live in its own kernels and are untouched.layers/moe.py: remove expert projection biases (GroupedExperts(bias=...)), the routergate_bias, and thetopk_softmaxscore function.layers/expert_compute.py: removebroadcast_expert_biasand the bias adds in the grouped-GEMM forward.layers/activations.py: removeclamped_swiglu(DeepSeek-V4 keeps its ownClampedSwigluwith a configurable limit).layers/sonic_moe.py: remove the bias-free check, since experts have no biases now.trainer/lora.py+layers/lora/multi_moe.py: remove themodel_type == "gpt_oss"branch andMultiLoRAGptOssGroupedExperts.test_moe.py: drop the clamped-swiglu/bias cases and the bias terms in the expert reference.Not touched:
DeepseekV4Router.forwardstill has agate.biasbranch copied from the base router. It is dead now. #3854 deletes that copy.Checks
ruff check .andruff format --check .: clean.pytest -m "not gpu" tests/unit/orchestrator tests/unit/test_configs.py tests/unit/test_parsers.py tests/unit/train tests/unit/inference: all pass excepttest_cpu_adam.py::test_native_cpu_adamw_matches_fused_torch_and_preserves_gradients, which fails the same way onmainon this CPU box.seq_lens, SDPA stand-in for the flash kernel). Logits, all parameter grads and state-dict keys are bitwise identical tomain. DeepSeek-V4 needs GPU kernels, so it was not run.docs/algorithms.md, which describes the externalrendererspackage).Conflicts with open PRs
gpt_oss/*,test_gpt_oss.py,lora.py,attn.py,models/__init__.py. Whichever lands second drops the gpt_oss hunks.MultiLoRAGptOssGroupedExperts→LoRAGptOssGroupedExperts. If this lands first, drop that class and branch there.init_buffers_post_meta): editsgpt_oss/modeling_gpt_oss.py. Drop the hunk.layers/moe.py/test_moe.pynear the removed bias/topk_softmaxlines. Small textual conflicts.attn.py/ulysses_attn.py.inference/patches.py, only the two removed GPT-OSS lines at the top ofapply_shared_vllm_patches.