Skip to content

chore(models)!: remove GPT-OSS support - #3866

Closed
faresobeid wants to merge 1 commit into
mainfrom
remove-gpt-oss-support
Closed

faresobeid wants to merge 1 commit into
mainfrom
remove-gpt-oss-support

Conversation

@faresobeid

Copy link
Copy Markdown
Contributor

Removes GPT-OSS (gpt_oss) support. No back-compat shims: a GPT-OSS checkpoint now fails at trainer setup like any other unsupported architecture.

Removed

  • trainer/models/gpt_oss/ (model, its own FA4 attention with learned sinks, and its own ring/Ulysses CP paths), tests/unit/train/models/test_gpt_oss.py.
  • inference/vllm/gpt_oss_weight_loading.py and its call in inference/patches.py.
  • Registry entries, README / docs/advanced.md rows and the GPT-OSS FA4/sinks note.
  • openai/gpt-oss tool-parser auto-detection, plus its parser test rows.

Shared code that only existed for GPT-OSS

  • layers/attn.py, layers/ulysses_attn.py: drop the GPT-OSS CP substitution hooks. Also drop the learnable_sink and softmax_scale args of ulysses_flash_attn_varlen_func, because GPT-OSS was their only caller. DeepSeek-V4's sinks live in its own kernels and are untouched.
  • layers/moe.py: remove expert projection biases (GroupedExperts(bias=...)), the router gate_bias, and the topk_softmax score function.
  • layers/expert_compute.py: remove broadcast_expert_bias and the bias adds in the grouped-GEMM forward.
  • layers/activations.py: remove clamped_swiglu (DeepSeek-V4 keeps its own ClampedSwiglu with a configurable limit).
  • layers/sonic_moe.py: remove the bias-free check, since experts have no biases now.
  • trainer/lora.py + layers/lora/multi_moe.py: remove the model_type == "gpt_oss" branch and MultiLoRAGptOssGroupedExperts.
  • test_moe.py: drop the clamped-swiglu/bias cases and the bias terms in the expert reference.

Not touched: DeepseekV4Router.forward still has a gate.bias branch copied from the base router. It is dead now. #3854 deletes that copy.

Checks

  • ruff check . and ruff format --check .: clean.
  • pytest -m "not gpu" tests/unit/orchestrator tests/unit/test_configs.py tests/unit/test_parsers.py tests/unit/train tests/unit/inference: all pass except test_cpu_adam.py::test_native_cpu_adamw_matches_fused_torch_and_preserves_gradients, which fails the same way on main on this CPU box.
  • CPU forward + backward check for qwen3_moe, glm4_moe and laguna (tiny configs, fixed seeds, packed seq_lens, SDPA stand-in for the flash kernel). Logits, all parameter grads and state-dict keys are bitwise identical to main. DeepSeek-V4 needs GPU kernels, so it was not run.
  • grep across src/tests/configs/examples/docs/skills/scripts/.github: no references left (except the renderer list in docs/algorithms.md, which describes the external renderers package).
  • A checkpoint of this family now fails at trainer setup with the usual "has no PrimeRL model implementation" error. No shim.

Conflicts with open PRs

@samsja samsja closed this Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants