Add support for Qwen3.5 MoE#1109

Merged

Mecoli1219 merged 7 commits intolinkedin:mainfrom

michaelroyzen:add-qwen3_5_moe

Mar 2, 2026

Commits on Mar 2, 2026

Add support for Qwen3.5 MoE

Michael Royzen
authored and
michaelroyzen
committed
Both Qwen3.5 MoE and Qwen3-Next should use Gemma-style RMSNorm instead of Llama-style

Michael Royzen
authored and
michaelroyzen
committed
Convergence test fixes

Michael Royzen
authored and
michaelroyzen
committed
Fix test imports

Ubuntu
authored and
michaelroyzen
committed
Add shift_labels to loss_function calls
michaelroyzen
committed
Match fp32 skip behavior for Qwen3.5 MoE (as with Qwen3-Next) and add bf16 test matching Qwen3 MoE tolerances

Ubuntu
authored and
michaelroyzen
committed
Rebase and lint
michaelroyzen
committed