Sync FORCE_FLOAT32 fallback with unsloth-zoo (gemma4, glm4_moe, qwen3_moe) - #6865
Conversation
Keeps the fallback list (used only if the unsloth_zoo import fails) in sync with unsloth_zoo/model_lists.py, which now force-float32s these MoE archs so a float16 request loads bf16 and trains finite instead of NaNing the grad_norm.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates unsloth/models/loader.py to add several new model identifiers (gemma4, glm4_moe, and qwen3_moe) to the list of models that experience float16 NaN gradient norms during backward passes. I have no feedback to provide as there are no review comments.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
@codex review |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request updates unsloth/models/loader.py to add several new model architectures ('gemma4', 'glm4_moe', and 'qwen3_moe') to the list of models that experience float16 NaN gradient norms during backward passes. There are no review comments, and I have no additional feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb371110fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "gemma3,", # Add comma bc gemma3 will match gemma3n | ||
| "gemma3text", # Gemma3TextModel (EmbeddingGemma, standalone text-only Gemma3) | ||
| "gemma3n", | ||
| "gemma4", # Gemma4 (gemma4 / gemma4_text): float16 NaNs grad norms in the backward |
There was a problem hiding this comment.
Adding gemma4 only to the except ImportError fallback does not cover the stale-unsloth_zoo case described in the commit: if the installed zoo package already exports FORCE_FLOAT32 but predates the entries from unsloth-zoo#866, the import at line 116 succeeds and this fallback list is bypassed, so Gemma 4 / GLM-4 MoE / Qwen3-MoE fp16 loads still will not be coerced. Please append/merge these required names into the imported list when they are missing, rather than only defining them in the import-failure path.
Useful? React with 👍 / 👎.
|
The fallback list is now unioned into whatever |
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Summary
Keeps the
FORCE_FLOAT32fallback list inunsloth/models/loader.pyin sync withunsloth_zoo/model_lists.py(unslothai/unsloth-zoo#866), which addsgemma4,glm4_moeandqwen3_moe.Loaded with
dtype=torch.float16, these MoE architectures keep a finite loss but NaN the grad_norm (a backward gradient overflow along the residual stream; bf16 is fully finite). Being inFORCE_FLOAT32makes a float16 request load bf16 and run finite, the same asgpt_oss/qwen3_5/gemma3. The targeted per-module float32 patches forgemma4/qwen3_moelive in unsloth-zoo (#866).This list is only the fallback used if
from unsloth_zoo import FORCE_FLOAT32fails; the authoritative list is in unsloth-zoo. Updating it keeps the two consistent.