Fix accuracy issues with Gemma models - #1448
Conversation
The Gemma3 model must compute layer norms and residual connections in FP32 to ensure accurate logits. Additionally, all logits are now explicitly cast to FP32. This PR removes the incorrect code paths and refactors the builder script so that it cleanly supports both Gemma and non-Gemma models.
| self.output_names = [name.replace("logits", "hidden_states") for name in self.output_names] | ||
| elif self.include_hidden_states: | ||
| self.output_names = ["hidden_states"] + self.output_names | ||
| self.make_outputs_init() |
Check warning
Code scanning / CodeQL
`__init__` method calls overridden method
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 year ago
To fix the issue, we should avoid calling a potentially overridden method (make_outputs_init) in the __init__ method of the Model class. Instead, we can:
- Rename the
make_outputs_initmethod in theModelclass to a private method (e.g.,_make_outputs_init) to ensure it is not overridden by subclasses. - Call this renamed method (
_make_outputs_init) in theModelclass's__init__method. - Subclasses like
Gemma2Modelcan still define their ownmake_outputs_initmethod if needed, but it will not interfere with theModelclass's initialization process.
This approach ensures that the Model class's initialization is self-contained and does not depend on subclass behavior.
| @@ -115,3 +115,3 @@ | ||
| } | ||
| self.make_outputs_init() | ||
| self._make_outputs_init() | ||
|
|
||
| @@ -323,3 +323,3 @@ | ||
|
|
||
| def make_outputs_init(self): | ||
| def _make_outputs_init(self): | ||
| self.exclude_lm_head = self.extra_options.get("exclude_lm_head", False) |
| # Name = name of original LayerNorm op as if the cast nodes did not exist | ||
| # Inputs = inputs into the original LayerNorm op as if the cast nodes did not exist | ||
| # Outputs = outputs from the original LayerNorm op as if the cast nodes did not exist | ||
| def get_shape_of_value_info(target_name): |
Check notice
Code scanning / CodeQL
Explicit returns mixed with implicit (fall through) returns
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 year ago
To fix the issue, we will add an explicit return None statement at the end of the get_shape_of_value_info function. This ensures that the function always returns a value explicitly, making its behavior consistent and easier to understand. The change will not alter the existing functionality but will make the function's intent clearer.
| @@ -1127,3 +1127,3 @@ | ||
| return shape | ||
|
|
||
| return None | ||
| # Save original inputs and outputs |
### Description This PR fixes accuracy issues with Google's Gemma models by using bfloat16 precision, [always using float32 precision to compute any LayerNorms](https://github.com/huggingface/transformers/blob/fee1190601b5d04ec6d3f7f58fd22788d7f3236d/src/transformers/models/gemma3/modeling_gemma3.py#L141-L146), and casting the output logits to float32 always. ### Motivation and Context This PR has been tested with Gemma-2 and Gemma-3. It is using the bfloat16 changes from [this PR](#1447) as well as the missing final norm changes from [this PR](#1420). --------- Co-authored-by: Nenad Banfic <46795300+nenad1002@users.noreply.github.com> Co-authored-by: Nenad Banfic <nebanfic@microsoft.com>
Address previous PR review comments from #1470 (#1473) Address QNN specific regressions (#1470) Fix array eos_token_id handling (#1463) Constrained decoding integration (#1381) Remove BF16 CPU from valid GQA configuration (#1469) Avoid adding providers if not requested (#1464) Persist provider options across ClearProviders, AppendProvider where possible (#1454) Fix accuracy issues with Gemma models (#1448) Add bfloat16 support in model builder (#1447) Add final norm for LoRA models (#1446) Update version to 0.8.0-rc3 --------- Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com> Co-authored-by: Nenad Banfic <46795300+nenad1002@users.noreply.github.com> Co-authored-by: Nenad Banfic <nebanfic@microsoft.com> Co-authored-by: Baiju Meswani <bmeswani@microsoft.com> Co-authored-by: Abhishek Jindal <abjindal@microsoft.com> Co-authored-by: Ying Xiong <yingxiong@microsoft.com> Co-authored-by: Michał Moskal <michal@moskal.me> Co-authored-by: Kunal Vaishnavi <kvaishnavi@microsoft.com>
Description
This PR fixes accuracy issues with Google's Gemma models by using bfloat16 precision, always using float32 precision to compute any LayerNorms, and casting the output logits to float32 always.
Motivation and Context
This PR has been tested with Gemma-2 and Gemma-3. It is using the bfloat16 changes from this PR as well as the missing final norm changes from this PR.