Fix LM head for Gemma-2 - #1420
Merged
Merged
Conversation
Baiju Meswani (baijumeswani)
previously approved these changes
Apr 24, 2025
Justin Chu (justinchuby)
added a commit
to justinchuby/onnxruntime-genai
that referenced
this pull request
Apr 24, 2025
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if args.timings: started_timestamp = time.time() | ||
|
|
||
| prompt = f'{args.chat_template.format(input=text)}' | ||
| prompt = f'{args.chat_template.format(system_prompt=system_prompt, input=text)}' |
Check warning
Code scanning / CodeQL
Unused named argument in formatting call
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI over 1 year ago
To fix the issue, we need to remove the surplus system_prompt argument from the format call on line 107. This involves:
- Verifying that the format string in
args.chat_templatedoes not include a placeholder forsystem_prompt. - Removing the
system_prompt=system_promptargument from theformatcall, leaving only the required arguments.
This change ensures that the code is cleaner and avoids unnecessary arguments in the formatting call.
Suggested changeset
1
examples/python/model-chat.py
| @@ -106,3 +106,3 @@ | ||
|
|
||
| prompt = f'{args.chat_template.format(system_prompt=system_prompt, input=text)}' | ||
| prompt = f'{args.chat_template.format(input=text)}' | ||
| input_tokens = tokenizer.encode(prompt) |
Copilot is powered by AI and may make mistakes. Always verify output.
Baiju Meswani (baijumeswani)
approved these changes
Apr 25, 2025
kunal-vaishnavi
added a commit
that referenced
this pull request
May 9, 2025
### Description This PR fixes accuracy issues with Google's Gemma models by using bfloat16 precision, [always using float32 precision to compute any LayerNorms](https://github.com/huggingface/transformers/blob/fee1190601b5d04ec6d3f7f58fd22788d7f3236d/src/transformers/models/gemma3/modeling_gemma3.py#L141-L146), and casting the output logits to float32 always. ### Motivation and Context This PR has been tested with Gemma-2 and Gemma-3. It is using the bfloat16 changes from [this PR](#1447) as well as the missing final norm changes from [this PR](#1420). --------- Co-authored-by: Nenad Banfic <46795300+nenad1002@users.noreply.github.com> Co-authored-by: Nenad Banfic <nebanfic@microsoft.com>
Ryan Hill (RyanUnderhill)
pushed a commit
that referenced
this pull request
May 12, 2025
### Description This PR fixes accuracy issues with Google's Gemma models by using bfloat16 precision, [always using float32 precision to compute any LayerNorms](https://github.com/huggingface/transformers/blob/fee1190601b5d04ec6d3f7f58fd22788d7f3236d/src/transformers/models/gemma3/modeling_gemma3.py#L141-L146), and casting the output logits to float32 always. ### Motivation and Context This PR has been tested with Gemma-2 and Gemma-3. It is using the bfloat16 changes from [this PR](#1447) as well as the missing final norm changes from [this PR](#1420). --------- Co-authored-by: Nenad Banfic <46795300+nenad1002@users.noreply.github.com> Co-authored-by: Nenad Banfic <nebanfic@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes a bug with the language modeling head (LM head) for Gemma-2.
Motivation and Context
When Gemma-3 was added, Gemma-2 was refactored to work for both Gemma-2 and Gemma-3. During the refactoring process, however, the setting of the
scaleattribute for the LM head of Gemma-2 was not removed. This causes an extra Mul node to be created. It also causes two Mul nodes to have the same name (/lm_head/Mul) as well.