Add Gemma4 GenAI text generation example with benchmarking - #249
Merged
Conversation
New example script for Gemma4 text generation via ORT GenAI: - Builds ONNX model with mobius CLI (--runtime ort-genai) - Runs multi-step generation with streaming output - Reports tokens/sec, time-to-first-token, decode throughput - Supports --device cpu/cuda, --dtype f32/f16/bf16 - Supports --model for different Gemma4 sizes - Supports --save-to for export-only and --model-dir for pre-built Usage: python examples/gemma4_genai.py --device cpu python examples/gemma4_genai.py --device cuda --dtype f16 Signed-off-by: Justin Chu <justinchu@microsoft.com>
Performance Comparison
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
37 tasks
Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
force-pushed
the
gemma4-genai-example
branch
from
May 5, 2026 19:42
54054d9 to
92dfce8
Compare
Member
Author
|
@copilot please fix lint |
Contributor
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
force-pushed
the
gemma4-genai-example
branch
from
May 5, 2026 21:29
36d0f2a to
d01c8a1
Compare
titaiwangms
approved these changes
May 5, 2026
Gemma4 instruction-tuned models require chat-formatted prompts. Raw text prompts produce degenerate repetitive output. Add format_chat_prompt() that wraps user messages in the Gemma chat template (<start_of_turn>user/model format). Verified: with chat template, ORT GenAI output is token-for-token identical to HuggingFace PyTorch (9/9 and 64/64 tokens matched). Signed-off-by: Justin Chu <justinchu@microsoft.com>
Gemma4 e2b/e4b are multimodal — support text + image + audio via GenAI's MultiModalProcessor. When --image or --audio is provided, the script uses the VLM generation path with set_inputs(). - Text-only: uses chat template, appends tokens - Multimodal: uses HF processor for template with media placeholders, loads media via og.Images/og.Audios, processes with MultiModalProcessor - Auto-detects VLM vs text-only build based on --image/--audio flags Signed-off-by: Justin Chu <justinchu@microsoft.com>
Gemma4 tokenizer has add_bos_token=False, so <bos> must be included explicitly in the chat template. Without it, generation produces garbled output because the model expects BOS as the first token. The HF apply_chat_template() includes <bos> automatically, but our manual format_chat_prompt() was missing it. Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
enabled auto-merge (squash)
May 6, 2026 03:54
justinchuby
disabled auto-merge
May 6, 2026 03:55
GenAI's tokenizer reads the chat template from tokenizer_config.json
and applies it correctly (including <bos>). Remove the manual
format_chat_prompt() function and use:
messages = json.dumps([{'role': 'user', 'content': prompt}])
chat_prompt = tokenizer.apply_chat_template(messages)
This is more robust — it uses the model's own template definition
rather than a hardcoded format string that can drift.
Signed-off-by: Justin Chu <justinchu@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
examples/gemma4_genai.py— a streamlined example for Gemma4 text generation via ORT GenAI with performance benchmarking.Features
mobius build --runtime ort-genaito create ONNX model with genai_config.json--device cpuor--device cuda--dtype f32/f16/bf16--model google/gemma-4-e2b-it(default) or any Gemma4 variant--model-dirto skip export,--save-tofor export-onlyUsage
Relation to existing examples
This is a simplified, benchmark-focused version of
gemma4_ort_genai.py. Key differences: