extend modelbuilder to build Olmo3, SmolLM3 and other models

[Xavier Dupré (xadupre)]

• it adds unit test to check discrepancies on all supported architectures
• it fixes a few things in existing architectures detected through the unit tests
• it supports transformers>=5 by modifying the final genai config to make it lookl like a transformers<5
• it adds a job to run the created models on cpu with the released version of onnxruntime-genai
• the tests runs in 3 or 4 minutes on two versions of transformers (4.57, 5.5)
• it uploads artifact showing the test results (below an example)

	model_id	experiment	precision	provider	input_type	max_abs_err	avg_abs_discrepancy	dnan	next_token	next_token_id_tch	next_token_id_ort	test	first_diff	total_diff	kind	step	%>0.1	%>0.01
0	arnir0/Tiny-LLM	attention_decode	fp32	cpu	text	1.3411e-07	3.00374e-08	0	OK	324	324	test_llama_attention_decode_discrepancies					0	0
1	arnir0/Tiny-LLM	attention_prefill	fp32	cpu	text	3.27826e-07	4.81524e-08	0	OK	155	155	test_llama_attention_prefill_discrepancies					0	0
2	THUDM/chatglm3-6b	generate	fp16	cpu	text							test_generation_chatglm_fp16_cpu		0	fast
3	THUDM/chatglm3-6b	generate	fp32	cpu	text							test_generation_chatglm_fp32_cpu		0	fast
4	THUDM/chatglm3-6b	forward	fp16	cpu	text	0.000906795	0.000173806	0	OK	905	905	test_discrepancies_chatglm_fp16_cpu			fast	prefill	0	0
5	THUDM/chatglm3-6b	forward	fp16	cpu	text	0.000891805	0.00018833	0	OK	1322	1322	test_discrepancies_chatglm_fp16_cpu			fast	decode	0	0
6	THUDM/chatglm3-6b	forward	fp32	cpu	text	8.34465e-07	1.66793e-07	0	OK	905	905	test_discrepancies_chatglm_fp32_cpu			fast	prefill	0	0
7	THUDM/chatglm3-6b	forward	fp32	cpu	text	7.7486e-07	1.58349e-07	0	OK	1322	1322	test_discrepancies_chatglm_fp32_cpu			fast	decode	0	0

[Copilot]

Pull request overview

This PR expands the Python modelbuilder to cover additional HF architectures (e.g., OLMo3, SmolLM3, Qwen3/Qwen3.5, Nemotron-H, Gemma3) and adds a comprehensive “fast” test suite that builds tiny random-weight models offline and validates ONNX vs PyTorch (and/or validates export output). It also adds a compatibility shim to post-process genai_config.json for transformers>=5 generation defaults.

Changes:

• Add many new offline “fast” unit tests that export minimal random-weight models and compare ORT outputs against PyTorch (prefill/decode and some greedy generation).
• Extend/fix multiple builder implementations (OLMo*, SmolLM, Nemotron/Nemotron-H, Gemma3, Ernie, GPT-OSS, etc.) to match architecture quirks and transformers v4/v5 config differences.
• Add fix_genai_config utility + tests, relax pytest --test_models option requirement, and introduce a GitHub Actions workflow for fast tests.

Reviewed changes

Copilot reviewed 52 out of 53 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/python/models/fast/test_random_whisper.py	Adds offline Whisper encoder/decoder export + ORT vs PyTorch discrepancy checks.
test/python/models/fast/test_random_smollm3.py	Adds SmolLM3 random-weight discrepancy tests and greedy generation checks.
test/python/models/fast/test_random_qwen3_vl.py	Adds Qwen3-VL text-path discrepancy tests (transformers>=5 guarded).
test/python/models/fast/test_random_qwen3_5.py	Adds Qwen3.5 full-attention execution test + hybrid build validation (transformers>=5).
test/python/models/fast/test_random_qwen3_0_6b.py	Adds Qwen3 random-weight discrepancy and greedy generation tests.
test/python/models/fast/test_random_qwen2_5_vl.py	Adds Qwen2.5-VL text-path discrepancy tests (transformers>=5 guarded).
test/python/models/fast/test_random_phi4mm.py	Adds Phi4MM synthetic-weight build + ORT vs PyTorch comparisons (uses PEFT LoRA).
test/python/models/fast/test_random_phi3.py	Adds Phi-3 random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_random_phi.py	Adds Phi-2 random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_random_olmo3.py	Adds OLMo3 random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_random_olmo2.py	Adds OLMo2 random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_random_olmo.py	Adds OLMo (v1) random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_random_nemotron.py	Adds Nemotron random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_random_nemotron_h.py	Adds Nemotron-H random-weight discrepancy + greedy generation tests (transformers>=5 guarded).
test/python/models/fast/test_random_granite.py	Adds Granite random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_random_gemma3.py	Adds Gemma3 random-weight discrepancy + greedy generation tests (transformers>=5 guarded).
test/python/models/fast/test_random_gemma.py	Adds Gemma random-weight discrepancy + greedy generation tests.
test/python/models/fast/test_onnx_generate.py	Adds unit tests for ONNX greedy generation helper (no-cache and KV-cache paths).
test/python/models/fast/test_llama_attention_discrepancies.py	Adds attention-only Llama export + numeric discrepancy tests.
test/python/models/fast/test_fix_genai_config.py	Adds tests validating fix_genai_config behavior for null generation defaults.
test/python/models/fast/test_check_extra_options.py	Adds tests for extra_options parsing/validation behavior.
test/python/conftest.py	Makes --test_models optional for pytest runs.
src/python/py/models/genai_config_utils.py	Introduces fix_genai_config to fill null search defaults for transformers>=5.
src/python/py/models/builders/smollm.py	Fixes per-layer RoPE enable/disable handling for SmolLM.
src/python/py/models/builders/phi.py	Minor formatting tweak around position_ids reformatting naming.
src/python/py/models/builders/olmo.py	Adds OLMo2/OLMo3 builders + OLMo v1 LayerNorm handling + OLMo2/3 q/k norm + post-norm residual flow.
src/python/py/models/builders/nemotron.py	Adds Nemotron-H builder and fixes Nemotron epsilon handling.
src/python/py/models/builders/internlm.py	Formatting/quoting cleanup (no apparent logic change in diff shown).
src/python/py/models/builders/gptoss.py	Fixes RoPE cache inv_freq math and tightens some style/robustness in bias combining.
src/python/py/models/builders/gemma.py	Improves Gemma3 RoPE parameter handling (v4/v5) and adds conditional-generation load_weights path.
src/python/py/models/builders/ernie.py	Patches rope_theta extraction from rope_parameters for Ernie4.5 configs.
src/python/py/models/builders/init.py	Re-exports fix_genai_config and narrows __all__.
.gitignore	Ignores dump_models/ and stats/ produced by fast tests.
.github/workflows/modelbuilder_fast_tests.yml	Adds a GitHub Actions workflow to run fast modelbuilder tests across transformers versions.

[Copilot]

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated 15 comments.

[Copilot]

Copilot resolve the merge conflicts in this pull request

Done in 8790306. Resolved the single conflict in test/python/requirements.txt — kept the HEAD version (no <5.0.0 constraint on transformers, plus onnxscript and huggingface_hub[cli,hf_xet]).

[Xavier Dupré (xadupre)]

Copilot resolve the merge conflicts in this pull request

[Copilot]

Copilot resolve the merge conflicts in this pull request

Done in f977a04. Resolved the single conflict in src/python/py/models/builders/base.py — took main's version ("sequence_length" + self.q_size) from PR #2137 (packed QKV MatMul under QK-Norm fix).

[Xavier Dupré (xadupre)]

Xavier Dupré (@xadupre), MacOS failure seems to come from your change. Can you fix the issue?

I assume the MacOS pipeline is failing because of :

##[warning].github/workflows/modelbuilder_fast_tests.yml - Found "extra-index-url". (https://aka.ms/cfs/pypi)"

I removed it. I used it to install onnxruntime nightly. I hope that's the issue because I did not see many successful MacOS jobs lately.

[Xavier Dupré (xadupre)]

Copilot resolve the merge conflicts in this pull request

[Copilot]

Copilot resolve the merge conflicts in this pull request

Done in 2b694c6. Resolved the single conflict in src/python/py/models/builder.py — kept the HEAD's relative/lazy-import style and added a lazy import block for HunyuanDenseV1Model (from HunYuanDenseV1ForCausalLM architecture added in main).

[Xavier Dupré (xadupre)]

Copilot resolve the merge conflicts in this pull request

[Copilot]

Copilot resolve the merge conflicts in this pull request

Done in d7b7be8. Resolved conflicts in builder.py and builders/__init__.py — added lazy import blocks for VideoChatFlashQwenModel (VideoChatFlashQwenForCausalLM) and Qwen35MoeTextModel (Qwen3_5MoeForConditionalGeneration), and updated __all__ to export both.

[Xavier Dupré (xadupre)]

Copilot explain why CAPITests.EndToEndPhiEOSPAD is failing

[Copilot]

Copilot explain why CAPITests.EndToEndPhiEOSPAD is failing

The CAPITests.EndToEndPhiEOSPAD test failure is not caused by this PR's changes.

What the test does: It runs Phi2 inference on WebGPU with input "This is a test.<|endoftext|>" (including an EOS token) and compares the 40 generated token IDs against hardcoded expected values using memcmp.

The actual failure (from c_api_tests.cpp:576):

Value of: 0 == std::memcmp(expected_output_start, sequence_data, sequence_length * sizeof(int32_t))
  Actual: false
  Expected: true

The generated text was "This is a test.\n\n\n challenginglyiniaourtNGsloopwichtowns..." instead of what the hardcoded expected tokens produce.

Why this PR is not the cause:

1. This PR only changes Python model-builder files, test infrastructure, and CI workflows — no C++ inference code was touched.
2. The test/c_api_tests.cpp file in this branch is identical to main (same expected token vector).
3. The same test passes on the yenshiw/trt-rtx-qwen35-35b-moe branch's WebGPU build (run 2026-05-28 on runner 3eee423fc000000: 49 passed, 0 failed), while this branch failed on runner c62ca287c000000.

Root cause: The hardcoded expected token IDs were recorded on a specific GPU/driver configuration. On the WebGPU execution provider, floating-point computation can differ between GPU hardware or driver versions in the self-hosted runner pool, producing different token IDs from a greedy decode even with identical code. The workflow was re-run 3 times (run_attempt: 3) and failed consistently on the same runner machine — confirming this is machine-specific, not a code regression.

This is a pre-existing flakiness in the WebGPU test that predates this PR.

[Baiju Meswani (baijumeswani)]

Is this PR still relevant? Could we update it?

[kunal-vaishnavi]

Is this PR still relevant? Could we update it?

We can leave it open for now. I will update it.