Skip to content

[qwen] Switch CI to transformers==5.12.1 - #381

Merged
xadupre merged 5 commits into
mainfrom
copilot/switch-ci-to-use-transformers-5-12-1
Jun 17, 2026
Merged

[qwen] Switch CI to transformers==5.12.1#381
xadupre merged 5 commits into
mainfrom
copilot/switch-ci-to-use-transformers-5-12-1

Conversation

Copilot AI commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Bumps the pinned transformers version used across CI workflows from 5.9 to 5.12.1.

Changes

  • .github/workflows/fast_tests.yml — matrix transformers-version["4.57", "5.6", "5.12.1"]
  • .github/workflows/trained_tiny_llm_tests.yml — matrix transformers-version["5.6", "5.12.1"]
  • .github/workflows/fast_tests_ort_nightly.yml — install step pin → transformers==5.12.1

Older compatibility versions in the matrices are retained.

Qwen2.5-Omni position_ids fix

While validating the CI bump, the two Qwen2.5-Omni genai tests failed with RuntimeError: Invalid rank for input: position_ids Got: 2 Expected: 3. This is a pre-existing model/pipeline mismatch (it also fails on transformers 5.9), not a regression from the version bump: the Omni multimodal pipeline is driven by the ORT-GenAI phi3v loader, which feeds 2D position_ids [B, S], while the Omni text decoder (inherited from Qwen25VLTextModel) was always built expecting 3D mRoPE position_ids [3, B, S].

  • modelbuilder/builders/qwen.py — Added an expand_position_ids flag to Qwen25VLTextModel (default False, so genuine Qwen2.5-VL / Qwen3-VL keep 3D position_ids). When enabled, the graph accepts 2D position_ids and expands them to the 3D mRoPE layout internally (via the make_mrope_position_ids() helper). Qwen25OmniThinkerModel sets this flag to True, matching the existing Qwen35TextModel phi3v approach.
  • modelbuilder/ext_test_case.py — Added an onnx_position_ids_2d option to the shared VL discrepancy helper so the ONNX session receives 2D position_ids while PyTorch still receives 3D for parity.
  • tests/fast/test_random_qwen2_5_omni.py — Updated the Omni tests to feed 2D position_ids.

pytest tests/fast now passes (305 passed), including both previously failing ..._genai tests; genuine Qwen2.5-VL / Qwen3-VL tests are unchanged.

This branch was merged with main, which had independently landed the same Qwen2.5-Omni fix (#382) via the make_mrope_position_ids() helper; the conflicts were resolved by adopting that upstream implementation.

CI check-run status fix

fast_tests.yml and trained_tiny_llm_tests.yml previously declared their matrix jobs with the identical name template ci (…, transformers-X). Because both matrices include transformers-5.6 and transformers-5.12.1, two pairs of check runs were emitted with colliding names, so GitHub deduplicated them and one job appeared to not report its status.

  • .github/workflows/trained_tiny_llm_tests.yml — Renamed the job to ci-trained (…) so its check-run names are unique across workflows, ensuring every job reports a distinct status.

Copilot AI changed the title [WIP] Switch CI to use transformers version 5.12.1 Switch CI to transformers==5.12.1 Jun 16, 2026
Copilot AI requested a review from xadupre June 16, 2026 15:56
@xadupre
xadupre marked this pull request as ready for review June 16, 2026 17:02
@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown

📂 Impacted Files

3 file(s) changed in this pull request:

🔵 .github/workflows/fast_tests.yml (+1 / -1)
🔵 .github/workflows/fast_tests_ort_nightly.yml (+1 / -1)
🔵 .github/workflows/trained_tiny_llm_tests.yml (+2 / -2)

@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.88%. Comparing base (571940e) to head (f38341c).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #381   +/-   ##
=======================================
  Coverage   78.88%   78.88%           
=======================================
  Files          27       27           
  Lines        8113     8113           
=======================================
  Hits         6400     6400           
  Misses       1713     1713           
Flag Coverage Δ
fast-tests 78.88% <ø> (ø)
fast-tests-ort-nightly 78.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@xadupre

xadupre commented Jun 16, 2026

Copy link
Copy Markdown
Owner

@copilot FAILED tests/fast/test_random_qwen2_5_omni.py::TestRandomQwen25OmniVision::test_qwen25omni_conditional_generation_fp16_cpu_genai - RuntimeError: Invalid rank for input: position_ids Got: 2 Expected: 3 Please fix either the inputs/outputs or the model.
FAILED tests/fast/test_random_qwen2_5_omni.py::TestRandomQwen25OmniVision::test_qwen25omni_conditional_generation_fp32_cpu_genai - RuntimeError: Invalid rank for input: position_ids Got: 2 Expected: 3 Please fix either the inputs/outputs or the model.
= 2 failed, 303 passed, 145 skipped, 3 warnings, 204 subtests passed in 302.85s (0:05:02) =

Copilot AI commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

@copilot FAILED tests/fast/test_random_qwen2_5_omni.py::TestRandomQwen25OmniVision::test_qwen25omni_conditional_generation_fp16_cpu_genai - RuntimeError: Invalid rank for input: position_ids Got: 2 Ex...

Fixed in the latest commit. Root cause: the Qwen2.5-Omni multimodal pipeline is driven by the ORT-GenAI phi3v loader, which feeds standard 2D position_ids [B, S], but the text decoder (shared with Qwen2.5-VL) was always built expecting 3D mRoPE position_ids [3, B, S] — hence Got: 2 Expected: 3. (This failed with both transformers 5.9 and 5.12.1.)

The Omni text decoder now accepts 2D position_ids and expands them to the 3D mRoPE layout internally (the same approach Qwen35TextModel uses for its phi3v pipeline). The genuine Qwen2.5-VL / Qwen3-VL models are unchanged (still 3D). Both ..._genai tests and the existing parity tests now pass (pytest tests/fast: 305 passed).

Copilot AI requested a review from xadupre June 16, 2026 18:09
@github-actions github-actions Bot changed the title Switch CI to transformers==5.12.1 [qwen] Switch CI to transformers==5.12.1 Jun 16, 2026
@xadupre

xadupre commented Jun 17, 2026

Copy link
Copy Markdown
Owner

@copilot resolve the merge conflicts in this pull request

…use-transformers-5-12-1

# Conflicts:
#	modelbuilder/builders/qwen.py
#	modelbuilder/ext_test_case.py
#	tests/fast/test_random_qwen2_5_omni.py

Copilot AI commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Merge conflicts resolved in fd4fe15. Conflicts were all in the Qwen2.5-Omni position_ids area where main (#382) had landed the same fix via a cleaner make_mrope_position_ids() helper — I adopted the upstream version in qwen.py, ext_test_case.py, and the Omni test. The CI transformers==5.12.1 bump in the workflow files is preserved. black --check and ruff check pass on the resolved files.

Copilot AI requested a review from xadupre June 17, 2026 07:49
@xadupre

xadupre commented Jun 17, 2026

Copy link
Copy Markdown
Owner

@copilot investigate why one job is not reporting its status

Copilot AI commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@copilot investigate why one job is not reporting its status

Root cause found: fast_tests.yml and trained_tiny_llm_tests.yml used the identical job-name template ci (…, transformers-X). Both matrices include 5.6 and 5.12.1, so two pairs of check runs were emitted with the same name (ci (ubuntu-latest, py3.13, torch-2.12.0, transformers-5.6) and …transformers-5.12.1). GitHub keys the PR checks UI / required-status contexts on the check name, so colliding checks get deduplicated and one job appears to not report its status.

Fixed by renaming the trained-tiny job to ci-trained (…) so its names are unique. Now each workflow produces distinct check contexts.

@xadupre
xadupre merged commit 65b1a42 into main Jun 17, 2026
11 checks passed
@xadupre
xadupre deleted the copilot/switch-ci-to-use-transformers-5-12-1 branch June 17, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants