Use ORT nightly in CI by justinchuby · Pull Request #287 · onnxruntime/mobius

justinchuby · 2026-05-07T23:30:24Z

Also add torchvision and torchaudio in test.

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Removed several dependencies from the CI requirements file. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

github-actions · 2026-05-07T23:33:45Z

Performance Comparison

Comparing 970cd41 → 53fb00a

Model	Metric	Baseline	Current	Delta
bert (feature-extraction)	model_size_bytes	359 KB	359 KB	+0.0%	⚪
bert (feature-extraction)	num_nodes	60	60	+0.0%	⚪
falcon	model_size_bytes	364 KB	364 KB	+0.0%	⚪
falcon	num_nodes	66	66	+0.0%	⚪
gemma2	model_size_bytes	428 KB	428 KB	+0.0%	⚪
gemma2	num_nodes	107	107	+0.0%	⚪
gpt2	model_size_bytes	388 KB	388 KB	+0.0%	⚪
gpt2	num_nodes	53	53	+0.0%	⚪
llama	model_size_bytes	425 KB	425 KB	+0.0%	⚪
llama	num_nodes	61	61	+0.0%	⚪
llama (static-cache)	model_size_bytes	425 KB	425 KB	+0.0%	⚪
llama (static-cache)	num_nodes	58	58	+0.0%	⚪
mamba (ssm-text-generation)	model_size_bytes	296 KB	296 KB	+0.0%	⚪
mamba (ssm-text-generation)	num_nodes	98	98	+0.0%	⚪
phi3	model_size_bytes	421 KB	421 KB	+0.0%	⚪
phi3	num_nodes	59	59	+0.0%	⚪
phi3 (static-cache)	model_size_bytes	421 KB	421 KB	+0.0%	⚪
phi3 (static-cache)	num_nodes	56	56	+0.0%	⚪
qwen2	model_size_bytes	425 KB	425 KB	+0.0%	⚪
qwen2	num_nodes	61	61	+0.0%	⚪
qwen2 (static-cache)	model_size_bytes	425 KB	425 KB	+0.0%	⚪
qwen2 (static-cache)	num_nodes	58	58	+0.0%	⚪
qwen3_5_moe (hybrid-text-generation)	model_size_bytes	506 KB	506 KB	+0.0%	⚪
qwen3_5_moe (hybrid-text-generation)	num_nodes	275	275	+0.0%	⚪
qwen3_5_text (hybrid-text-generation)	model_size_bytes	458 KB	458 KB	+0.0%	⚪
qwen3_5_text (hybrid-text-generation)	num_nodes	129	129	+0.0%	⚪
qwen3_5_vl (hybrid-qwen-vl)	model_size_bytes	977 KB	977 KB	+0.0%	⚪
qwen3_5_vl (hybrid-qwen-vl)	num_nodes	408	408	+0.0%	⚪
t5 (seq2seq)	model_size_bytes	836 KB	836 KB	+0.0%	⚪
t5 (seq2seq)	num_nodes	166	166	+0.0%	⚪
whisper (speech-to-text)	model_size_bytes	1008 KB	1008 KB	+0.0%	⚪
whisper (speech-to-text)	num_nodes	128	128	+0.0%	⚪

No performance regressions.

Removed ort-genai from installation and added additional packages. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Removed caching of pip packages and updated dependencies installation. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Removed caching for pip packages and updated dependencies installation. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Removed caching step for pip packages in workflow. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Updated dependencies in CI workflow to include soundfile, librosa, ml_dtypes, flatbuffers, numpy, packaging, protobuf, sympy, and coloredlogs. Also modified onnxruntime-gpu installation to use a pre-release version. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

codecov · 2026-05-07T23:41:35Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Co-authored-by: Justin Chu <justinchuby@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

justinchuby · 2026-05-08T22:30:33Z

@copilot find the correct index for onnxruntime-genai-cuda (or onnxruntime-genai) nightly build, and address comments

- Move nightly CUDA wheel install to run after pip install -e '.[testing]' in main.yml, gpu_l4_golden_parity.yml, gpu_l5_generation_e2e.yml so the CUDA onnxruntime-genai-cuda is never overwritten by the CPU onnxruntime-genai package pulled in by the testing extra. - Change --index-url to --extra-index-url in all four GPU workflows so PyPI is still available for transitive dependency resolution. - Drop --no-deps so pip properly enforces and satisfies dependency constraints. Agent-Logs-Url: https://github.com/onnxruntime/mobius/sessions/98cea0c5-39e8-4f58-9ace-4d7a927d5078 Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>

Copilot · 2026-05-08T22:33:38Z

@copilot find the correct index for onnxruntime-genai-cuda (or onnxruntime-genai) nightly build, and address comments

The nightly index for both onnxruntime-gpu and onnxruntime-genai-cuda is https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-13-nightly/pypi/simple/ (the ort-cuda-13-nightly feed already used in the PR).

Addressed all the review comments in commit 099f6bc:

Switched from --index-url to --extra-index-url across all four GPU workflows so PyPI remains available for transitive dependency resolution and --no-deps can be dropped.
Moved the nightly CUDA wheel install to run after pip install -e '.[testing]' in main.yml, gpu_l4_golden_parity.yml, and gpu_l5_generation_e2e.yml, so onnxruntime-genai-cuda is never overwritten by the CPU onnxruntime-genai that comes in via the testing extra.

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Add additional dependencies to requirements.txt

184cf24

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby requested review from a team and Copilot May 7, 2026 23:30

justinchuby changed the title ~~Add additional dependencies to requirements.txt~~ Use ORT nightly in CI May 7, 2026

Copilot started reviewing on behalf of justinchuby May 7, 2026 23:31 View session

Update CI requirements by removing unused packages

c4e9e32

Removed several dependencies from the CI requirements file. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Copilot AI reviewed May 7, 2026

justinchuby added 2 commits May 7, 2026 16:34

Update dependency installation in CI workflow

7c8b98d

Removed ort-genai from installation and added additional packages. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Refactor GPU E2E workflow by removing pip cache

02e065e

Removed caching of pip packages and updated dependencies installation. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

tadani3 approved these changes May 7, 2026

View reviewed changes

justinchuby added 3 commits May 7, 2026 16:35

Refactor GPU workflow by removing pip cache

8826f64

Removed caching for pip packages and updated dependencies installation. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Remove pip package caching from GPU validation workflow

23e889a

Removed caching step for pip packages in workflow. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby commented May 8, 2026

View reviewed changes

Comment thread .github/workflows/gpu_l4_golden_parity.yml Outdated

justinchuby commented May 8, 2026

View reviewed changes

Comment thread .github/workflows/gpu_l5_generation_e2e.yml Outdated

justinchuby commented May 8, 2026

View reviewed changes

Comment thread .github/workflows/main.yml Outdated

justinchuby commented May 8, 2026

View reviewed changes

Comment thread .github/workflows/validation_examples_gpu.yml Outdated

Apply suggestions from code review

12f13ac

Co-authored-by: Justin Chu <justinchuby@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

titaiwangms requested a review from Copilot May 8, 2026 21:53

Copilot started reviewing on behalf of titaiwangms May 8, 2026 21:54 View session

Copilot AI reviewed May 8, 2026

View reviewed changes

justinchuby marked this pull request as draft May 8, 2026 21:59

Copilot started work on behalf of justinchuby May 8, 2026 22:30 View session

Copilot finished work on behalf of justinchuby May 8, 2026 22:34

Update gpu_l4_golden_parity.yml

5b4f752

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby added 3 commits May 9, 2026 08:03

Update PyTorch installation to CUDA 1.30

cb78474

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Update main.yml

ae5176d

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Update validation_examples_gpu.yml

8b8572b

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby commented May 9, 2026

View reviewed changes

Comment thread .github/workflows/main.yml Outdated

Apply suggestion from @justinchuby

5406b10

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby commented May 9, 2026

View reviewed changes

Comment thread .github/workflows/gpu_l5_generation_e2e.yml Outdated

Apply suggestion from @justinchuby

ba986ba

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby marked this pull request as ready for review May 9, 2026 15:06

Merge branch 'main' into justinchuby-patch-1

53fb00a

justinchuby enabled auto-merge (squash) May 9, 2026 15:06

justinchuby merged commit 47da8cb into main May 9, 2026
20 checks passed

justinchuby deleted the justinchuby-patch-1 branch May 9, 2026 15:12

Uh oh!

Conversation

justinchuby commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison

Uh oh!

codecov Bot commented May 7, 2026

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

justinchuby commented May 8, 2026

Uh oh!

Copilot AI commented May 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

justinchuby commented May 7, 2026 •

edited

Loading

github-actions Bot commented May 7, 2026 •

edited

Loading