Skip to content

CI: Fix GPU workflow dependencies and increase timeouts - #179

Merged
tadani3 merged 6 commits into
mainfrom
tommasoadani/gpu_workflow_fixes
Apr 21, 2026
Merged

CI: Fix GPU workflow dependencies and increase timeouts#179
tadani3 merged 6 commits into
mainfrom
tommasoadani/gpu_workflow_fixes

Conversation

@tadani3

@tadani3 tadani3 commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

CI: Fix GPU workflow dependencies and increase timeouts

Summary

Ensures all GPU CI jobs use CUDA-enabled dependencies (torch with CUDA index, onnxruntime-gpu) and increases test step timeouts to 60 minutes across GPU workflows.

Changes

1. Bump onnxruntime-easy minimum version

  • pyproject.toml: onnxruntime-easyonnxruntime-easy>=0.1.1
  • >=0.1.1 no longer bundles a specific onnxruntime variant, allowing explicit control over CPU vs GPU installs without conflicts.

2. Explicit onnxruntime / onnxruntime-gpu installs

Since onnxruntime-easy>=0.1.1 no longer pulls in onnxruntime itself, every workflow now explicitly installs the correct variant:

Workflow Job Package
main.yml smoke-tests onnxruntime
main.yml synthetic-parity onnxruntime
main.yml multi-python onnxruntime
main.yml golden-comparison onnxruntime-gpu
main.yml integration-fast onnxruntime-gpu
gpu_l4_golden_parity.yml golden-tests onnxruntime-gpu
gpu_l5_generation_e2e.yml generation-tests onnxruntime-gpu
validation_examples_gpu.yml run-example onnxruntime-gpu
golden_regen.yml regenerate conditional (onnxruntime or onnxruntime-gpu based on inputs.device)
pages.yml build onnxruntime

3. Fix PyTorch install on GPU runner

  • main.yml (golden-comparison): Changed from pip install torch --index-url .../cpu to pip install torch --index-url .../cu124. This job runs on an A10 GPU but was previously installing CPU-only PyTorch.

4. Increase GPU test step timeouts to 60 minutes

Workflow Previous New
gpu_l5_generation_e2e.yml (step) 30 min 60 min
gpu_l4_golden_parity.yml (step) 30 min 60 min
main.yml — golden-comparison (step) 30 min 60 min

Job-level timeouts on the dedicated GPU workflows were already 60 minutes; only the step-level timeouts were lagging behind.

Testing

No code changes — CI configuration only. Changes can be verified by observing the next workflow runs install the correct packages.

@tadani3 tadani3 self-assigned this Apr 21, 2026
@tadani3
tadani3 requested review from a team and Copilot April 21, 2026 00:21
@github-actions

Copy link
Copy Markdown

Performance Comparison

Comparing dbcc8b66029b2c

Model Metric Baseline Current Delta
bert (feature-extraction) model_size_bytes 359 KB 359 KB +0.0%
bert (feature-extraction) num_nodes 61 61 +0.0%
falcon model_size_bytes 364 KB 364 KB +0.0%
falcon num_nodes 66 66 +0.0%
gemma2 model_size_bytes 428 KB 428 KB +0.0%
gemma2 num_nodes 107 107 +0.0%
gpt2 model_size_bytes 388 KB 388 KB +0.0%
gpt2 num_nodes 53 53 +0.0%
llama model_size_bytes 425 KB 425 KB +0.0%
llama num_nodes 61 61 +0.0%
llama (static-cache) model_size_bytes 425 KB 425 KB +0.0%
llama (static-cache) num_nodes 58 58 +0.0%
mamba (ssm-text-generation) model_size_bytes 360 KB 360 KB +0.0%
mamba (ssm-text-generation) num_nodes 98 98 +0.0%
phi3 model_size_bytes 421 KB 421 KB +0.0%
phi3 num_nodes 59 59 +0.0%
phi3 (static-cache) model_size_bytes 421 KB 421 KB +0.0%
phi3 (static-cache) num_nodes 56 56 +0.0%
qwen2 model_size_bytes 425 KB 425 KB +0.0%
qwen2 num_nodes 61 61 +0.0%
qwen2 (static-cache) model_size_bytes 425 KB 425 KB +0.0%
qwen2 (static-cache) num_nodes 58 58 +0.0%
qwen3_5_moe (hybrid-text-generation) model_size_bytes 506 KB 506 KB +0.0%
qwen3_5_moe (hybrid-text-generation) num_nodes 275 275 +0.0%
qwen3_5_text (hybrid-text-generation) model_size_bytes 458 KB 458 KB +0.0%
qwen3_5_text (hybrid-text-generation) num_nodes 129 129 +0.0%
qwen3_5_vl (hybrid-qwen-vl) model_size_bytes 977 KB 977 KB +0.0%
qwen3_5_vl (hybrid-qwen-vl) num_nodes 408 408 +0.0%
t5 (seq2seq) model_size_bytes 836 KB 836 KB +0.0%
t5 (seq2seq) num_nodes 166 166 +0.0%
whisper (speech-to-text) model_size_bytes 1008 KB 1008 KB +0.0%
whisper (speech-to-text) num_nodes 128 128 +0.0%

No performance regressions.

@tadani3
tadani3 enabled auto-merge (squash) April 21, 2026 00:24
@codecov

codecov Bot commented Apr 21, 2026

Copy link
Copy Markdown

The author of this PR, tadani3, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

@tadani3
tadani3 merged commit 95473ab into main Apr 21, 2026
21 of 22 checks passed
@tadani3
tadani3 deleted the tommasoadani/gpu_workflow_fixes branch April 21, 2026 00:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates CI dependency installation to ensure GPU workflows use CUDA-capable packages and extends GPU test step timeouts, aligning with the repo’s ONNX Runtime inference-based test suite.

Changes:

  • Bump onnxruntime-easy minimum version to allow workflows to choose CPU vs GPU ONNX Runtime explicitly.
  • Explicitly install onnxruntime (CPU) or onnxruntime-gpu (GPU) across workflows instead of relying on transitive installs.
  • Increase GPU test step timeouts to 60 minutes in golden/generation workflows and the main GPU golden comparison job.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
pyproject.toml Bumps onnxruntime-easy minimum version in the testing extra.
.github/workflows/main.yml Installs onnxruntime for CPU test jobs, fixes GPU job to install CUDA torch + onnxruntime-gpu, and increases golden step timeout.
.github/workflows/gpu_l4_golden_parity.yml Installs onnxruntime-gpu and increases test step timeout to 60 minutes.
.github/workflows/gpu_l5_generation_e2e.yml Installs onnxruntime-gpu and increases test step timeout to 60 minutes.
.github/workflows/validation_examples_gpu.yml Installs onnxruntime-gpu for GPU example validation runs.
.github/workflows/golden_regen.yml Conditionally installs onnxruntime vs onnxruntime-gpu based on inputs.device.
.github/workflows/pages.yml Installs onnxruntime for docs build environment consistency.

Comment thread pyproject.toml
Comment on lines 31 to 35
testing = [
"onnxruntime-easy",
"onnxruntime-easy>=0.1.1",
"torch",
"transformers>=5.0",
"safetensors",
Comment on lines 72 to +75
--junitxml=junit-l4.xml \
--cov=src --cov-report=xml --cov-branch \
--tb=short
timeout-minutes: 30
timeout-minutes: 60
Comment on lines 72 to +75
--junitxml=junit-l5.xml \
--cov=src --cov-report=xml --cov-branch \
--tb=short
timeout-minutes: 30
timeout-minutes: 60
Comment on lines +76 to +83
- name: Install onnxruntime (CPU)
if: inputs.device != 'cuda'
run: pip install onnxruntime

- name: Install onnxruntime (GPU)
if: inputs.device == 'cuda'
run: pip install onnxruntime-gpu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants