Skip to content

ci: Split L5 GPU workflows from L4 - #164

Merged
tadani3 merged 6 commits into
mainfrom
tommasoadani/ci_l5_test_split
Apr 14, 2026
Merged

ci: Split L5 GPU workflows from L4#164
tadani3 merged 6 commits into
mainfrom
tommasoadani/ci_l5_test_split

Conversation

@tadani3

@tadani3 tadani3 commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

This pull request restructures the GPU CI workflows by splitting the previous combined L4/L5 GPU test workflow into two distinct workflows: one for L4 Golden Checkpoint Parity tests and another for L5 End-to-End Generation tests. This improves clarity, maintainability, and scheduling flexibility for GPU-based tests.

Workflow restructuring and separation:

  • Added a new workflow .github/workflows/gpu_l4_golden_parity.yml dedicated to running L4 Golden Checkpoint Parity tests on GPU, including setup, caching, test execution, and result uploads.
  • Added a new workflow .github/workflows/gpu_l5_generation_e2e.yml for L5 End-to-End Generation tests on GPU, structured similarly to the L4 workflow but targeting generation correctness.

Modifications to the existing GPU test workflow:

  • The original .github/workflows/gpu_tests.yml is now deleted. The new L4 and L5 workflows have also been updated to run on a nightly schedule.

@tadani3 tadani3 self-assigned this Apr 13, 2026
@tadani3
tadani3 requested review from a team and Copilot April 13, 2026 18:27
@github-actions

github-actions Bot commented Apr 13, 2026

Copy link
Copy Markdown

Performance Comparison

Comparing 2c2e2cd5735a02

Model Metric Baseline Current Delta
bert (feature-extraction) model_size_bytes 359 KB 359 KB +0.0%
bert (feature-extraction) num_nodes 61 61 +0.0%
falcon model_size_bytes 364 KB 364 KB +0.0%
falcon num_nodes 66 66 +0.0%
gemma2 model_size_bytes 428 KB 428 KB +0.0%
gemma2 num_nodes 107 107 +0.0%
gpt2 model_size_bytes 388 KB 388 KB +0.0%
gpt2 num_nodes 53 53 +0.0%
llama model_size_bytes 425 KB 425 KB +0.0%
llama num_nodes 61 61 +0.0%
llama (static-cache) model_size_bytes 425 KB 425 KB +0.0%
llama (static-cache) num_nodes 58 58 +0.0%
mamba (ssm-text-generation) model_size_bytes 360 KB 360 KB -0.0%
mamba (ssm-text-generation) num_nodes 98 103 +5.1% ⚠️
phi3 model_size_bytes 421 KB 421 KB +0.0%
phi3 num_nodes 59 59 +0.0%
phi3 (static-cache) model_size_bytes 421 KB 421 KB +0.0%
phi3 (static-cache) num_nodes 56 56 +0.0%
qwen2 model_size_bytes 425 KB 425 KB +0.0%
qwen2 num_nodes 61 61 +0.0%
qwen2 (static-cache) model_size_bytes 425 KB 425 KB +0.0%
qwen2 (static-cache) num_nodes 58 58 +0.0%
qwen3_5_moe (hybrid-text-generation) model_size_bytes 506 KB 506 KB +0.0%
qwen3_5_moe (hybrid-text-generation) num_nodes 275 275 +0.0%
qwen3_5_text (hybrid-text-generation) model_size_bytes 458 KB 458 KB +0.0%
qwen3_5_text (hybrid-text-generation) num_nodes 129 129 +0.0%
qwen3_5_vl (hybrid-qwen-vl) model_size_bytes 977 KB 977 KB +0.0%
qwen3_5_vl (hybrid-qwen-vl) num_nodes 409 409 +0.0%
t5 (seq2seq) model_size_bytes 836 KB 836 KB +0.0%
t5 (seq2seq) num_nodes 174 174 +0.0%
whisper (speech-to-text) model_size_bytes 1008 KB 1008 KB +0.0%
whisper (speech-to-text) num_nodes 128 128 +0.0%

Warning: minor regressions detected. Review flagged metrics.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Splits GPU CI coverage into separate workflows for L4 golden checkpoint parity and L5 end-to-end generation tests to improve clarity and scheduling flexibility.

Changes:

  • Refactors existing gpu_tests.yml to run only L4 golden parity and updates Codecov flags/artifact naming accordingly.
  • Adds gpu_l4_golden_parity.yml to run L4 golden parity as its own GPU workflow.
  • Adds gpu_l5_generation_e2e.yml to run L5 generation E2E as its own GPU workflow.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/gpu_tests.yml Updated to L4-only GPU golden parity, with renamed workflow/job metadata and uploads.
.github/workflows/gpu_l4_golden_parity.yml New dedicated L4 GPU golden parity workflow (currently overlaps with gpu_tests.yml).
.github/workflows/gpu_l5_generation_e2e.yml New dedicated L5 GPU generation E2E workflow with its own schedule and uploads.

Comment thread .github/workflows/gpu_l4_golden_parity.yml
Comment thread .github/workflows/gpu_tests.yml Outdated
@codecov

codecov Bot commented Apr 13, 2026

Copy link
Copy Markdown

The author of this PR, tadani3, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

@tadani3
tadani3 merged commit 2024747 into main Apr 14, 2026
18 of 19 checks passed
@tadani3
tadani3 deleted the tommasoadani/ci_l5_test_split branch April 14, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants