Skip to content

Commit fdd1219

Browse files
tadani3Tommaso AdaniCopilot
authored
CI: Extract L4/L5 GPU jobs into reusable workflows and wire detect-affected (#185)
## Summary Refactors the CI pipeline to eliminate duplicate GPU runs, fix an `onnxruntime-gpu` install-order bug, and connect GPU test jobs to the `detect-affected` model detection system for PR-scoped test runs. Also simplifies L3 filtering to use `--models` instead of `-k` for exact `model_type` matching. ## Changes ### `gpu_l4_golden_parity.yml` - Removed `push` trigger to `main` (was causing duplicate runs alongside `main.yml`'s `workflow_call`) - Kept `schedule` (nightly at 4am UTC), `workflow_dispatch`, and `workflow_call` triggers - Added golden-data presence check (`check_golden` step) to skip gracefully when no `.json` files exist - Added affected-model filtering: scopes to `--models` on PRs, runs all on nightly/dispatch - Handles pytest exit code 5 (no matching tests) without failing the job - Conditional Codecov upload and artifact upload gated on `has_golden` ### `gpu_l5_generation_e2e.yml` - Removed `push` trigger to `main` (same duplicate-run fix as L4) - Kept `schedule` (nightly at 5am UTC), `workflow_dispatch`, and `workflow_call` triggers - Added affected-model filtering with the same `--models` flag and exit-code-5 handling ### `validation_examples_gpu.yml` - Fixed `onnxruntime-gpu` being overwritten by CPU `onnxruntime`: the `ort-genai` extra transitively depends on `onnxruntime` (CPU), which was installed *after* `onnxruntime-gpu`, replacing it - Moved `pip install onnxruntime-gpu` after the editable install so the GPU package wins ### `main.yml` - Replaced inlined L4 golden-comparison job with `uses: ./.github/workflows/gpu_l4_golden_parity.yml` - Added new L5 generation-e2e job delegating to `gpu_l5_generation_e2e.yml` - Both jobs depend on `detect-affected` and pass `affected_models` output - Added `needs.detect-affected.result == 'failure'` safety fallback to L4 and L5 conditions — if detection errors out, GPU tests run all models rather than being silently skipped - L3 synthetic parity: replaced `-k` substring filtering with `--models` for exact `model_type` matching ## Behavior | Trigger | L3 | L4 | L5 | |---|---|---|---| | PR with affected models | Runs scoped to affected models | Runs scoped to affected models | Runs scoped to affected models | | PR with no affected models | Skipped | Skipped | Skipped | | PR where `detect-affected` fails | Runs all (safety fallback) | Runs all (safety fallback) | Runs all (safety fallback) | | Push to main | Runs all | Runs all (via `main.yml`) | Runs all (via `main.yml`) | | Nightly schedule | N/A | Runs all (standalone) | Runs all (standalone) | | Manual dispatch | Runs all | Runs all | Runs all | ## Bug fix **`validation_examples_gpu.yml` — `CUDAExecutionProvider` unavailable.** The `ort-genai` pip extra pulls in `onnxruntime` (CPU), which overwrote the previously installed `onnxruntime-gpu`. This caused GPU example validation (e.g. `nemotron_3_nano_text_generation`) to silently fall back to CPU-only execution and fail. Fixed by reordering the install so `onnxruntime-gpu` is installed last. --------- Signed-off-by: Tommaso Adani <83273681+tadani3@users.noreply.github.com> Co-authored-by: Tommaso Adani <tommasoadani@microsoft.com> Co-authored-by: Copilot <copilot@github.com>
1 parent 8d04ba8 commit fdd1219

4 files changed

Lines changed: 124 additions & 117 deletions

File tree

‎.github/workflows/gpu_l4_golden_parity.yml‎

Lines changed: 57 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,24 @@
11
name: "L4: Golden Checkpoint Parity (GPU)"
22

33
on:
4-
push:
5-
branches: [ "main" ]
6-
paths:
7-
- 'src/**'
8-
- 'tests/**'
9-
- 'testdata/**'
10-
- 'scripts/**'
114
schedule:
125
# Run nightly at 4am UTC (after golden regen on Sunday)
136
- cron: '0 4 * * *'
147
workflow_dispatch:
8+
workflow_call:
9+
inputs:
10+
affected_models:
11+
description: >-
12+
JSON array of affected model names (e.g. '["qwen2", "llama"]').
13+
Empty string or empty array means run all models.
14+
required: false
15+
type: string
16+
default: ""
17+
secrets:
18+
HF_TOKEN:
19+
required: false
20+
CODECOV_TOKEN:
21+
required: false
1522

1623
permissions:
1724
contents: read
@@ -61,28 +68,61 @@ jobs:
6168
pip install onnxruntime-gpu
6269
pip install -e '.[testing,transformers]'
6370
71+
- name: Check for golden test data
72+
id: check_golden
73+
run: |
74+
# Only run if there are actual golden .json files (not just .gitkeep)
75+
if find testdata/golden -name '*.json' | grep -q .; then
76+
echo "has_golden=true" >> "$GITHUB_OUTPUT"
77+
else
78+
echo "has_golden=false" >> "$GITHUB_OUTPUT"
79+
echo "No golden files found — skipping"
80+
fi
81+
6482
- name: Run L4 checkpoint-verified tests
83+
if: steps.check_golden.outputs.has_golden == 'true'
6584
env:
6685
HF_TOKEN: ${{ secrets.HF_TOKEN }}
6786
MOBIUS_TEST_DEVICE: cuda
6887
run: |
69-
pytest tests/e2e_golden_test.py \
70-
-m golden \
71-
-v \
72-
--timeout=300 \
73-
--junitxml=junit-l4.xml \
74-
--cov=src --cov-report=xml --cov-branch \
75-
--tb=short
88+
AFFECTED='${{ inputs.affected_models }}'
89+
if [ -z "$AFFECTED" ] || [ "$AFFECTED" = "[]" ]; then
90+
echo "Running all L4 golden comparison tests"
91+
pytest tests/e2e_golden_test.py \
92+
-m golden \
93+
-v \
94+
--timeout=300 \
95+
--junitxml=junit-l4.xml \
96+
--cov=src --cov-report=xml --cov-branch \
97+
--tb=short
98+
else
99+
echo "Running L4 golden tests for affected models: $AFFECTED"
100+
# Convert JSON array to comma-separated list for --models
101+
MODELS=$(echo "$AFFECTED" | python -c "import json, sys; print(','.join(json.load(sys.stdin)))")
102+
if [ -n "$MODELS" ]; then
103+
pytest tests/e2e_golden_test.py \
104+
-m golden \
105+
-v \
106+
--models "$MODELS" \
107+
--timeout=300 \
108+
--junitxml=junit-l4.xml \
109+
--cov=src --cov-report=xml --cov-branch \
110+
--tb=short \
111+
|| { rc=$?; if [ $rc -eq 5 ]; then echo "No matching tests found — skipping"; else exit $rc; fi; }
112+
else
113+
echo "No affected models — skipping"
114+
fi
115+
fi
76116
timeout-minutes: 60
77117

78118
- name: Upload coverage to Codecov
79-
if: always()
119+
if: always() && steps.check_golden.outputs.has_golden == 'true'
80120
uses: codecov/codecov-action@v6
81121
with:
82122
token: ${{ secrets.CODECOV_TOKEN }}
83123
flags: gpu-l4
84124
- name: Upload L4 test results to Codecov
85-
if: ${{ !cancelled() }}
125+
if: steps.check_golden.outputs.has_golden == 'true' && !cancelled()
86126
uses: codecov/codecov-action@v6
87127
with:
88128
token: ${{ secrets.CODECOV_TOKEN }}
@@ -91,7 +131,7 @@ jobs:
91131
files: junit-l4.xml
92132

93133
- name: Upload test results
94-
if: always()
134+
if: always() && steps.check_golden.outputs.has_golden == 'true'
95135
uses: actions/upload-artifact@v7
96136
with:
97137
name: gpu-l4-test-results

‎.github/workflows/gpu_l5_generation_e2e.yml‎

Lines changed: 42 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,24 @@
11
name: "L5: End-to-End Generation (GPU)"
22

33
on:
4-
push:
5-
branches: [ "main" ]
6-
paths:
7-
- 'src/**'
8-
- 'tests/**'
9-
- 'testdata/**'
10-
- 'scripts/**'
114
schedule:
125
# Run nightly at 5am UTC (after L4 golden tests)
136
- cron: '0 5 * * *'
147
workflow_dispatch:
8+
workflow_call:
9+
inputs:
10+
affected_models:
11+
description: >-
12+
JSON array of affected model names (e.g. '["qwen2", "llama"]').
13+
Empty string or empty array means run all models.
14+
required: false
15+
type: string
16+
default: ""
17+
secrets:
18+
HF_TOKEN:
19+
required: false
20+
CODECOV_TOKEN:
21+
required: false
1522

1623
permissions:
1724
contents: read
@@ -66,13 +73,34 @@ jobs:
6673
HF_TOKEN: ${{ secrets.HF_TOKEN }}
6774
MOBIUS_TEST_DEVICE: cuda
6875
run: |
69-
pytest tests/e2e_golden_test.py \
70-
-m generation \
71-
-v \
72-
--timeout=300 \
73-
--junitxml=junit-l5.xml \
74-
--cov=src --cov-report=xml --cov-branch \
75-
--tb=short
76+
AFFECTED='${{ inputs.affected_models }}'
77+
if [ -z "$AFFECTED" ] || [ "$AFFECTED" = "[]" ]; then
78+
echo "Running all L5 generation E2E tests"
79+
pytest tests/e2e_golden_test.py \
80+
-m generation \
81+
-v \
82+
--timeout=300 \
83+
--junitxml=junit-l5.xml \
84+
--cov=src --cov-report=xml --cov-branch \
85+
--tb=short
86+
else
87+
echo "Running L5 generation tests for affected models: $AFFECTED"
88+
# Convert JSON array to comma-separated list for --models
89+
MODELS=$(echo "$AFFECTED" | python -c "import json, sys; print(','.join(json.load(sys.stdin)))")
90+
if [ -n "$MODELS" ]; then
91+
pytest tests/e2e_golden_test.py \
92+
-m generation \
93+
-v \
94+
--models "$MODELS" \
95+
--timeout=300 \
96+
--junitxml=junit-l5.xml \
97+
--cov=src --cov-report=xml --cov-branch \
98+
--tb=short \
99+
|| { rc=$?; if [ $rc -eq 5 ]; then echo "No matching tests found — skipping"; else exit $rc; fi; }
100+
else
101+
echo "No affected models — skipping"
102+
fi
103+
fi
76104
timeout-minutes: 60
77105

78106
- name: Upload coverage to Codecov

‎.github/workflows/main.yml‎

Lines changed: 24 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -148,15 +148,9 @@ jobs:
148148
else
149149
AFFECTED='${{ needs.detect-affected.outputs.affected }}'
150150
echo "Running parity for affected models: $AFFECTED"
151-
# Escape hyphens to underscores for pytest -k compatibility
152-
FILTER=$(echo "$AFFECTED" | python -c "
153-
import json, sys, re
154-
names = json.load(sys.stdin)
155-
escaped = [re.sub(r'[^a-zA-Z0-9_]', '_', n) for n in names]
156-
print(' or '.join(escaped))
157-
")
158-
if [ -n "$FILTER" ]; then
159-
pytest tests/synthetic_parity_test.py -v --tb=short -n auto -k "$FILTER" \
151+
MODELS=$(echo "$AFFECTED" | python -c "import json, sys; print(','.join(json.load(sys.stdin)))")
152+
if [ -n "$MODELS" ]; then
153+
pytest tests/synthetic_parity_test.py -v --tb=short -n auto --models "$MODELS" \
160154
--cov=src --cov-report=xml --cov-branch --junitxml junit.xml
161155
else
162156
echo "No affected models — skipping"
@@ -179,88 +173,33 @@ jobs:
179173

180174
golden-comparison:
181175
name: L4 Golden Comparison
182-
runs-on:
183-
- self-hosted
184-
- "1ES.Pool=onnxruntime-ep-mobius-github-linux-a10"
185-
- "JobId=gpu-golden-comparison-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}"
186176
needs: [detect-affected]
187177
if: >-
188178
always() && !cancelled() &&
189179
(needs.detect-affected.result == 'failure' ||
190180
needs.detect-affected.outputs.has_affected == 'true' ||
191181
github.event_name != 'pull_request')
192-
steps:
193-
- uses: actions/checkout@v6
194-
- name: Setup Python
195-
uses: actions/setup-python@v6
196-
with:
197-
python-version: "3.12"
198-
- name: Cache pip packages
199-
uses: actions/cache@v5
200-
with:
201-
path: ~/.cache/pip
202-
key: pip-golden-${{ hashFiles('pyproject.toml', 'requirements/ci/requirements.txt') }}
203-
restore-keys: |
204-
pip-golden-
205-
- name: Cache HuggingFace models
206-
uses: actions/cache@v5
207-
with:
208-
path: ~/.cache/huggingface
209-
key: hf-golden-${{ hashFiles('testdata/cases/**/*.yaml') }}
210-
restore-keys: |
211-
hf-golden-
212-
- name: Install PyTorch CUDA
213-
run: pip install torch --index-url https://download.pytorch.org/whl/cu124
214-
- name: Install dependencies
215-
run: |
216-
pip install -r requirements/ci/requirements.txt
217-
pip install onnxruntime-gpu
218-
pip install -e '.[testing,transformers]'
219-
- name: Check for golden test data
220-
id: check_golden
221-
run: |
222-
# Only run if there are actual golden .json files (not just .gitkeep)
223-
if find testdata/golden -name '*.json' | grep -q .; then
224-
echo "has_golden=true" >> "$GITHUB_OUTPUT"
225-
else
226-
echo "has_golden=false" >> "$GITHUB_OUTPUT"
227-
echo "No golden files found — skipping"
228-
fi
229-
- name: Run L4 golden comparison tests
230-
if: steps.check_golden.outputs.has_golden == 'true'
231-
env:
232-
HF_TOKEN: ${{ secrets.HF_TOKEN }}
233-
MOBIUS_TEST_DEVICE: cuda
234-
run: |
235-
if [ ! -f tests/e2e_golden_test.py ]; then
236-
echo "tests/e2e_golden_test.py not found — skipping"
237-
exit 0
238-
fi
239-
pytest tests/e2e_golden_test.py -m golden -v --tb=short \
240-
--cov=src --cov-report=xml --cov-branch \
241-
--junitxml golden-results.xml
242-
timeout-minutes: 60
243-
- name: Upload coverage to Codecov
244-
if: steps.check_golden.outputs.has_golden == 'true'
245-
uses: codecov/codecov-action@v6
246-
with:
247-
token: ${{ secrets.CODECOV_TOKEN }}
248-
flags: golden
249-
- name: Upload test results to Codecov
250-
if: steps.check_golden.outputs.has_golden == 'true' && !cancelled()
251-
uses: codecov/codecov-action@v6
252-
with:
253-
token: ${{ secrets.CODECOV_TOKEN }}
254-
flags: golden
255-
report-type: test_results
256-
files: golden-results.xml
257-
- name: Upload golden test results
258-
if: always() && steps.check_golden.outputs.has_golden == 'true'
259-
uses: actions/upload-artifact@v7
260-
with:
261-
name: golden-test-results
262-
path: golden-results.xml
263-
retention-days: 7
182+
uses: ./.github/workflows/gpu_l4_golden_parity.yml
183+
with:
184+
affected_models: ${{ needs.detect-affected.outputs.affected }}
185+
secrets:
186+
HF_TOKEN: ${{ secrets.HF_TOKEN }}
187+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
188+
189+
generation-e2e:
190+
name: L5 Generation E2E
191+
needs: [detect-affected]
192+
if: >-
193+
always() && !cancelled() &&
194+
(needs.detect-affected.result == 'failure' ||
195+
needs.detect-affected.outputs.has_affected == 'true' ||
196+
github.event_name != 'pull_request')
197+
uses: ./.github/workflows/gpu_l5_generation_e2e.yml
198+
with:
199+
affected_models: ${{ needs.detect-affected.outputs.affected }}
200+
secrets:
201+
HF_TOKEN: ${{ secrets.HF_TOKEN }}
202+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
264203

265204
test:
266205
name: test

‎.github/workflows/validation_examples_gpu.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,8 +89,8 @@ jobs:
8989
- name: Install dependencies
9090
run: |
9191
pip install -r requirements/ci/requirements.txt
92-
pip install onnxruntime-gpu
9392
pip install -e '.[testing,transformers,ort-genai]'
93+
pip install onnxruntime-gpu
9494
pip install soundfile librosa ml_dtypes
9595
9696
- name: Run ${{ matrix.name }}

0 commit comments

Comments
 (0)