fix(ci): restore cross-platform test and quality gates by justinchuby · Pull Request #1576 · justinchuby/onnx-genai

justinchuby · 2026-08-20T14:53:05Z

Summary

The seven failed CI jobs reduced to four independent baseline defects:

Nested offline Cargo metadata resolved the full graph and required registry packages absent from CI caches. The MLAS exclusion gate now uses dependency-free metadata and resolves workspace feature activation deterministically in memory.
The ARM SIMD log sweep treated zero numerical error as proof that no samples ran. Exercise accounting is now independent of error magnitude, with direct AVX2 and pure ARM/NEON sample-generation coverage.
The CELU unsupported sentinel became stale after CELU support landed. Latest main now supplies the replacement Det negative control via ci: restore a red main — refresh the conformance negative control and formatting #1578, so the redundant PR commit was dropped during rebase.
A source-structure test assumed LF line endings. It now strips test-only items structurally, normalizes CRLF/LF input, and covers CRLF explicitly.

The PR also retains the original CUDA CI repairs:

classify only genuinely CUDA-bound integration targets in the honesty checker
make GPU-enabled Marlin numerics fail loudly when CUDA initialization is unavailable
resolve Rust 1.97 CUDA Clippy/rustfmt failures and remove the duplicate split-K test attribute

Validation

Validated after rebasing onto main at 0faa6354:

cargo fmt --all -- --check
CPU EP, CPU plugin, and EP plugin test suites
relevant CPU/plugin/session Clippy with -D warnings
CUDA test suite: 451 passed, 25 ignored
CUDA Clippy with -D warnings
CUDA honesty with CUDA hidden: 433 tests / 68 targets; 425 fail-loud, 8 ignored, 0 false passes
conformance harness self-tests: 5 passed
full EP conformance: 28 PASS, 1 expected UNSUPPORTED, 0 mismatch/error

All commits are signed off.

codecov · 2026-08-20T15:08:18Z

Codecov Report

❌ Patch coverage is 96.70330% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.91%. Comparing base (0faa635) to head (806d553).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	96.15%	2 Missing ⚠️
crates/onnx-runtime-ep-plugin/src/compute.rs	97.43%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1576      +/-   ##
==========================================
+ Coverage   80.77%   80.91%   +0.13%     
==========================================
  Files         381      382       +1     
  Lines      173645   178155    +4510     
  Branches   173645   178155    +4510     
==========================================
+ Hits       140262   144150    +3888     
- Misses      28492    29073     +581     
- Partials     4891     4932      +41

Flag	Coverage Δ
cli-ort-linux	`82.60% <ø> (ø)`
cli-ort-windows	`82.19% <ø> (+0.09%)`	⬆️
mlas	`85.09% <ø> (ø)`
offline	`80.78% <96.70%> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-plugin/src/compute.rs	`83.20% <97.43%> (+0.11%)`	⬆️
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	`91.18% <96.15%> (+0.20%)`	⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Bind the optional interleaved kernel entry directly in the guarded branch so Rust 1.97 Clippy no longer reports unnecessary_unwrap. The let-chain retains the interleave enablement short-circuit and leaves the original kernel and packed pointer fallback unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39ff6824-d35f-4d3b-8f5f-043a7119a100 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Filter Cargo integration-test artifacts by the established _gpu naming convention, while retaining the historical Marlin numerics target explicitly. This keeps CPU-only tests out of CUDA libtest accounting without weakening coverage of genuine device tests. Make the Marlin numerics target fail loudly when gpu-tests is enabled but CUDA initialization is unavailable, instead of returning success after a silent skip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39ff6824-d35f-4d3b-8f5f-043a7119a100 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Apply the current rustfmt layout and resolve new Clippy diagnostics without changing CUDA execution behavior. Keep the explicit argument-count exemption on the raw dequant kernel launcher, matching its fp16 sibling and CUDA ABI mapping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39ff6824-d35f-4d3b-8f5f-043a7119a100 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Apply rustfmt to the newly merged bandwidth-threshold assertion and remove a duplicate test attribute so the latest main baseline is warning-free. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39ff6824-d35f-4d3b-8f5f-043a7119a100 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Use cargo metadata --no-deps so the nested probe only reads workspace manifests and never needs registry package sources. Resolve the plugin's default workspace dependency and feature closure deterministically, with positive controls for unconditional and feature-forwarded MLAS activation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

The dense log sweep treated any nonzero approximation error as proof that samples were exercised. ARM dispatches this kernel to the exact scalar implementation, so every comparison was bit-identical and the test incorrectly reported an empty sweep. Count checked samples independently, call AVX2 directly when available, and verify the shared input generator produces valid four-lane ARM batches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Scan production items structurally instead of splitting on an LF-only test-module marker. This keeps the scratch TLS resolution invariant meaningful while excluding interleaved test helpers and adds explicit CRLF coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby · 2026-08-20T16:14:55Z

Rebased onto current main and force-updated the branch at 806d5536. The previously red matrix was traced to four independent defects; three remain fixed in this PR, while the stale conformance sentinel was independently fixed upstream in #1578 and the redundant commit was dropped. Full CPU/plugin, CUDA, formatting, Clippy, honesty, and EP conformance validation passes on the rebased head.

main moved by six commits while this PR was open. One real conflict, two auto-merges, twenty main-only files. .github/scripts/verify_cuda_test_honesty.py -- took main's version wholesale, dropping this stack's three `ALWAYS_RUN` entries (`deferred_release_queue`, `vmm_release_quarantine`, `no_built_in_eager_allocator`) and the comment block arguing for them. This is not a loss. #1576 replaced the mechanism: instead of exempting CPU-only probes from the "ignored, not passed" GPU rule via a whitelist, the script now only classifies a target as GPU-bound in the first place if its name ends in `_gpu` or it is in the explicit `CUDA_TARGETS_WITHOUT_SUFFIX` set (line 93-94). Checked mechanically: none of the three probe targets end in `_gpu` and none are in that set, so none is policed, so none needs exempting. Keeping the whitelist entries would have left a second, redundant mechanism doing nothing. The rationale text is not lost either -- each of the three test files already carries its own module header explaining why it runs on the CPU, which is where that argument belongs. main dropped the equivalent comments for its own probes for the same reason, so re-adding only ours would have been asymmetric. Resolved file is byte-identical to origin/main's blob. crates/onnx-runtime-ep-cuda/src/provider.rs and crates/onnx-genai-engine/src/pipeline/flat_autoregressive.rs -- auto-merged. Verified by extracting every line main added over the merge-base and confirming each is present in the merge result; both came back clean. The merge result is main plus this stack's additions (bound-ownership docs and imports; the CUDA governor wiring). Twenty main-only files verified byte-identical to origin/main by blob hash, not by reading the diff. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c80f8522-983c-47f7-8241-2155a823aabe

…site Two gates are currently red on pristine main (0da1fe0), independently of this branch: * `Rust quality` runs `cargo fmt --all -- --check`, and #1598 landed six files unformatted (decode/mod.rs, decode/state.rs, native_decode/{cuda, mod,tests}.rs). * `Fast` runs clippy with -D warnings, and #1576 added one more chunks_exact site in simd_activations.rs's ARM sweep test. Both are fixed here so the branch can prove the gates green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby and others added 7 commits August 20, 2026 16:10

justinchuby force-pushed the copilot/fix-main-ci branch from 021e2e4 to 806d553 Compare August 20, 2026 16:14

justinchuby changed the title ~~fix(ci): restore CUDA honesty and Rust quality gates~~ fix(ci): restore cross-platform test and quality gates Aug 20, 2026

justinchuby merged commit 136ce20 into main Aug 20, 2026
18 of 19 checks passed

justinchuby deleted the copilot/fix-main-ci branch August 20, 2026 17:25

justinchuby mentioned this pull request Aug 20, 2026

Merge the #1186 memory stack (phases 1-7) onto main #1579

Merged

justinchuby mentioned this pull request Aug 20, 2026

matmul_nbits_marlin_numerics silently skips instead of failing loud, so the CUDA honesty guard is red on main #1589

Closed

Navigation Menu

Repository navigation

fix(ci): restore cross-platform test and quality gates - #1576

fix(ci): restore cross-platform test and quality gates#1576
justinchuby merged 7 commits into
mainfrom
copilot/fix-main-ci

justinchuby commented Aug 20, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 20, 2026 •

edited

Loading

Uh oh!

justinchuby commented Aug 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

codecov Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

justinchuby commented Aug 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

justinchuby commented Aug 20, 2026 •

edited

Loading

codecov Bot commented Aug 20, 2026 •

edited

Loading