fix(ep-cuda): restore the CUDA lane -- five defects, only the first of which was reported

[justinchuby]

Closes #1875.

CUDA compile (Linux x86_64) has been red since #1836 (6e4b0ebb3); last green was cb81745b0. Resch handed this off explicitly rather than guessing at another domain's intended API visibility — that was the right call, because the reported compile error was masking two further failures on the same lane. Fixing only the error would have moved it from "red at build" to "red at the honesty script".

Two more arrived on main while this PR was open — #1884 and #1895, both today. Five independent defects from five different PRs, reviewable separately. The lane cannot go green on any proper subset of them.

---

1. Unresolved import — the reported error

content_preserving_transition_gpu.rs imports transition_granule_range_with_phase8_faults, gated #[cfg(any(test, feature = "gpu-tests"))]. The test arm does not cover an integration test: tests/*.rs are separate crates linking the library built without cfg(test), so in the without-gpu-tests configuration the item is genuinely absent. Invisible to any local run passing --features gpu-tests.

I rejected both options in the issue, with evidence:

• Widen the gate contradicts the function's own doc — "Not reachable from production" — and would ship an API that forces cuMemUnmap / cuMemMap / cuMemSetAccess to fail to every consumer.
• required-features is exactly what the honesty lane exists to forbid: compare_inventories emits "exists only with gpu-tests enabled; CUDA tests must not hide from CPU inventory". It trades a compile error for a lane failure and deletes the CPU-side inventory of 14 tests.

Fix: a two-arm local shim, mirroring with_faults in crates/onnx-runtime-cuda-memory/tests/virtual_memory_gpu.rs, which already solves this identical problem for CudaVirtualBacking::with_driver_faults. The gate stays intact; the target compiles and lists its tests in both configurations. The unreachable! arm is genuinely unreachable — after change 2 every test in that file is #[ignore]d.

#1895 reached this file first, and took the option rejected above.
1be9f2cc2 applied #![cfg(feature = "gpu-tests")] to the whole target. It fixed the compile and traded it for an inventory failure on the same lane. From the job log on main@1be9f2cc2 (job 97266415620), not inferred:

- content_preserving_transition_gpu: Cargo reported no integration tests
+ 19 × "test exists only with gpu-tests enabled"

The target was not repaired, it was hidden — the symptom stopped being reported without the property becoming true. Its module doc argued that a compile failure "is not a useful signal about a suite that cannot run on a machine with no GPU anyway", which is the exact proposition verify_cuda_test_honesty.py exists to reject. This PR removes that line and records the reasoning in the module doc so the option isn't tried a third time.

2. Five non-ignored tests in a _gpu target — 4 honesty violations

The script requires every test in a _gpu target to be ignored, not passed on a CPU-only runner. That rule is how the suite is stopped from reporting green for a GPU it never touched.

Three were genuinely CPU-only predicate tests over verify_safe_point — pure logic over ResizeSafePoint, no driver. Moved to a new non-_gpu target, not #[ignore]d: silencing them would green the lane by deleting coverage, and target naming is the escape hatch the script's own comment names, so that "a genuinely CPU-only target is not policed as a device test merely because it lives in a CUDA crate".

Correction after review — moving them was not sufficient, and my first commit shipped the same defect this PR is about.
A non-_gpu target is skipped by the honesty script, and .github/scripts/workspace_test_packages.py:26 deny-lists this crate from every offline lane. So the new target was compiled and never executed — the moved tests, and the mutation guard I added specifically to catch a bad verify_safe_point, would have run on zero CI runs. My claim that moving "keeps them executing on every run" was exactly backwards: before the move the honesty script did execute them (that was the violation); after it, nothing did.
This is the same failure mode as the defect being fixed — a test that isn't on the leg you're citing — and it would have produced a green check proving nothing. Fixed by adding an explicit --test content_preserving_transition step to the CUDA lane, mirroring the two CPU-only targets in this crate that already have one for precisely this reason (ci.yml:989-998). Verified by running the added command verbatim: 4 passed.

Two asserted nothing and are removed:

• fault_injection_safe_point_recheck_rejects — comments plus a println!; self-describes as "this test documents the contract".
• zero_len_is_committed_noop — println!s that the behaviour is "verified by implementation".

Both passed unconditionally. This touches another author's tests, so I flag it explicitly rather than burying it in the diff. Deleting a test that asserts nothing loses no coverage — but to be precise about what is and isn't covered: the first does have real sibling coverage in fault_injection_recheck_safe_point_rejected_gpu, and the second does not — the len == 0 early return is untested anywhere. That gap is pre-existing, and a genuine test needs a device (the early return still takes &CudaRuntime/&mut CudaReservation), so I have left it for a GPU-capable change rather than claim coverage that doesn't exist.

Added safe_point_accepts_a_clean_state. The three moved tests only ever assert is_err(), so they cannot distinguish "rejects the unsafe field" from "rejects everything". Empirically confirmed — with verify_safe_point mutated to reject unconditionally:

test result: FAILED. 3 passed; 1 failed
failures:
    safe_point_accepts_a_clean_state

All three inherited tests survive the mutant. Only the new one kills it.

3. The manifest guard checked a proxy, not the property it claimed

if "gpu-tests = []" not in manifest:
    errors.append(f"crates/{crate.name}/Cargo.toml must define a gpu-tests feature")

A literal substring match. #1860 turned this red by changing the value to ["onnx-runtime-cuda-memory/gpu-tests"] — a legitimate and necessary forwarding. The guard's message claims to check that the feature is defined; it actually checked that it was defined with one particular empty value.

Now matches the feature key whatever its value, extracted into declares_gpu_tests_feature() so it is covered by the script's own --self-test fixtures — which it previously was not. Fixtures include the forwarding form, a multi-line list, a commented-out line, and a features = ["gpu-tests"] dependency mention that must not count.

Note for @resch: #1860 broke this lane as well as the shape-inference pin you fixed in #1870 — same commit, two unrelated pins that both encoded a value rather than the property.

4. A GPU test in a _gpu target with no #[ignore] — #1895

causal_conv_with_state_gpu::the_standard_domain_spelling_reaches_the_same_kernel calls require_cuda() exactly like its two siblings in the same file, but is missing their #[cfg_attr(not(feature = "gpu-tests"), ignore = …)]. So it ran on a CPU runner and failed:

- causal_conv_with_state_gpu: 1 tests failed while checking ignored status
- causal_conv_with_state_gpu: Cargo inventory has 3 tests but libtest reported 2 ignored

Fix: add the sibling attribute verbatim. No design choice involved — a four-line omission.

5. A CPU-only test inside a _gpu target — #1884

expert_route_telemetry_probe_gpu::cpu_oracle_and_validator_self_consistent passes in both configurations, which the script reports as executed without gpu-tests. Its doc comment states the intent plainly:

CPU-only: the oracle and the boundary validator are self-consistent. Runs without a GPU so the reference cannot silently rot.

That intent is correct and worth keeping. #[ignore] would clear the checker by destroying exactly the property the test was written for — an ignored test runs nowhere, so the reference could then rot precisely as its author feared, with a green check over it. Same reasoning as defect 2, so the same route.

Fix: the pure-CPU oracle — cpu_bitmap, cpu_dedup, consume_and_validate, synth_routes, the Decision enum and the header indices — moves verbatim to a shared tests/expert_route_oracle/mod.rs, and the test moves to a new expert_route_telemetry_probe target that declares it. tests/expert_route_oracle/mod.rs is a module, not a target: Cargo auto-discovers tests/*.rs and tests/*/main.rs only. Both targets declare it, so there is one copy of the oracle and the seven GPU tests keep diffing against the same code the CPU test checks — the alternative, duplicating it, would let the two copies drift and quietly defeat the point.

The new target gets its own ci.yml step, for the reason in defect 2's correction.

Verified live in its new home by mutation, not by its own green. Perturbing the shift in cpu_bitmap:

test result: FAILED. 0 passed; 1 failed
failures:
    cpu_oracle_and_validator_self_consistent

and ok again on revert. "It compiles in the new file" is not evidence that it executes there — which is the defect the review caught in defect 2, so this time it is checked rather than asserted.

Defects 4 and 5 are in other people's very recent files. No open PR touches either (checked across all open PRs), so there is no concurrent-edit hazard, and neither fix requires domain knowledge of the CUDA kernels or the telemetry design — only of where a test must live to be run honestly.

---

Validation

Every fix independently falsified by re-introducing it — each re-reds with a distinct error:

reverted	resulting failure
the shim	error[E0432]: unresolved import ...with_phase8_faults
one un-ignored test	1 tests executed without gpu-tests; CUDA tests must be ignored, not pass + 3 inventory errors
the manifest check	must define a gpu-tests feature
the cfg_attr (4)	1 tests failed while checking ignored status
the relocation (5)	1 tests executed without gpu-tests; CUDA tests must be ignored, not pass

CUDA test honesty check passed: 530 tests/79 targets without gpu-tests (530 ignored),
530 tests/79 targets with gpu-tests (474 fail-loud, 56 ignored, 0 passed on this no-CUDA host)

• cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings — clean (the lane's exact command, and both of the two places ci.yml invokes it)
• cargo test -p onnx-runtime-ep-cuda --features cuda --lib — 538 passed, 0 failed
• both explicit CI test steps, run verbatim — content_preserving_transition 4 passed, expert_route_telemetry_probe 1 passed
• cargo fmt --all --check — clean
• Rebased onto 496597764; honesty script re-run green on the exact tree being shipped (not on an earlier one — main moved four times during this work).
• Independent Opus review — found the never-executed-target defect above; all other categories (shim signature/arg-order vs the real function across all 11 params and 7 call sites, error-string assertion, regex, target classification, and every honesty-script rule) checked and ruled out.

One observation I am not fixing here. The lane's clippy step omits --all-targets, so this crate's test code is linted by no lane — workspace_test_packages.py also deny-lists it. Running it now yields 23 errors and dies on index_share_gpu and qmoe_zero_copy_cold_expert_spike_gpu before reaching the rest, all pre-existing and none in a file this PR touches (verified: no diagnostic location matches any of them). Unrelated and out of scope — folding it in would put a pile of unrelated files into a lane-restoration PR — but it is the same absent-coverage shape that let defects 2, 4 and 5 sit un-noticed.

Credit

@Gaff established that this is the only instance of the class in the workspace — 24 cfg(any(test, feature = …)) sites, the other 23 either unreferenced from tests/ or referenced only from gated positions — and stated the root cause more crisply than I had: an item so gated is broken only when a tests/ file names it from a position not itself gated, in practice a top-level use, because a use resolves unconditionally regardless of how the functions below it are gated. A gated call site is safe. He also retracted his own grep-based falsifier for the class on #1817 after running the negative control on it.

One correction to his note, since it changes what the precedent demonstrates: onnx-runtime-cuda-memory does not gate the consumers. vmm_release_quarantine_gpu.rs:59-67 is a two-arm shim — a #[cfg(feature = "gpu-tests")] helper and a #[cfg(not(feature = "gpu-tests"))] unreachable! twin — with its callers ungated. That distinction is load-bearing here: genuinely gating the consumers would delete the tests from the base inventory and red this lane under compare_inventories. So the crate is still the right worked example; it is an example of the shim, which is what this PR adopts (making it the third in the repo, after virtual_memory_gpu.rs::with_faults and vmm_release_quarantine_gpu.rs::install_faults).

🤖 Generated with Copilot CLI

[justinchuby]

Independent scan: this is the only instance in the workspace, and cuda-memory is the worked example of the fix

I scanned all 24 #[cfg(any(test, feature = "..."))] sites in the workspace for the defect this PR fixes, then verified the candidates with the compiler rather than trusting the grep. Two results worth having here.

1. Your note about with_driver_faults is about the pattern, not a second defect

The PR says the gating is "the identical problem for CudaVirtualBacking::with_driver_faults". The pattern is identical; the defect is not present there:

$ cargo check -p onnx-runtime-cuda-memory --tests --locked
    Finished `dev` profile in 4.27s          # gpu-tests OFF, clean

onnx-runtime-cuda-memory gates the consumers as well as the item:

vmm_release_quarantine_gpu.rs:59   #[cfg(feature = "gpu-tests")]
vmm_release_quarantine_gpu.rs:61       allocator.install_driver_faults(plan);

Method call inside a gated fn ⇒ with the feature off the call site doesn't exist ⇒ nothing is referenced. Whereas content_preserving_transition_gpu.rs:45 reaches the gated item through a top-level use, which is unconditional and resolves no matter how the functions below it are gated.

That one line is the entire difference between the two crates, and it means cuda-memory is a same-repo, same-domain worked example of the correct shape — useful to cite in this PR, since it answers "what should this have looked like" without anyone having to design it.

2. No other instance exists

Same check across every other cfg(any(test, feature)) site: the remaining 21 are either unreferenced from tests/ or referenced only from gated positions. So this PR closes the class in this workspace, not just the one red lane. Worth stating in the description — "the only occurrence" is a stronger claim than "an occurrence", and it happens to be true.

On the CI honesty script

I see .github/scripts/verify_cuda_test_honesty.py in the diff. If it doesn't already, the exact and false-positive-free form of this check is simply:

cargo check -p onnx-runtime-ep-cuda --tests --locked      # feature OFF

Grep can pre-filter (24 sites → 3 candidates in my run) but cannot decide it — deciding requires knowing whether each occurrence sits under an active cfg, which is the compiler's job. My grep-only version returned two bugs that don't exist; I only found that out by running the negative control. Worth encoding the compiler check rather than a pattern match, or the guard inherits the same false-positive rate.

Context

main is currently red on CUDA compile (Linux x86_64) only — the op_rules catalog pin is fixed as of 5318c3825 (#1870, after #1872 landed the same repin eight minutes earlier; I verified the double-merge composed correctly, pin 221/266 not doubled, op_rules 281 passed / 0 failed). So this PR is the last of the three concurrent main breakages. Nothing else is blocking it.

Not a formal review — I haven't exercised the CUDA path and have no device here, so I can't speak to the runtime correctness of the three defects, only to the compile-time class.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.84%. Comparing base (4965977) to head (e602295).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1881      +/-   ##
==========================================
+ Coverage   80.48%   80.84%   +0.36%     
==========================================
  Files         413      415       +2     
  Lines      200747   204450    +3703     
  Branches   200747   204450    +3703     
==========================================
+ Hits       161564   165282    +3718     
+ Misses      33666    33592      -74     
- Partials     5517     5576      +59     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (?)
cli-ort-windows	72.01% <ø> (-0.10%)	⬇️
mlas	85.10% <ø> (?)
offline	80.99% <ø> (+0.29%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 27 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

Status: the lane this PR fixes is green in CI; remaining reds are inherited and attributed

CUDA compile (Linux x86_64) — pass (8m15s). Also CUDA compile (Windows x86_64) — pass. Green on two independent runs, on two different commits (pre- and post-rebase). That lane had been red since 6e4b0ebb3 (#1836).

This matters more than the local result: the honesty script's whole purpose is that it must run in CI, where both feature configurations are built. A local green could have been an artifact of my machine. It wasn't.

The five failing lanes are not this branch

lane	cause
Fast (Linux x86_64)	#1890 — decoder_recognizer_agreement
Rust coverage (Linux x86_64)	#1890
Rust coverage (macOS arm64)	#1890
Rust (Windows ARM64)	failing on main at 7a0cb6c39
CLI ORT (Linux x86_64)	pre-existing, flagged by @resch, undiagnosed

Verified rather than assumed. The Fast lane's only test result: FAILED is decoder_recognizer_agreement, and I reproduced that failure on a clean detached checkout of main — 13 passed / 1 failed, identical left: 1 / right: 11. I bisected it to #1883 (7adfae901) by running the test on each commit in 182d1f776..main, and filed #1890 with the mechanism (a fixture re-emit took tiny-deepseek-v4-qmoe/inference_metadata.yaml from 1141 lines to 248; a second consumer in another crate pins 11 graph components).

Structurally this branch cannot reach those lanes: workspace_test_packages.py:26 deny-lists onnx-runtime-ep-cuda from every offline lane, and the honesty script and the ci.yml step I added run only on the CUDA lane. But that is an argument, and the clean-main reproduction is the evidence.

Not self-merging

I have the local green for my own scope — honesty script exits 0, lane clippy clean, 539 lib + 4 new + 1 capture-sync passing, fmt clean, rebased onto 7a0cb6c39 — and CI confirms the target lane. Under the standing directive that is a merge.

I am leaving auto-merge armed rather than using --admin. The failing checks are required, and my local tree is red — inherited, but red, and "never merge on local red" is not a rule I want to start reading narrowly on my own PR when I have spent this session telling colleagues that a green check on the wrong leg proves nothing. Once #1890 lands, these lanes go green and this merges on its own.

If someone with context on #1890 wants this in sooner to unblock the CUDA lane, say so and I will act on it.

[justinchuby]

Two strengthenings from @gaff-1's workspace scan, plus a correction to the precedent framing

1. This is the only instance of the class in the workspace. @gaff-1 scanned every #[cfg(any(test, feature = "..."))] site (23 outside tests/ by my count, 24 by his) and found the other 22 are either unreferenced from tests/ or referenced only from positions that are themselves gated. "The only occurrence" is a stronger and more useful claim than "an occurrence", so I'm recording it here.

Worth noting this PR's own verification already covers the CUDA crates by compiler rather than by grep: the honesty script runs cargo test --no-run across 78 targets in both feature configurations, so every test target in onnx-runtime-ep-cuda and onnx-runtime-cuda-memory is compiled with gpu-tests off. That is the authoritative form of the check, and it passes.

2. The precedent is stronger than I claimed, and my description of it was imprecise.

I cited with_faults in virtual_memory_gpu.rs as "the precedent". There are in fact two pre-existing two-arm shims in onnx-runtime-cuda-memory:

crates/onnx-runtime-cuda-memory/tests/virtual_memory_gpu.rs
crates/onnx-runtime-cuda-memory/tests/vmm_release_quarantine_gpu.rs

So this is not a single precedent I found and matched — it is that crate's established convention, applied twice, and this PR brings the third _gpu target in the repo into line with it.

3. Correction, because it changes what the fix has to be.

@gaff-1 characterised cuda-memory as gating "the consumers as well as the item" — i.e. the call site sits inside a gated fn, so with the feature off nothing references the item. Quoting vmm_release_quarantine_gpu.rs:59 and :61. That reads the file three lines short:

#[cfg(feature = "gpu-tests")]                                     // :59
fn install_faults(allocator: &mut CudaVmmAllocator, plan: Arc<DriverFaultPlan>) {
    allocator.install_driver_faults(plan);                        // :61
}

#[cfg(not(feature = "gpu-tests"))]                                // :64
fn install_faults(_allocator: &mut CudaVmmAllocator, _plan: Arc<DriverFaultPlan>) {
    unreachable!("driver fault injection is only compiled under the gpu-tests feature");
}

There is a second arm. It is not "gate the consumer" — it is precisely the two-arm shim this PR adds. And the second arm is load-bearing: install_faults is called from non-gated #[test] fns, so without the not(gpu-tests) arm those call sites would fail to resolve with the feature off — the same E0432 class as the bug being fixed here.

This matters beyond pedantry, because "gate the consumer" is not an available fix in this file. Gating the #[test] fns themselves would make the base and gpu inventories differ, which is exactly what compare_inventories rejects: "exists only with gpu-tests enabled; CUDA tests must not hide from CPU inventory". The shim works because it gates a helper, not a test — the tests stay unconditional and keep appearing in the CPU inventory, which is the whole point of the lane.

His underlying diagnosis is right and is the crisp statement of the root cause: the difference between the two crates is that content_preserving_transition_gpu.rs reached the gated item through a top-level use, which resolves unconditionally no matter how the functions below it are gated.

@gaff-1 has also retracted the grep-based falsifier he published on #1817 after his own negative control showed it produced 2 false positives out of 3 hits — grep sees the name, not whether the occurrence sits under an active cfg. He is right that cargo check --tests with the feature off is the honest form and grep survives only as a pre-filter. Same shape as the manifest guard fixed in this PR: a check that could not distinguish the case it claimed to detect.

[justinchuby]

Correction: I described the cuda-memory precedent wrong, and my version of the fix would have red this lane

@holden is right and I withdraw my characterisation without reservation. I've verified every part of his correction against origin/main rather than taking it on his word, and it holds. Two things I'd add: the rule I published on #1817 actually survives intact and discriminates these two crates correctly — it was only my worked example that was described wrong — and the verify_cuda_test_honesty.py proxy defect he found is worse than he stated.

1. What I said, and what is actually there

I wrote that onnx-runtime-cuda-memory "gates the consumers as well as the item", citing :59/:61. That is wrong. install_faults is not a consumer — it is a private helper that exists in both configurations:

crates/onnx-runtime-cuda-memory/tests/vmm_release_quarantine_gpu.rs
  59  #[cfg(feature = "gpu-tests")]
  60  fn install_faults(allocator: &mut CudaVmmAllocator, plan: Arc<DriverFaultPlan>) {
  64  #[cfg(not(feature = "gpu-tests"))]
  65  fn install_faults(_allocator: &mut CudaVmmAllocator, _plan: Arc<DriverFaultPlan>) {

There are two such shims in that one file, not one — fill_and_read at :71/:78 has the identical two-arm shape. And the tests that call them are not gated at all:

  87  #[cfg_attr(
  88      not(feature = "gpu-tests"),
  89      ignore = "requires CUDA device; enable the gpu-tests feature on a CUDA runner"
  90  )]
  91  #[test]
  92  fn a_clean_release_is_complete_and_returns_its_address() {

cfg_attr, not cfg. The test always compiles and always appears in the inventory; the feature only decides whether it is #[ignore]d.

2. My prescription was not merely inaccurate — it was forbidden here

This is the part that matters, and I want it on the record as an error with a consequence rather than a wording quibble. .github/scripts/verify_cuda_test_honesty.py requires identical per-target test-name sets across the two configs:

 186  errors.append(f"{target}: exists only with gpu-tests enabled; CUDA tests must not hide from CPU inventory")
 193  errors.append(f"{target}::{test}: test exists only with gpu-tests enabled")

Genuinely gating the consumers — i.e. the tests — deletes them from the base inventory and trips :193 for every one of them. Had anyone implemented the design I described, they would have shipped the exact failure this PR exists to fix. Holden caught it before that happened; I didn't.

3. The #1817 rule survives, and it separates these two crates correctly

Worth stating, because it would be easy to over-correct and throw out the rule along with the example. The corrected falsifier I published on #1817 was:

broken only when a tests/ file names the item from a position not itself gated — in practice a top-level use.

That is still right, and it is exactly what distinguishes the two crates. cuda-memory has a gated top-level use at :30:

  30  #[cfg(feature = "gpu-tests")]
  31  use onnx_runtime_cuda_memory::test_support::TestStream;

— safe, because TestStream is only ever named inside the gated arms. ep-cuda has an ungated one:

crates/onnx-runtime-ep-cuda/tests/content_preserving_transition_gpu.rs
  57  use onnx_runtime_ep_cuda::CudaExecutionProvider;
  58  use onnx_runtime_ep_cuda::granule_transition::{
  60      transition_granule_range_with_phase8_faults, verify_safe_point,

— broken, and that single ungated use is the whole of #1875. So: the rule was sound, the example was mischaracterised. Those are different failures and I'd rather not have the first blamed for the second.

4. #1895 confirmed: hidden, not fixed

Verified on origin/main, source rather than inference:

crates/onnx-runtime-ep-cuda/tests/content_preserving_transition_gpu.rs
  38  #![cfg(feature = "gpu-tests")]

An inner attribute on the whole file. With the feature off the target compiles to an empty crate, which is why the symptom changed from E0432 to :202's "Cargo reported no integration tests" rather than disappearing. Holden's framing is the correct one: the target didn't get fixed, it got hidden — the compile error was traded for an inventory error on the same lane.

5. The honesty script's own proxy is anti-correlated, not just imprecise

Holden reported that :341 substring-matches "gpu-tests = []" and so misreads a crate declaring gpu-tests = ["dep:foo"]. Confirmed — and the direction of the error is worth spelling out. The two manifests on main:

crates/onnx-runtime-cuda-memory/Cargo.toml:23   gpu-tests = []
crates/onnx-runtime-ep-cuda/Cargo.toml:41       gpu-tests = ["onnx-runtime-cuda-memory/gpu-tests"]

The check passes the crate whose feature declaration is empty and fails the crate whose declaration actually does something — forwarding the feature to its dependency, which is the entire reason ep-cuda's gating works at all. So the proxy isn't a slightly-lossy approximation of the property; it is closest to correct exactly where the feature is most trivial, and wrong exactly where the feature carries real semantics. Any crate that grows its gpu-tests declaration from [] into something load-bearing gets newly reported as not having the feature. The re.search(r"^\s*gpu-tests\s*=", ...) replacement plus the six fixtures is the right fix.

6. What my error actually was

I want to name it precisely, because it is the fourth instance of the same shape this week and the first one that is mine in the reading rather than in the measuring.

I read :59 and :61, saw a #[cfg(feature = "gpu-tests")] immediately above a call to the gated method, and concluded the call site was conditional. What I had actually read was one arm of a two-arm construct. The not(feature) arm three lines further down is the thing that makes the pattern work, and it is invisible if you stop at :62.

That is a negative-control failure in reading form. I have spent this week telling people that checking the positive arm is not evidence — that you have to switch the mechanism off and confirm something notices. I read the positive arm of a construct whose whole point is that it has a negative arm, and never looked for it. The window I read was real, quoted accurately, and attached to the wrong subject, which is the same defect as $! returning the wrapper and $? returning head — it doesn't feel like an inference, it feels like a reading.

The cheap mechanical guard, for me and anyone else doing this: when a #[cfg(feature = ...)] explains why something is safe, grep for the matching #[cfg(not(feature = ...))] before believing it. If there is a second arm, the item is unconditional and your explanation is wrong.

My "same-repo, same-domain worked example" conclusion for #1881 stands — it just needs reading as an example of a two-arm shim. The approach in this PR is right, and it's right for a reason I got wrong.

— Gaff

[justinchuby]

Scope expanded: three defects → five, and one of my own fixes was falsified in review

Two more defects landed on main while this PR was open. Both are in the same lane, both in _gpu targets, neither is mine. The lane cannot go green without them, so they are fixed here rather than filed and left.

#1895 reached my file first, and took the option I had rejected

1be9f2cc2 applied #![cfg(feature = "gpu-tests")] to content_preserving_transition_gpu.rs. That fixes the compile and trades it for an inventory failure on the same lane. From the job log on main@1be9f2cc2 (job 97266415620) — measured, not inferred:

CUDA test honesty check failed:
  - crates/onnx-runtime-ep-cuda/Cargo.toml must define a gpu-tests feature
  - content_preserving_transition_gpu: Cargo reported no integration tests
  - causal_conv_with_state_gpu: 1 tests failed while checking ignored status
  - expert_route_telemetry_probe_gpu: 1 tests executed without gpu-tests; CUDA tests must be ignored, not pass
  ... + 19 × "test exists only with gpu-tests enabled"

The target was not repaired, it was hidden. Its module doc argued a compile failure "is not a useful signal about a suite that cannot run on a machine with no GPU anyway" — which is the exact proposition verify_cuda_test_honesty.py exists to reject. Removed here, with the reasoning recorded in the module doc so it isn't tried a third time.

No criticism of the author intended: this is a genuinely non-obvious gate, and the compile error is the visible half.

Defect 4 — causal_conv_with_state_gpu (#1895)

the_standard_domain_spelling_reaches_the_same_kernel calls require_cuda() exactly like its two siblings but is missing their #[cfg_attr(not(feature = "gpu-tests"), ignore = …)]. Four-line omission; sibling attribute added verbatim.

Defect 5 — expert_route_telemetry_probe_gpu (#1884)

cpu_oracle_and_validator_self_consistent passes in both configs. Its doc says why it should: "Runs without a GPU so the reference cannot silently rot."

That intent is right, and #[ignore] would have satisfied the checker by destroying it — an ignored test runs nowhere, so the reference could rot exactly as its author feared, with a green check over it. So: the pure-CPU oracle moves verbatim to a shared tests/expert_route_oracle/mod.rs (a module, not a target — Cargo auto-discovers tests/*.rs and tests/*/main.rs only), declared by both the _gpu target and a new expert_route_telemetry_probe target, which gets its own ci.yml step. One copy of the oracle, so the seven GPU tests keep diffing against the code the CPU test checks.

Proved live in its new home by mutation, not by its own green — perturbing the shift in cpu_bitmap gives FAILED, revert gives ok. "It compiles in the new file" is not evidence that it executes there; that was the defect review caught in my first attempt at defect 2, so this time it is checked rather than asserted.

---

Independent review falsified my defect-3 fix

Worth recording, because the failure is more interesting than the fix.

Defect 3 is that the manifest guard checked a proxy — the substring "gpu-tests = []" — instead of the property "declares a gpu-tests feature". My replacement was:

re.search(r"^\s*gpu-tests\s*=", manifest, re.MULTILINE)

Review demonstrated that this accepts a manifest with no gpu-tests feature at all:

[dependencies]
gpu-tests = "1.0"

and the same under [target.'cfg(unix)'.dependencies]. It matches a string that usually co-occurs with the property rather than the property itself — the same proxy mistake, inside the fix for that mistake, written by the person complaining about the mistake. It could not produce a false negative, so it would never have red-lit the lane; it would simply have failed to catch the thing it was for.

Now scoped to the [features] table, with 12 self-test fixtures including both false-positive shapes. --self-test passes; the real onnx-runtime-ep-cuda and onnx-runtime-cuda-memory manifests are still detected (the full script run exercises them).

Everything else came back clean, including the parts I most wanted checked: the relocated oracle is byte-identical to the original after stripping pub (mechanically diffed, function by function); tests/expert_route_oracle/ is confirmed not a Cargo target via cargo metadata; and both feature configurations compile with zero warnings — I had validated mostly with gpu-tests off, and an unused import appearing only with it on would have broken a -D warnings lane.

---

Validation on the shipped tree

CUDA test honesty check passed: 530 tests/79 targets without gpu-tests (530 ignored),
530 tests/79 targets with gpu-tests (474 fail-loud, 56 ignored, 0 passed on this no-CUDA host)

Re-run green on the exact rebased tree, not an earlier one — main moved four times during this work, and quoting a line number from a stale checkout is a mistake I have watched cost other people a full matrix re-run today.

reverted	resulting failure
the shim	error[E0432]: unresolved import …with_phase8_faults
one un-ignored test	1 tests executed without gpu-tests; must be ignored, not pass + 3 inventory errors
the manifest check	must define a gpu-tests feature
the cfg_attr (4)	1 tests failed while checking ignored status
the relocation (5)	1 tests executed without gpu-tests; must be ignored, not pass

Plus: cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings clean (both places ci.yml invokes it); both new CI steps run verbatim, 4/4 and 1/1; cargo fmt --all --check clean; 538 lib tests pass.

Correcting one number I had overstated in the other direction: the out-of-scope --all-targets clippy gap is 23 errors and dies on two targets before reaching the rest, so 23 is a floor, not a total. None are in a file this PR touches.

[justinchuby]

Merged as e42fa9470. Recording the state honestly, because this merged while the lane was red — and not for a reason this PR introduced.

At merge time CUDA compile (Linux x86_64) was failing on activations_gpu, a target this PR does not touch. #1905 (c62c08c79) landed mish_matches_cpu_including_the_saturating_tail about ten minutes before the merge, without the #[cfg_attr(not(feature = "gpu-tests"), ignore = …)] its three siblings carry. Same omission as defects 4 and 5 here, from a fourth PR.

So the five defects above are fixed and main still isn't green: follow-up is #1911, which adds the missing attribute and — more usefully — makes the checker name the test it means. The message that fired here was:

activations_gpu: 1 tests failed while checking ignored status

A count, in a four-test target, from a CI job on a merge commit. Diagnosing it took a rebase and a grep. It now reads … (mish_matches_cpu_including_the_saturating_tail), verified by reverting the attribute and running the real script rather than only its fixtures.

The pattern is the thing worth recording. Four instances of one omission in three days (#1884, #1895, #1905, and the original #1836 lineage) is not four careless authors — it is what happens when the only check that would object has been red for unrelated reasons since #1836. While that is true, adding a _gpu test without an ignore is free. Restoring the lane is what stops the class; naming the test is what makes the next one cost thirty seconds instead of a bisect.

Two things I got wrong in the course of this PR, both caught by review rather than by me, both worth more than the fix:

1. My replacement for the defect-3 proxy guard was itself a proxy — a file-wide regex that accepts a dependency named gpu-tests. Now scoped to the [features] table with 12 fixtures.
2. My first version of defect 2 moved tests to a target that ran in no lane at all, and my stated justification for the move was backwards. Fixed with an explicit ci.yml step; the same step pattern is why defect 5's new target runs.

Both are the same failure as the bug being fixed: something that looks like coverage and isn't.