test(bench): count the nodes on our EP instead of asking whether any are

[justinchuby]

What

The dispatch benchmark's EP-assignment check was a membership test:

assert!(info.ops_on_our_ep().contains(&case.op),
        "{}: not assigned to this EP, refusing to report a ratio", case.name);

It answers "is at least one node of this type ours". Every per-node figure in #1077 comes from running a chain of N nodes and dividing by N, so the question that actually matters is "are all N of them ours" — and contains cannot tell 100 from 1.

This replaces it with a count:

assert!(ours.len() == case.expected_ep_nodes && theirs.is_empty(), ...);

expected_ep_nodes is a new field on the bench case struct: depth for a chain, 1 for the single-node cases.

What it found on its first run

grid_identity_10_static: expected all 10 node(s) on this EP,
got 1 ours ["Identity"] and 0 elsewhere []

Ten Identity nodes go in; one arrives. ORT collapses the redundant chain during session build, so our EP never sees the other nine. (That is the observation. Which ORT pass does it is not checkable from this tree, and the comment no longer claims it.)

The row is kept, with its expectation set to the observed 1, as a pin on that folding behaviour — deleting it would throw away the only place in the repo that notices ORT does this. Its comment says plainly that it must not be read as a depth-10 point.

Was anything published wrong?

No — and that is a measured answer, not a hoped-for one. Per-case audit of the whole grid, one process per case so a mismatch could not mask the rest:

exact match	16 of 17
mismatch	grid_identity_10_static only
grid_relu_100_tiny, grid_relu_1000_tiny	exact

The two relu_*_tiny rows are the cases behind every Ir/node number in the #1077 ledger, including the −8.3% in #2240. Their denominators are now verified rather than assumed.

The in-code per-node probe was already honest: report_probe divides by ops_on_our_ep().len(), not by the case definition, with a comment saying why. The hazard was a human dividing a total by a nominal depth — which is exactly what the offline callgrind harness does with DEPTH=100.

Mutation proofs

Both under the mechanical host lock (#1806).

	mutation	result
M2	expected_ep_nodes: depth + 1	healthy grid_relu_10_tiny FAILS, listing all ten Relus — the count is compared, not merely required non-zero. Incidentally proves that case really has 10 nodes on our EP.
M3	restore contains-only semantics	grid_identity_10_static PASSES again and prints a ratio — the old check could not see a folded chain. The new one is load-bearing, not decorative.

An earlier M3 attempt excised the block and did not compile, so its run used a stale binary; that run proved nothing and was redone as a one-line semantic swap. Recording it because a mutation that silently runs the wrong binary is the same class of error this PR is about.

Validation

check	result
cargo fmt --check	clean
cargo clippy --release --all-targets	clean
grid set, one process	17/17 rows, rc=0
default bench set (NXRT_MM_BENCH=1, no grid)	44/44 rows, rc=0
full onnx-runtime-ep-cpu-plugin suite	green (plugin_ort_e2e 58 passed / 1 ignored)

The default-set run is there because the new assert is stricter than the old one and gates that set too — the reviewer pointed out my first audit only covered the opt-in grid.

Review

Independent adversarial Opus review, read-only, no cargo (shared host under lock). Verdict APPROVE WITH NITS. It independently verified every construction site's node count against the generated ONNX TextFormat, that no path reports a ratio without passing the assert, and that theirs.is_empty() cannot fire spuriously under disable_cpu_ep_fallback=1. Two findings, both addressed:

• SF-1 — my comment claimed the row "had been reporting a per-node cost divided by ten." Not supported: the in-code probe divides by the real count, and this row was never the source of a published number. Comment now states the hazard, not a history of misreporting.
• N-1 — "ORT's L1 optimiser eliminates redundant Identity nodes" attributes a mechanism I cannot verify from this tree. Comment now states the observation.

SF-2 (the default-set gap) is closed by the run in the table above.

Scope

Test-only. No production file changes, no behaviour change, no runtime cost — the bench is #[ignore]d and the assert runs once per case at session build.

The same membership-vs-count vacuity exists in assert_ops_assigned_to_our_ep, used by ~20 conformance tests. Deliberately not touched here: those are correctness tests where partial assignment is sometimes legitimate, and converting them needs its own audit. Noted as a follow-up candidate.

Refs #1077

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.04%. Comparing base (26dbc0f) to head (8024925).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2250      +/-   ##
==========================================
+ Coverage   80.71%   81.04%   +0.33%     
==========================================
  Files         433      433              
  Lines      221629   221629              
  Branches   221629   221629              
==========================================
+ Hits       178889   179623     +734     
+ Misses      36815    36086     -729     
+ Partials     5925     5920       -5     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (ø)
cli-ort-windows	72.10% <ø> (ø)
mlas	86.02% <ø> (+0.12%)	⬆️
offline	81.17% <ø> (+0.34%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 11 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.