build(ci): pin the Rust toolchain to 1.98.0

[justinchuby]

Closes the root cause behind the 1.98.0 CI incidents tracked in #1600.

What the failure was

Every blocking gate in ci.yml runs under -D warnings — either as RUSTFLAGS or as a trailing -- -D warnings on clippy. CI resolved stable at job time, so when runner images rolled forward to rustc 1.98.0 (88d9e12ae 2026-08-18), every lint that release made warn-by-default became an instant failure on code that was clean the day before. No commit to blame, and a contributor on 1.97.0 could not reproduce it — clippy 0.1.97 does not carry those lints at all, so it does not even print them.

Four consecutive incidents, all the same cause:

PR	Lint	Code from
#1604	cargo fmt drift, manual_slice_fill, needless_late_init	assorted
#1609	collapsible_if ×5 in qmoe.rs + ep-cpu lints	#1602
#1615	collapsible_if in runtime.rs	#1612
#1603	chunks_exact_to_as_chunks, 12 crates	assorted

Each fixed symptoms; none could prevent the next. Formatting is the self-perpetuating one — an unpinned rustfmt means whoever formats next re-flows the file back the other way, so the tree oscillates.

How I reproduced it

I read the version out of a failing job log rather than guessing, then installed it:

rustup toolchain install 1.98.0    # -> rustc 1.98.0 (88d9e12ae 2026-08-18), byte-identical build hash

That is what turned "unreproducible locally" into reproducible, and it is what made #1609 and #1603 diagnosable at all.

What I changed

rust-toolchain.toml (new) — channel = "1.98.0", profile = "minimal", components = ["clippy", "rustfmt", "llvm-tools-preview"]. No targets key: Rust (Windows ARM64) runs natively on windows-11-arm and needs no cross target, and scripts/check_cross_compile.sh adds its own.

ci.yml — the eight setup steps collapse from rustup toolchain install stable --profile minimal --component … + rustup default stable to a bare rustup toolchain install, which reads the file and installs the declared components. The version= cache-key output is unchanged. Policy comment rewritten.

ci.yml, changes job — separate bug, fixed while here. It was gated on github.event_name == 'pull_request' || 'push', with a comment claiming schedule and workflow_dispatch runs would skip only that job and still get full CI. They did not. Every job below is needs: changes with a plain if: (no always()/!cancelled()), and GitHub skips a job whose needs dependency was skipped regardless of its own if:. So the gate skipped the entire workflow: the nightly cron and every manual dispatch reported success without running a single check — a green with no verdict behind it, which is the same class of problem this PR is about. The classify step already leaves docs_only=false for any event it does not diff, so removing the gate makes the documented behaviour real.

How I verified it

All local, on this branch, with my rustup default left at 1.97.0 so the file is doing the work:

Negative control — the pin is what changes the resolution:

without rust-toolchain.toml:  rustc 1.97.0   clippy 0.1.97   rustfmt 1.9.0 (2d8144b788)
with    rust-toolchain.toml:  rustc 1.98.0   clippy 0.1.98   rustfmt 1.9.0 (88d9e12ae1)

clippy 0.1.97 → 0.1.98 is precisely the gap that made these failures invisible to contributors.

Both gates, run with bare commands (no +1.98.0), which is the point — a 1.97.0 contributor now gets CI's behaviour by default:

• cargo fmt --all -- --check → clean
• the verbatim 30-package clippy gate from ci.yml ~326, including the trailing -- -D warnings → exit 0

Component auto-install: llvm-tools was absent from my 1.98.0 install and rustup fetched it on first use inside the repo, so dropping the per-job --component flags is safe. Confirmed end-to-end with cargo llvm-cov --locked -p onnx-runtime-cpuinfo → exit 0, which is the one job whose component requirement changed.

rustup override semantics, empirically checked (I had this wrong earlier and said so on #1600): rustup component add and rustup target add respect the directory pin — I expected them to hit the default toolchain and silently break the coverage and cross-compile jobs. They do not; both landed on 1.98.0. Explicit +toolchain still beats the file (rustc +stable -Vv → 1.97.0 inside the repo), so miri's cargo +nightly is unaffected.

YAML: all workflow files parse; changes confirmed to have no if: key.

What I could not verify

• Everything about the CI runners themselves. I am on macOS aarch64. That a Linux or Windows runner materializes 1.98.0 from this file, and that the cache key still resolves correctly there, is only checkable in CI. This PR's own run is the oracle.
• The workflow_dispatch/schedule fix. I verified the mechanism by reading the gating (all 8 downstream jobs are needs: changes with plain if:), but I have not observed a dispatch run execute the matrix. Worth triggering one after merge to confirm.
• Other workflows. audit.yml, publish.yml, publish-ep-plugins.yml, wheels.yml still say rustup default stable. They inherit the pin anyway — any cargo run inside the repo resolves through the file — so their behaviour is already correct and their explicit install is merely redundant. I left them rather than widen the blast radius onto release workflows. One exception worth flagging: benchmark.yml uses dtolnay/rust-toolchain@stable, which sets RUSTUP_TOOLCHAIN — that env var overrides the file, so benchmark is genuinely not pinned. It is non-blocking, but it is a real gap, not an oversight.

Related, not fixed here

No CI job runs clippy on macOS. rust-coverage matrixes macOS but only installs clippy (~line 458) and never runs it; all six clippy-running jobs are Linux/Windows. That means #1609's macOS-only fixes in accelerate_gemm.rs and matmul.rs (both #[cfg(target_os = "macos")]) were structurally unverifiable by CI. I ran CI's exact clippy package list on macOS under 1.98.0 → exit 0, so this is a prevention gap rather than an outstanding defect. Filing separately.

Conflict risk with #1579

None. This PR touches only .github/workflows/ci.yml and a new root file. No overlap with crates/onnx-runtime-ep-cpu, crates/onnx-runtime-ir, or crates/onnx-genai-engine/src/{decode,native_decode}/.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.53%. Comparing base (2f0d08a) to head (9b3d116).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1620      +/-   ##
==========================================
+ Coverage   81.49%   81.53%   +0.03%     
==========================================
  Files         383      383              
  Lines      179257   179257              
  Branches   179257   179257              
==========================================
+ Hits       146086   146157      +71     
+ Misses      28231    28161      -70     
+ Partials     4940     4939       -1     

Flag	Coverage Δ
cli-ort-linux	82.60% <ø> (ø)
cli-ort-windows	82.11% <ø> (ø)
mlas	85.19% <ø> (ø)
offline	81.43% <ø> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.