test(onnx): fix magika idle-thread test on macOS with ORT 1.31

[rominparekh]

Description

tests/magika_idle_threads.rs (added in #4189) fails on macOS with ONNX Runtime 1.31. Building the magika session leaves 8 extra threads against a fixed allowance of 2.

The cause is not a thread pool. ORT 1.31 registers Core ML as a built-in EP that claims the Apple NPU/GPU (microsoft/onnxruntime#31975). The ort 2.0.0-rc.12 crate sets the MAX_EFFICIENCY (= PREFER_NPU) EP selection policy on every SessionBuilder::new(). So magika's session now runs on Core ML / Apple Neural Engine (ANE), and building it starts 5–7 ANE service threads that are idle, not spinning. The fixed allowance of 2 can't account for that.

The test now builds a pool-free control session (magika::Session::builder().with_intra_threads(1)) first and uses the threads it adds as the allowance. The control goes through the same EP selection, so whatever the EP starts cancels out, while a reintroduced intra-op pool still shows up on top.

As a precaution, Rust CI now also runs cargo test on macos-26. It only ran on Ubuntu, which is why this slipped through.

Closes #4191

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Performance improvement
• Code refactoring (no functional changes)

Changes Made

• crates/headroom-core/tests/magika_idle_threads.rs: measure the allowance against a pool-free control session instead of a fixed 2, with comments explaining the ORT 1.31 Core ML/ANE cause. Test-only; no production code changes.
• .github/workflows/rust.yml: turn the test job into a matrix with ubuntu-latest and macos-26, using the same ONNX Runtime setup step. Ubuntu keeps its test (ubuntu) check name and runs the same steps as before. macOS runs only cargo test; cargo fmt and cargo clippy stay on Ubuntu. fail-fast: false, so one OS failing doesn't cancel the other.

Testing

• Rust: cargo fmt --all -- --check, cargo clippy --workspace -- -D warnings, cargo test --workspace with ORT_DYLIB_PATH set so the magika tests actually run
• Unit tests pass (pytest): only the pre-push ci-precheck-python subset was run (177 passed, 4 skipped); the full suite was not
• Linting passes (ruff check .): no Python files changed; pre-commit hooks passed
• Type checking passes (mypy headroom): N/A, no Python changes
• New tests added for new functionality (updated regression test)
• Manual testing performed (see Real Behavior Proof)

Test Output

cargo test -p headroom-core --test magika_idle_threads, with the #4187 fix (with_intra_threads(1)) in place and with it temporarily reverted:

# macOS arm64, ORT 1.31.0, before this PR (fixed allowance of 2)
magika session left 8 extra OS thread(s) running ... FAILED
# macOS arm64, ORT 1.31.0, this branch
test result: ok. 1 passed; 0 failed      (6/6 repeat runs)
# macOS arm64, ORT 1.29.0, this branch
test result: ok. 1 passed; 0 failed

# #4187 fix reverted, this branch: the test still catches the pool
macOS, ORT 1.31:                      12 extra vs 9 allowed -> FAILED
macOS, ORT 1.29:                       7 extra vs 2 allowed -> FAILED
Linux (Docker aarch64), ORT 1.31:      7 extra vs 1 allowed -> FAILED

Full Rust suite on macOS with ORT 1.31.0 (the same command the new macos-26 job runs):

ORT_DYLIB_PATH=.../libonnxruntime.1.31.0.dylib cargo test --workspace
68 suites, 1658 passed, 0 failed

Pre-push make ci-precheck on this branch:

ci-precheck-python: 177 passed, 4 skipped
✅ ci-precheck PASSED — safe to push.

Real Behavior Proof

• Environment: macOS 26 arm64 (16 cores), Rust 1.95, pip install onnxruntime==1.31.0 and 1.29.0 dylibs via ORT_DYLIB_PATH; Linux via Docker rust:1.95 (aarch64) with pip onnxruntime 1.31.0.
• Exact command / steps: the reporter's repro from test(onnx): magika_idle_threads fails on macOS with ORT 1.31 (Neural Engine threads, not an intra-op pool) #4191 (pip install 'onnxruntime==1.31.0', export ORT_DYLIB_PATH=<capi/libonnxruntime.*.dylib>, cargo test -p headroom-core --test magika_idle_threads), on main and on this branch; then the same test with the fix(onnx): magika detector session still spins idle cores (#2540 covered only the Python factory) #4187 with_intra_threads(1) fix temporarily reverted, on macOS 1.31, macOS 1.29 and Linux 1.31. To find the cause, a raw ort probe built the magika model with intra_threads=1 under the default policy and under with_auto_device(PreferCPU), counting threads with ps -M and listing the EP devices.
• Observed result: fails on main with ORT 1.31 (8 extra vs 2) and passes on this branch, including the reporter's exact steps. With the fix(onnx): magika detector session still spins idle cores (#2540 covered only the Python factory) #4187 fix reverted it fails on all three setups (12 vs 9, 7 vs 2, 7 vs 1), so it still guards fix(onnx): magika detector session still spins idle cores (#2540 covered only the Python factory) #4187. Probe: EP devices are CPU, CoreML/NPU and CoreML/GPU. The default policy adds 6–7 threads and takes ~420 ms to build; PreferCPU adds 0 and takes ~12 ms.
• Not tested: the new macos-26 CI job on GitHub runners (it runs on this PR), ORT 1.30.0 at runtime, x86_64 Linux, Intel macOS, and low-core macOS hosts. Details:
  • ORT 1.30.0: the 1.30 vs 1.31 difference comes from the ORT source diff and release notes, plus the reporter's data.
  • Low-core macOS hosts: with only 2–4 cores the control allowance can absorb a small pool, so the guard is weaker there.

Runtime Rollout Safety

• Rollout-managed feature(s): none.
• Minimum rollout channel: N/A
• Stable/default behavior changed: no. The PR changes only the test and CI; runtime code is untouched.
• Kill switch / disable path: N/A
• Unsafe override required: no.
• Qualification impact: adds one macOS CI job (only when Rust paths change).
• Rollback path: revert this PR.

Review Readiness

• I have performed a self-review
• This PR is ready for human review

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (none needed)
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I did not edit CHANGELOG.md — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)

Screenshots (if applicable)

N/A

Additional Notes

• Not fixed here: on macOS with ORT ≥ 1.31, headroom's magika detection runs through Core ML on the ANE instead of the CPU EP. That comes from ort's default policy; nobody chose it. Visible cost: a one-off ~400 ms session build plus 5–7 idle threads. Per-call latency and output parity were not measured. Pinning it to CPU needs with_auto_device(PreferCPU), which magika 1.1.0's Builder doesn't expose. An environment-level CPU EP does not override the policy. This is raised on test(onnx): magika_idle_threads fails on macOS with ORT 1.31 (Neural Engine threads, not an intra-op pool) #4191 as a possible follow-up issue on the magika crate.
• Option 1 from the issue (count CPU-burning threads instead) was rejected: the fix(onnx): magika detector session still spins idle cores (#2540 covered only the Python factory) #4187 pool only spun briefly after each inference, so idle CPU was ~0% with or without it and the check wouldn't catch a regression.
• This change was developed with GitHub Copilot CLI assistance; all measurements above were run locally by me.

---

[github-actions]

PR governance

This PR follows the template and is marked ready for human review.

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🔍 Test-only fix conflicts with contribution policy

The contribution policy excludes test/CI-only PRs for known main failures. This change updates only the regression test and CI for #4191; maintainer approval of an exception remains necessary.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[KennethWKZ]

Thanks for this. Closed #4193 in favour of it. Three notes:

1. The new test (macos) job never reached the magika test. It failed earlier on ccr_backends::in_memory_get_refreshes_idle_ttl (an entry accessed within its idle window must stay alive, left None). That test touches the entry every 60 ms against a 120 ms idle TTL, and the macOS runner's sleep overshoot is enough to expire it. Cargo stops at the first failing test binary, so the log ends at tests/ccr_backends.rs and magika_idle_threads never runs (job log). Giving that test more slack on macOS, or running cargo test --workspace --no-fail-fast on the macOS leg, would let this PR's own check show up there.

2. The allowance includes the control's persistent threads. allowance is everything the control adds while it's alive. The threads it leaves running after drop(control) (the 2–4 your comment mentions) are already in the second before, though. If the magika session reuses those rather than starting its own, the comparison gets that many threads of extra headroom, which is part of why macOS has only 12 vs 9 to spare. Logging settled_thread_count() after drop(control) would show whether the allowance could be just the threads that go away with the session.

3. The Core ML default. I'm happy to open the issue on the magika crate asking for a pass-through for with_auto_device / the EP selection policy on its Builder, so headroom can pin magika to the CPU if maintainers want that. Say the word, or tell me if you'd rather track it here first.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[chopratejas]

Hi @rominparekh - thanks for the PR - there is a test failing - can you help fix it?

[JerrettDavis]

Reviewed head b245ce2, its inline feedback and documented local measurements. Current main is an ancestor, formatting passes, and the Ubuntu Rust test job succeeded. The control-session approach correctly distinguishes provider-created idle threads from a simple fixed allowance, with the documented low-core limitation retained.

The newly added macOS job is not yet qualified: job 115651049966 in run 38502033892 failed in ccr_backends::in_memory_get_refreshes_idle_ttl at line 173 (None versus Some("hot entry")). Cargo stopped before executing magika_idle_threads, so the hosted run does not yet verify the regression this PR fixes. This is an existing test outside the PR's code changes; I am not attributing its failure to the new allowance calculation.

Please ensure the macOS workflow executes the targeted Magika integration test even when another suite fails, and retain the full-suite check. The unrelated idle-TTL failure needs maintainer qualification or a justified test-fixture repair before this can be green for human merge review. No contributor CI action is requested. Coordinate with #4193's CPU-time alternative so we retain a single effective regression guard and avoid conflicting replacements of this test.

[JerrettDavis]

Reviewed updated head 345480f against the previous requested changes and discussion. The macOS workflow now runs the dedicated Magika integration test even if another full-suite binary fails, while retaining the full-suite check. The idle-TTL fixture uses wider timing margins and still proves both sliding retention beyond the original expiry and eventual expiry after inactivity; runtime storage logic is unchanged.

The previously missing hosted evidence is now present: both Ubuntu and macOS full Rust suites passed, and the dedicated macOS Magika test executed and passed without skipping. Its log reports start=2, control alive=4, control dropped=4, and zero extra Magika threads against allowance 2. The documented low-core limitation of a thread-count allowance remains; this is not a universal proof against every small spinning pool.

Current main is an ancestor, GitHub reports conflict-free, and current-head checks are terminal with 50 successful check runs plus a successful commit status, including Docker native E2E. Formatting and whitespace checks passed locally. Existing comments, the earlier CI failure and the test-only contribution-policy note were considered in this maintainer review.

Approved for final human merge review, replacing the earlier validation hold. No contributor runtime logic changed, contributor CI action requested, or merge performed.

[rominparekh]

Thanks for the review and approval @JerrettDavis . Can this PR be merged?