Repository navigation
fix(engine): correct the macOS struct statvfs FFI layout - #1586
Merged
Merged
Conversation
`statvfs_capacity` hand-declared a `#[repr(C)] struct StatVfs` that matches 64-bit glibc only. On macOS `fsblkcnt_t`/`fsfilcnt_t` are 4 bytes, not 8, and there is no trailing `f_spare` member, so the declaration was a 112-byte struct laid over a real 64-byte one: `f_blocks` and `f_bfree` were read as one fused u64, the product overflowed `checked_mul`, and the disk reported as unmeasurable. Measured on arm64 macOS by compiling against <sys/statvfs.h>: sizeof = 64, fsblkcnt_t = 4, and f_fsid at offset 40 rather than the 64 the old declaration implied — so three fields past the block counts were also wrong, not just the six count fields. Gate the widths per platform. The non-macOS branch expands to exactly the types present before, so Linux codegen is unchanged. Arithmetic moves to `u64::from`, which widens correctly at either width. Adds compile-time `size_of` guards (macOS 64, 64-bit Linux 112) so a wrong layout is a build failure rather than silent runtime garbage, plus field-offset tests. Verified the guard fires by reverting the width locally. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…s FFI The module header justified the hand-written `unsafe extern "C"` declarations with "the workspace carries no `sysinfo`/`libc` today". The `libc` half of that is no longer true — it is a dependency of onnx-runtime-ep-cpu, onnx-runtime-virtual-memory, onnx-runtime-tracer and onnx-genai-cli. Left as written, the next reader either inherits a false premise or concludes the whole hand-rolled approach is unmotivated. State the actual reason: `libc` exists in the workspace but deliberately is not a dependency of this crate, because taking one on for a single FFI struct is not worth it. Also name the cost that choice carries — hand-declared #[repr(C)] types have to be checked against each platform ABI rather than assumed — and point at the compile-time guards that now discharge it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 20, 2026
main advanced past this branch's base (#1578, #1580, #1583..#1585, #1588, #1491, #1571, #1592). 23 files changed on main, 3 of them also touched here. Conflict resolution, and the audit behind it: kernels/matmul_nbits.rs -- taken from main wholesale. This branch's blob is byte-identical to main's blob at 3542ae7 (#1580): the only "change" on our side was restoring main's probe after an earlier merge dropped it (the Tier 2 audit). main has since carried that same file forward through #1584/#1585, so our content is a strict ancestor of main's and taking main loses nothing. Verified by blob hash, not by reading the diff. .github/workflows/ci.yml -- auto-merged. Result equals main plus the four `-p onnx-runtime-memory-api` package selections this stack adds. main's two new `cargo fetch --locked` steps are both present (counted). pipeline/decoder_component.rs -- auto-merged. Result equals main plus the two `ProcessMemoryManager` lines this stack adds. main's #1592 additions (supports_argmax, step_argmax, decode_argmax_with_step_inputs, captured_step_input_greedy_supported) all survive at main's reference counts. The other 20 files main changed are byte-identical to origin/main in the merge result, checked by blob hash for every one rather than by eye. Verified: cargo check --workspace --all-targets and cargo check -p onnx-genai-engine --features cuda,native-backend clean; cargo fmt --all --check clean; onnx-genai-engine native-backend suite 559 passed / 2 failed / 1 ignored, the two failures being the same macOS statvfs pair that fails on main and is fixed by #1586. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c80f8522-983c-47f7-8241-2155a823aabe
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1586 +/- ##
========================================
Coverage 80.77% 80.78%
========================================
Files 381 381
Lines 173645 173786 +141
Branches 173645 173786 +141
========================================
+ Hits 140264 140386 +122
- Misses 28491 28505 +14
- Partials 4890 4895 +5
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
The previous fix kept the hand-written `#[repr(C)] struct StatVfs` and gated the field widths per platform. That works, but it makes this crate responsible for every platform's ABI: it needed two type aliases, a cfg'd-out `f_spare`, two compile-time size guards and a table of field offsets, and it still only covered macOS and Linux -- every other unix kept the widths that were wrong on macOS. `libc` already defines `struct statvfs` per target, and is already in this crate's dependency graph via memmap2 and tokio, so declaring it adds no new package (the Cargo.lock delta is one line). Use it and delete the hand-rolled layout: there is no longer a layout here to get wrong, on any unix. This also drops the `useless_conversion` error that broke the Rust quality lane on Linux, where `u64::from` on an already-64-bit count is an identity. The casts are now plain `as u64` under one function-level allow, which is what the surrounding code already does for `f_frsize`/`f_bsize`. Verified: engine lib suite 430 passed; the CI clippy command (--locked --all-targets --features native-backend -- -D warnings) clean; and the module compiled standalone for x86_64-unknown-linux-gnu against libc's Linux definition, so the Linux path is exercised rather than only reasoned about. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 20, 2026
Picks up #1586, #1601 and #1602. No conflicts; Cargo.lock was the only file both sides touched and it auto-merged. Audited the same way as the previous two syncs rather than by reading the diff: every main-only file verified byte-identical to origin/main by blob hash, and for Cargo.lock every line main added over the merge-base confirmed present in the result, with this stack's three onnx-runtime-memory-api entries still intact. #1586 is the reason this sync matters beyond staying mergeable. It corrects the macOS `struct statvfs` FFI layout, which is what made `platform_capacity::tests::disk_capacity_is_measured_for_the_working_directory` and `engine::tests::an_explicit_byte_limit_is_honored_without_a_device_query` fail on every local run of this branch. Those two were the only failures in the crate suite here, and every "559 passed / 2 failed" figure reported against this PR was that pair. They are expected to be gone now; verified in the following run rather than assumed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c80f8522-983c-47f7-8241-2155a823aabe
justinchuby
added a commit
that referenced
this pull request
Aug 22, 2026
…ees were destroyed (#1717) Two decision records existed **only on this machine** — written into `.squad/decisions/inbox/` but never committed. Recovered here. The inbox README already names this exact failure mode as the reason the directory was made tracked on 2026-07-29: > **Drops survive worktree deletion.** Previously the inbox was gitignored, so drops existed only on the machine that wrote them. Removing a worktree before Scribe ran destroyed unmerged drops — this cost real records more than once. The policy fixed the *gitignore* half of the problem. It cannot fix the half where a drop is written and simply never `git add`ed, which is what happened to both of these. I found them while cleaning up 11 stale worktrees; one was inside a worktree I was about to delete. ## What is being recovered **1. `Copilot-phase-5-process-memory-manager-ownership-boundarie.md`** Written in the `justinchuby-phase-5-process-memory-manager` worktree (PR #1426, now closed as superseded by #1579). Copied out immediately before that worktree was removed. Verified it is not a duplicate before adding it: ``` $ git grep 'Phase 5 process memory manager ownership boundaries' origin/main -- .squad (no match) ``` So the Scribe never merged it — it was not sitting redundantly next to an existing entry in `.squad/decisions.md`. Content-wise this one is not merely historical. It records the Phase 5 ownership split: `ProcessMemoryManager` owns process quota, canonical authority/view mapping, holder identity, context transaction gates, loss broadcast and settlement coordination, while **governors keep budget policy, holders keep victim choice, and EP contexts keep stream/copy/commit/decommit/fence/deferred-release ordering**. It also records two invariants that are easy to get wrong later — quarantine never infers a refund from allocation length, and confirmed context termination is the sole device-loss discharge boundary. That boundary is still load-bearing after #1579, and **#1186 is still open precisely on the provider/context-pinning half of it**, so losing this note would have removed context from work that has not happened yet. **2. `coordinator-criterion-failure-is-structural-not-probabilistic.md`** Written in this worktree. References #1579, #1586, #1533, #1600 and three sessions that converged on it independently. ## Scope No code changes; two files under `.squad/decisions/inbox/`. Per the inbox contract these are transient — the Scribe merges them into `.squad/decisions.md` and deletes the drops. ## Worth considering separately Nothing detects an uncommitted drop. Both of these survived by coincidence, because a human happened to be cleaning worktrees and looked at `git status` before running `git worktree remove --force`. A pre-removal check for untracked files under `.squad/decisions/inbox/` would close that gap; I have not written one here. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c80f8522-983c-47f7-8241-2155a823aabe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
platform_capacity.rshand-declared a#[repr(C)] struct StatVfsto callstatvfs(3). That declaration matches 64-bit glibc only. On macOSfsblkcnt_t/fsfilcnt_tare 4 bytes, not 8, and there is no trailingf_sparemember — so a 112-byte struct was laid over a real 64-byte one.f_blocksandf_bfreewere read as one fusedu64, the garbage overflowedchecked_mul, that returnedNone, and the disk reported as unmeasurable.Measured, not assumed
Compiled and ran a C program against
<sys/statvfs.h>on arm64 macOS:The damage was wider than the fused block counts:
f_fsid/f_flag/f_namemaxwere also at wrong offsets (the old declaration impliedf_fsidat 64; the real offset is 40), andf_sparedoes not exist on macOS at all. Onlyf_bsize/f_frsizewere correct. A fix driven purely by the reported symptom would have left two of these in place.Fix: use
libc's definition, don't hand-roll oneThe struct is now
libc::statvfs.libcwas already in this crate's dependency graph viamemmap2andtokio— both direct dependencies — so declaring it explicitly adds no new package. TheCargo.lockdelta is literally one line:crates/onnx-genai-engine dependencies: + "libc",libcdefines the layout per target, so there is no longer a layout in this crate to get wrong — on macOS, on Linux, or on the other unixes an earlier revision of this PR left with the broken widths.This is a net deletion: two type aliases, a cfg'd-out
f_spare, two compile-time size guards and a field-offset table all go away.Verification
Before (reverse-verified against the unmodified tree)
After
Both pass. Full crate suite:
430 passed; 0 failed; 1 ignored.Executed
cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings— now clean for this crate.platform_capacity.rshas no crate-internal dependencies, so it was built standalone forx86_64-unknown-linux-gnuagainstlibc's real Linuxstruct statvfs, under-D warnings. Clean. A full-workspace Linux cross-build is not possible here (onnx-genai-ort-sysneeds a Linux C sysroot for bindgen), so this is the module, not the whole crate.cargo fmt --checkclean.Not executed
Linux test execution, and Windows anything. Those need CI.
On the CI failures for this PR
mainis currently red on its own, so "CI is red" is not by itself attributable here. Run 32388200098 ismainat3b3e5f79— this PR's exact merge-base — and it already fails the same job and same step for:CUDA compile (Linux x86_64)→Verify CUDA test inventory and skip honesty(tracked as matmul_nbits_marlin_numerics silently skips instead of failing loud, so the CUDA honesty guard is red on main #1589)CUDA compile (Windows x86_64)→Clippy CUDA execution providerRust (Windows ARM64)→Test cross-platform offline cratesRust coverage (Windows x86_64)andRust coverage (macOS arm64)→Test cross-platform offline crates with coverageThose coverage/offline lanes do not include
onnx-genai-engineat all —python .github/scripts/workspace_test_packages.py cargo-args offline-cross-platformdoes not list it — so this PR cannot be their cause. Pre-existing, not addressed here.Rust qualitywas genuinely mine. It passed onmainat3b3e5f79and failed here. Cause: an earlier revision usedu64::from(buf.f_blocks), which on Linux is au64→u64identity and tripsclippy::useless_conversionunder-D warnings— precisely the Linux-only blind spot flagged as unverified. Reproduced locally by forcing the Linux widths, and fixed: thelibcversion uses plainas u64casts under one function-level allow, matching what the surrounding code already does forf_frsize/f_bsize.