libstore: Enforce Mach-O code-signature validity when bytes enter the store - #4
Closed
ak2k wants to merge 3 commits into
Closed
libstore: Enforce Mach-O code-signature validity when bytes enter the store#4ak2k wants to merge 3 commits into
ak2k wants to merge 3 commits into
Conversation
ak2k
force-pushed
the
darwin-signature-guard
branch
from
July 3, 2026 19:56
60f9c42 to
e6e63ba
Compare
Owner
Author
|
Real-binary validation against broken cache.nixos.org artifacts (branch HEAD, on x86_64 Linux, the fixup binary built with ASan): I re-fetched 10 genuinely-broken signed Mach-O files from cache.nixos.org spanning every signature class in the failing population, ran the branch's
All 8 repairable files: every code slot matches its page content after repair (verified independently), the file length is preserved to the byte, and only hash-slot bytes change — a multiple of the hash width, confirming the repair is byte-surgical. The two Developer-ID binaries are correctly refused with zero bytes touched. Note on method: an initial pass using The engine also ran here compiled with AddressSanitizer, so the parser was exercised on these real 25–105 MB binaries with no memory error. |
…atures When registering build outputs, the daemon sometimes rewrites store path hashes inside the output's files: when an output being built was already present in the store at build start, its scratch path is a synthesised fallback path that must be substituted with the final one after the builder exits, and content-addressed outputs need their self-references rewritten once the final hash is known. If the rewritten bytes sit inside a Mach-O binary carrying LC_CODE_SIGNATURE, the substitution invalidates the signature's page hashes, and the macOS kernel kills the binary with SIGKILL at first page-in. The corruption was silent: the build succeeds and the registered output is broken. This is the mechanism behind the recently reported darwin startup failures of fish (nixpkgs issue 507531) and one of the mechanisms behind Nix issue 6065. It has also reached cache.nixos.org — directly evidenced for ffmpeg, whose cached build log references fallback paths, and consistent with the stale page contents of several cached Haskell outputs. Before applying a rewrite, scan the output for regular files that both carry a Mach-O code signature and contain one of the hashes about to be substituted, and fail the build with an error naming the affected files and the already-present store paths whose deletion allows a clean rebuild. CMS-signed files (Developer ID) are called out separately since no re-signing without the original identity can ever fix those. The check also fires under --check, replacing the spurious "may not be deterministic" failure previously reported for signed binaries. The new macho-signature-rewrite-check setting controls the behaviour: refuse (default), warn (diagnose but register the broken output — the previous behaviour plus a diagnostic), ignore (previous behaviour). Detection is purely content-based, so cross-builds of darwin binaries on Linux are covered too; the Mach-O constants are vendored rather than taken from Apple headers for the same reason. Refusing by default is a deliberate behaviour change with a visible blast radius: --check / --rebuild of any signed darwin binary that was previously reported as spuriously non-deterministic now fails with this error instead, and content-addressed cold builds of self-referential signed Mach-O files fail loudly rather than registering silently broken outputs. This adds a Mach-O / code-signature parser to the daemon, over bytes produced by untrusted builders. It is deliberately detection-only: read-only, no writes, every read bounds-checked against the buffer, walk lengths capped (fat_arch and SuperBlob counts, file size — files over the limit are refused as unverifiable, not waved through), fat slices validated by offset and size, and unit-tested against malformed inputs. Repair — which needs substantially more parsing — is out of scope here and belongs outside the daemon's privileged context. Refusing at the rewrite is a finer-grained sibling of bailing out at build start when only some outputs could be substituted: the coarse form is simpler and catches non-signature rewrite damage too, but cannot cover content-addressed cold builds, where nothing is present at build start and the damaging rewrite is unconditional.
Detection alone (the previous commit) leaves the user of a multi-output rebuild with an error and a manual deletion step, and --check of a signed self-referential binary with no way to complete. This adds the repair: a new macho-signature-repair-hook setting, defaulting to the Nix-shipped tool `nix __fixup-macho` (registered like __build-remote), which recomputes exactly the stale signature page hashes in place. The repair is deterministic: only hash slots whose stored value disagrees with the page contents are rewritten, and every other byte — the linker-signed flag, the original page size, the identifier — is preserved, so the same input bytes always yield the same output bytes. That property is what --check and content-addressing require, and what re-signing with codesign(1) cannot provide (it switches page size and clears linker-signed). Both SHA-256 and SHA-1 CodeDirectories are recomputed when present, since the kernel validates every one at page-in. The daemon does not run the repair itself. At the detect point it chowns the affected files to the build user and execs the hook with that user's privileges (the diff-hook pattern) and a minimal environment; the complex parse of untrusted bytes thus happens outside the daemon's own context. A nonzero exit from the hook fails closed to the detection error. Setting macho-signature-repair-hook to an empty string disables repair entirely, restoring plain detect-and-refuse. The hook's exit status says it ran, not that the signatures are now valid: the tool skips what it cannot process (an unsupported CodeDirectory hash type, a malformed header), and a custom hook may do less than it claims. So after the repair the daemon re-invokes the hook with --check (same privileges) and registers the output only if every signature verifies — the hook contract, documented in the setting, is that --check modifies nothing and exits 0 when all signatures are valid, 2 when any is stale or cannot be verified. In check mode the tool counts a signature it cannot verify as a failure for the same reason: exit 0 promises all signatures are valid, and "could not parse" is not "valid". Not repairable, and still refused: CMS/Developer-ID signatures (the certificate chain commits to the directory hash; only the original identity can re-sign), files too large to have been inspected, and the self-reference rewrite of a content-addressed output — there the hashed pages contain the output's own path, which is itself a function of those pages, so no consistent signature value exists (issue 6065). The repair scope is thus exactly the damage the daemon's own rewrite causes; breakage introduced by build tools before registerOutputs is out of scope and stays visible. The hook runs between the rewrite and the metadata canonicalisation, so the NAR hash always covers the repaired bytes and ownership and permissions are restored over the hook's intermediate state. For fixed-output and impure derivations, whose outputs sit in a daemon-private 0700 temporary directory at this point, the directory is chowned along with the files; it is transient and deleted after registration. The functional tests assert the full matrix: default hook repairs (codesign --verify passes, output runs and prints the rewritten path), empty hook refuses, failing hook fails closed, a hook without --check support fails closed, a repairable-looking file whose signature the tool cannot process (unsupported hash type) is refused after the re-check rather than registered broken, --check completes with only the genuine LC_UUID nondeterminism, and a direct dual-oracle exercise of the tool (corrupt a signed byte, codesign rejects, repair, codesign accepts, byte content intact).
…d at rest The build-door check (the previous commits) covers damage the daemon itself causes when registering outputs. But broken signed binaries mostly reach users through substitution: the artifact was already broken where it was built — by the producing daemon, a build tool (bun, electron-builder, install_name_tool), or a broken upstream release — and the substituting machine registers it verbatim. This completes the check at the two remaining doors. Substitution: the new macho-signature-verify setting (default ignore) checks substituted paths in LocalStore::addToStore between restorePath and registerValidPath. A cheap in-daemon scan finds signed Mach-O files; the page-hash verification itself runs in a child process (the repair hook with --check) with the privileges of a build user acquired from the same pool as builds, falling back to the daemon's own uid in single-user mode. Modes: warn names the path and turns mystery SIGKILLs into a download-time diagnostic; refuse fails the substitution (falling back to a local build where possible); repair recomputes the stale hashes before registration, after which the path's NAR hash no longer matches the substituter's advertised one, so its signatures are dropped and it is registered unsigned. Content-addressed paths and CMS-signed files are never repaired and fall back to warn. A path is never accepted on evidence that doesn't exist: a Mach-O file too large to parse is refused under refuse (warned otherwise) rather than passed on the check child's silence, and after a repair the path only counts as repaired if a re-check comes back valid — either way the recorded NAR hash describes the bytes actually on disk, which a partial repair may have changed. Under refuse the path is never modified at all, unlike the build door, whose refuse mode repairs its own rewrite damage first; an operator who wants neither must also empty macho-signature-repair-hook (both settings document this). At rest: the new 'nix store fixup-macho' command repairs broken signatures in paths already registered. It never modifies files in place — with auto-optimise-store, a file may be hard-linked into other store paths, and an in-place write would corrupt every path sharing the inode. Instead the contents are copied, repaired in the copy, verified, swapped in, and the path's NAR hash updated in the database via the new LocalStore::replaceStorePath (which no existing primitive provides: verifyStore only fills in missing hashes and repairPath restores the original contents). The swap window is the same as repairPath's. Content-addressed paths are skipped; a copy whose signatures still fail the post-repair check is discarded, not swapped in. darwin gotcha encoded in replaceStorePath: renaming a read-only directory fails with EACCES on APFS, so owner-write is temporarily restored around the renames; timestamps and permissions are re-canonicalised after the swap. The functional tests manufacture a genuinely broken cached artifact (rewrite under warn with repair disabled, published to a file:// cache) and assert all four modes at the substitution door, the at-rest sweep including dry-run, idempotence, and batch resilience (a CMS path early in the batch does not prevent later repairs), the partial-repair outcome (one supported and one unsupported CodeDirectory: NAR hash updated, signatures dropped, path reported unrepaired, database consistent), and the unverifiable cases — an oversized Mach-O refused at the door and failed by the tool's own --check.
ak2k
force-pushed
the
darwin-signature-guard
branch
from
July 4, 2026 20:53
9745aae to
21119ac
Compare
ak2k
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.