libstore: Enforce Mach-O code-signature validity when bytes enter the store

[ak2k]

Note

This PR was restructured in July 2026 following the discussion below. The original shape — the repair helper linked into libstore and run unconditionally by the daemon — is gone; the repair now runs exclusively in a privilege-dropped child process, wired as a hook setting with a Nix-shipped default payload, alongside a detect-and-refuse mode and an opt-in substitution-time check. Comments before this point reference the earlier shape. The repair mathematics (slot-only recompute, dual-CD, fat containers) is unchanged from what was verified against real cache.nixos.org binaries below.

Originally based on the direction of #14999 by @andrewgazelka; the comparison table below explains where the approaches differ and why.

Fixes the root cause of NixOS/nixpkgs#507531 (fish and the wider multi-output family): the daemon's rewrite damage is now repaired and re-verified before registration, so affected packages build correctly by default.

Addresses #6065 by converting it from silent corruption to a loud, actionable refusal; the full fix (CA hashing modulo signature slots) is out of scope and described under "CA endgame".

Motivation

On darwin, DerivationBuilderImpl::registerOutputs calls RewritingSink to substitute scratch-path bytes in build outputs. When an output being built was already in the store at build start, its scratch path is a makeFallbackPath-synthesised stand-in, and RewritingSink rewrites those scratch-path bytes to the final path after the builder exits. The substitution is byte-level and has no knowledge of Mach-O code signatures, but Apple's ld ad-hoc-signs every binary at link time with the linker-signed flag set in LC_CODE_SIGNATURE. The signature covers the very bytes that were just rewritten, so one or more page hashes in the CodeDirectory are stale after the rewrite. At first page-in, the macOS kernel SIGKILLs the process with cs_invalid_page.

This is the root cause of NixOS/nixpkgs#507531 (fish on nixpkgs-darwin fails to start) and one mechanism behind #6065 (open since 2022 against CA derivations). It has also reached cache.nixos.org — directly evidenced for ffmpeg, whose cached build log references fallback paths, and consistent with the stale page contents of several cached Haskell outputs. A mechanism classification of the full failing-cache population (101 derivations, 6 families) is on this thread; the hash-rewriting family is the one only Nix can fix.

What this PR does now

Following the discussion here — @edolstra's rewrite-hook direction, @emilazy's disable-fallback-paths proposal, and @xokdvium's bail-out framing — the PR makes a valid Mach-O signature a property Nix preserves when bytes enter a local store: checked where the NAR hash is already checked, repaired where permissions are already canonicalised, by one tool that never runs as root. Three commits, one per enforcement point:

1. Build door: detect and refuse (macho-signature-rewrite-check, default refuse)

Before applying a rewrite, the output is scanned for regular files that both carry a Mach-O code signature and contain one of the hashes about to be substituted. Under refuse (the default) the build fails with an error naming the affected files and the already-present store paths whose deletion allows a clean rebuild; warn and ignore restore the previous behaviour with and without a diagnostic.

This is @emilazy's "disable fallback paths on macOS" scoped precisely — the build only fails when the rewrite would actually break a signature — and @xokdvium's build-start bail-out at finer granularity (the coarse form is simpler and catches non-signature rewrite damage too, but cannot cover content-addressed cold builds, where nothing is present at build start and the damaging self-reference rewrite is unconditional; happy to add the coarse form as well if preferred).

• Fires under --check too, replacing the spurious "may not be deterministic" failure previously reported for signed binaries.
• On CA builds, the self-reference rewrite of a signed binary refuses on every cold build — intentionally converting Content-addressed derivation fails to build on aarch64-darwin #6065 from silent corruption to a loud error (see "CA endgame" below).
• Detection is content-based (constants vendored, no Apple headers), so Linux cross-builds of darwin binaries get the same guard; hence no darwin- prefix on the settings.
• The in-daemon part is deliberately detection-only: read-only over a memory-mapped view (a large binary costs a few touched pages, not heap), every read bounds-checked, walk lengths capped, files beyond what the signature format can cover (4 GiB, the 32-bit codeLimit bound) treated as unverifiable rather than waved through, fuzz-tested against malformed input and ASan/UBSan-clean on x86_64-linux. This is the small parser that has to live where the decision is made; everything that writes runs elsewhere:

2. Build door: repair via a privilege-dropped hook (macho-signature-repair-hook)

Refusal alone leaves users of a multi-output rebuild with an error and a manual deletion step, and --check of a signed self-referential binary unable to complete. The new hook setting defaults to a Nix-shipped tool (nix __fixup-macho, registered like __build-remote) that recomputes exactly the stale page-hash slots in place. This is the rewrite-hook mechanism proposed above, with a default payload: the daemon chowns the affected files to the build user and execs the hook with that user's privileges (the diff-hook pattern) and a minimal environment, so the complex parse of untrusted bytes happens outside the daemon's context — and because the default payload ships with Nix rather than being declared by derivations, it also covers every already-published nixpkgs revision, which no derivation-declared hook can reach.

• The repair is deterministic: same input bytes, same output bytes, only stale slots touched — what --check and content-addressing require, and what re-signing with codesign(1) cannot provide (it switches page size and clears linker-signed; see the fix(darwin): handle code signatures for CA derivations #14999 comparison below). Both SHA-256 and SHA-1 CodeDirectories are recomputed, since the kernel validates every one at page-in and dual-CD binaries exist in the cache.
• "Repaired" is a verified claim, not the hook's word: after the repair the daemon re-runs the hook with --check (same privileges) and registers the output only if every signature verifies. The tool's --check counts a signature it cannot verify — unsupported hash type, malformed CodeDirectory, oversized file — as a failure, because "could not parse" is not "valid". The --check contract (modifies nothing; exit 0 = valid, 2 = stale or unverifiable) is documented in the setting for custom hook authors.
• Never repaired, still refused: CMS/Developer-ID signatures (only the original identity can re-sign) and CA self-references (the hashed pages contain the output's own path, which is a function of those pages — no consistent value exists).
• Hook failure, a hook that skips what it cannot process, or a hook that does not implement --check: all fail closed to the refusal.

3. Substitution door + at rest (macho-signature-verify, nix store fixup-macho)

Broken signed binaries mostly reach users through substitution — broken where they were built, by whatever broke them (the producing daemon, bun --compile, electron-builder, install_name_tool, or a broken upstream artifact; see the classification comment for the family breakdown). macho-signature-verify (default ignore) checks substituted paths before registration: warn turns mystery SIGKILLs into a named download-time diagnostic; refuse fails the substitution (falling back to a local build where possible); repair fixes the path before registration — its NAR hash then no longer matches the substituter's advertised one, so the substituter's signatures are dropped and the path is registered unsigned. The verification child runs with the privileges of a build user acquired from the same pool as builds; pool exhaustion fails the substitution rather than running the child as root.

nix store fixup-macho [--dry-run] covers what is already inside a store. It never repairs in place — auto-optimise-store hard-links files across paths, and an in-place write would corrupt every sharing path — but copies, repairs, verifies, and swaps, updating the database through a new LocalStore::replaceStorePath (no existing primitive replaces a valid path's non-null NAR hash).

Asymmetry worth knowing: under refuse the substitution door never modifies a path, while the build door's refuse repairs the daemon's own rewrite damage before deciding. An operator who wants no Nix-modified signed binaries at all must also empty macho-signature-repair-hook (documented in both settings).

What this deliberately does not do

• CA endgame: the only true fix for CA self-references is extending CA hashing to be modulo signature slots exactly as it is already modulo self-ref strings — a store-path-changing decision far beyond this PR. Until then, loud refusal at both doors is the honest bridge.
• Producer hygiene: nixpkgs re-signing (e.g. autoSignDarwinBinariesHook, already used by 54 packages) and Hydra substitution fixes reduce how often the guards fire, and are worth pursuing — but only the consumer-side check makes the guarantee, and the Haskell family shows why: those packages use the signing hook and still break, because the daemon's rewrite runs after fixupPhase. Complementary tracks, not competing ones.
• Laundering: detect modes surface problems, never hide them. Repair modes are explicit opt-in on the user's own store — except the build door's default, which repairs only damage the daemon itself just caused, and verifies the result.

Differences from #14999

Aspect	#14999 (darwin-codesign.cc)	This PR
Call-site coverage	CA visitor only	Shared rewriteOutput lambda → both IA and CA
Structure preservation	No — re-signing diverges from the linker's original signature (three differences below), so a repaired rebuild never matches a clean build	Yes — only stale hash slots change; the result is what the linker produced, with corrected hashes
CodeDirectory flags	linker-signed cleared by codesign -s -	Preserved
Special-slot layout	codesign adds special slots (non-minimal CodeDirectory)	Preserved
Default page size	16 KiB on arm64 (fixable via codesign -P 4096)	Preserved (4 KiB)
Execution context	Forks /usr/bin/codesign from the daemon	Nix-shipped tool exec'd as the build user; daemon does read-only detection only

Evidence

The repair engine is the same mathematics that was verified end-to-end before the restructure; the blocks below carry over and were re-validated against the current branch.

Empirical scope scan across the darwin channels

A scope scanner at ak2k/nix-507531-scope walks each darwin channel's cache and re-verifies every Mach-O slice's page hashes. As of 2026-07-04 it reports 436 failing slices across three channels (nixpkgs-25.11-darwin 55, nixpkgs-darwin 121, nixpkgs-unstable 260) in 47 distinct packages; live numbers in REPORT.md (auto-updated daily).

The mechanism classification on this thread breaks that population into families. The hash-rewriting family is what the build door prevents and repairs; the other families (bun --compile self-rewriting, electron fuse flips, install_name_tool, upstream-shipped breakage) are producer bugs that the build door does not touch — for those, the substitution door's warn/refuse/repair is the user-facing mitigation at download time, regardless of which producer broke the binary. Developer-ID-signed files remain unrepairable by anyone but the signer, at every door.

The failing list is a lower bound, not a census: packages matching the trigger variants pass the scan only because the worker that built the current cached version happened not to hit scratchPath != finalPath.

Real-binary validation with an independent oracle (re-run at the current head)

10 genuinely broken signed Mach-O files fetched from cache.nixos.org, spanning every failing signature class (thin linker-signed, fat32 single- and multi-arch, dual-CD SHA-1+SHA-256, 56-slot Bun single-file executables, Developer-ID CMS), run through nix __fixup-macho built with ASan on x86_64-linux at this PR's head commit, and validated by a from-scratch Python page-hash verifier that clamps to codeLimit, handles dual CDs, and shares no code with the engine:

• all 8 repairable files → 0 stale slots, length preserved, only hash-slot bytes changed (32 bytes for single-slot thin binaries up to 1783 bytes for a 56-slot Bun executable);
• both Developer-ID CMS files correctly refused, 0 bytes touched;
• the tool's own --check agreed with the oracle before (exit 2) and after (exit 0) each repair.

(An earlier rcodesign-based pass false-failed the two Bun binaries — rcodesign reads past codeLimit into Bun's embedded payload, a bug its own output flags; the independent recompute confirms 56 stale slots → 0.)

Negative control: unpatched daemon (Nix 2.24.10 — has the bug)

Cold build: codesign --verify rc=0, binary runs — a cold build never triggers the bug. Delete one output of the multi-output reproducer, rebuild with the sibling present: codesign --verify rc=1, running the binary rc=137 — SIGKILL by the macOS kernel, cs_invalid_page in the system log. This is the corruption every mode of this PR either refuses, repairs, or (under warn/ignore) at least names.

Source-level trace of the trigger (Nix master a37db9d24)

• Scratch-path selection: an already-present output gets a fallback path via makeFallbackPath (derivation-builder.cc, scratch-path selection in registerOutputs).
• outputRewrites is populated per output where scratchPath != finalPath.
• The shared rewriteOutput lambda applies the map via RewritingSink to each output — both the input-addressed and content-addressed visitors pass through it, which is why the guard sits there.

Two trigger variants, both covered: sibling-reference (fish: bin/fish embeds the -doc output's path in __TEXT,__const) and self-reference (zsh: three self-referential paths, no sibling refs). On Hydra the trigger state is a routine consequence of per-output substitution plus long-running worker store state — no failed build required.

Testing

• 30 platform-independent unit tests over hand-built Mach-O byte fixtures (detection, repair, check-mode contract, malformed input) plus a seeded mutation fuzzer — these run on Linux too, so CI exercises the parser everywhere.
• Darwin functional suites for every door and mode: the partial-substitution and --check triggers; CMS refusal; unsupported-hash-type (detection says repairable, the repair skips it, the re-check refuses — the case that distinguishes "the hook ran" from "the signatures verify"); partial repair (one supported + one unsupported CodeDirectory: NAR hash updated to match the bytes on disk, signatures dropped, path reported unrepaired); a custom hook that does not implement --check; oversized files (the fixture generates a sparse file at test time under a test-shrunk size bound — nothing large is checked in); fixed-output/impure temp-dir handling; and the at-rest sweep including dry-run, idempotence, and batch resilience.
• Known coverage limits, disclosed rather than papered over: the privilege-drop uid assertion is not testable in nix's darwin CI harness (no darwin VM tests — same standing as diff-hook and pre-build-hook), and the check child's crash path at the substitution door cannot be forced deterministically.

Known standing costs

• macho-signature-verify adds per-path work at substitution time proportional to the number and size of Mach-O files in the path; the default ignore costs nothing. The detection scan memory-maps files (falling back to a bounded read where mapping fails), so it does not hold file contents in the daemon's heap.

[Ericson2314]

To be clear, this not about placeholders, but scratch paths, right?

[ak2k]

Yes, thank you for the catch. RewritingSink substitutes scratch-path bytes (from makeFallbackPath-synthesised paths populated in outputRewrites when a sibling output is already in the store at build start), not builtins.placeholder bytes. The placeholder is the user-facing hook that causes a scratch path to end up embedded in the binary in the first place, but the bug is specifically in the scratch-path → final-path byte substitution after the builder exits.

Amending the PR description, commit message, and rl-next entry to use the precise terminology. The mechanism and the fix are unchanged — only the wording was wrong.

[Ericson2314]

So I'll admit my current plan is to... just not have self references in the binary. How important is it to have paths like these?

(For other-output references, the idea is that you imperatively registered outputs, so you could do this sort of thing manually with Nix helping. E.g. install lib output, get store path back, use it in bin output.)

[emilazy]

This is the root cause of NixOS/nixpkgs#507531

I doubt it: I expect this is an instance of the long-standing NixOS/nixpkgs#208951 bug, which is cursed and nondeterministic (fresh rebuilds tend to fix it and @zhaofengli found that it even depends on whether a machine has built the derivation before at all).

Since --check/--rebuild have the separate issue of clobbering code signatures, this patch will appear to fix the root cause under an obvious testing methodology without actually doing so.

(And TBH encoding Mach-O knowledge deep in Nix guts is a pretty awful layering violation, there are avenues to fix the rewriting issue that don't require that.)

[ak2k]

So I'll admit my current plan is to... just not have self references in the binary.

Makes sense, and seems like the vastly superior architecture move. I wonder if something like this PR offers a short-term bridge for the existing darwin breakage (nixpkgs#507531, nixpkgs#208951) in the interim?

[emilazy]

I don't see evidence that this PR fixes those issues.

[ak2k]

Thank you, @emilazy. I'll try to address those three:

On whether this PR addresses the root cause of NixOS/nixpkgs#507531 / NixOS/nixpkgs#208951. You observed in nix-darwin#693 comment 38: "The bin/git in the broken derivation is identical to the correct one except for 32 bytes in the code signature section (a hash, maybe?). The share/man/man1/git.1.gz files, when uncompressed, differ only in the derivation hashes of the git-2.41.0-doc paths they link to." 32 bytes is the size of a SHA-256 hash entry in a CodeDirectory (CS_HASHSIZE_SHA256 = 256 bits), and the inline verification script in the PR description shows a single code-slot mismatch in each reproduction. The differently-embedded -doc paths match what RewritingSink rewrites when outputRewrites gets populated by a sibling output's presence in the store. My read is that this is the mechanism you were looking at. @winterqt later made an adjacent observation in #208951 comment 23: "It overrides the store path in the binaries, which breaks the code signatures of (at least) libraries (as they have their path embedded within). […]" I've added a source-level trace to the PR description for anyone who wants to follow the code path.

What this PR addresses is that specific mechanism — scratch-path → final-path byte substitution inside Mach-O code pages already covered by linker-signed page hashes. I shouldn't claim it covers every report in #208951; @tomberek's recent observation about "corrupted signature has been seen on different parts of the closure" may be a different root cause, of course.

This also explains the apparent nondeterminism that's made the bug hard to pin down. The trigger is hidden state: whether a sibling output happened to be in the store when the build started. On a fresh machine building an IA multi-output derivation cold, both outputs come out of the same builder run, neither is in the store yet, outputRewrites is empty for the sibling, and the binary is clean. On a machine that already has the sibling — from a prior build of this derivation, or substitution of one output but not the other — outputRewrites gets populated and RewritingSink runs over the binary. As I read it, this predicts that a cold-store build will produce a clean binary (it's the absence of the sibling at build start that matters, not the rebuild itself), and that whether a particular machine has previously built the derivation is what determines whether the bug surfaces. This would also explain why Hydra's cached artifacts tend to be valid: Hydra typically schedules a derivation's outputs together from a state where neither is in the store yet, so the rewrite path isn't exercised.

On the test methodology. I've added a non---check reproduction to the PR description: nix-store --delete <fish-out> followed by an ordinary nix build, with fish-doc still in the store, reproducing the same single page-hash mismatch. A positive counterpart against the patched daemon is included alongside it, showing the same derivation rebuilt cleanly (0/2526 mismatches). As I read the trigger condition, it isn't --check — it's "a sibling output is present in the store when the build starts". --check happens to force that state in a fresh store, which is convenient for the functional test, but an ordinary upgrade, a nix-store --delete of one output, or a failed build that left half the outputs behind would reach the same code path.

On the layering violation. I agree — Mach-O parsing logic in libstore is structurally wrong, and this PR makes it a little more wrong by adding page-hash recomputation alongside the existing darwin-specific code. My read is that the deeper cause is length-preserving byte substitution on already-signed outputs: ld -adhoc_codesign signs bytes that the daemon then rewrites via RewritingSink, and the Mach-O fixup only exists to repair what that substitution broke. The two avenues I can see that would actually move Mach-O knowledge out are @Ericson2314's imperative-output rework above (which would make the whole substitution machinery unnecessary) and a nixpkgs-side post-build hook (I'm not sure if the existing post-build-hook mechanism runs early enough to intercept the rewrite).

Short of those, both #14999 and this PR fix the corruption from inside the daemon, via different mechanisms. #14999 (@andrewgazelka) shells out to codesign -f -s - to re-sign the binary. That produces a working signature but not a bit-identical one — codesign's re-signed CodeDirectory differs from the original linker-signed signature in flags (the linker-signed bit is cleared), special-slot layout, and default page size. codesign -P 4096 would address the page size, but the other two remain, so rebuilds still fail --check. This PR recomputes the affected page hashes in place, which preserves bit-reproducibility.

[emilazy]

a nix-store --delete of one output, or a failed build that left half the outputs behind would reach the same code path.

I don’t think these correspond to the circumstances in which we see Hydra produce these broken outputs.

It’s expected that local rebuilds of the derivations won’t have any issues, since it’s apparently nondeterministic and seemingly partially dependent on persistent system state of some kind. So building a broken‐in‐the‐cache derivation locally and seeing that it doesn’t exhibit the issue does not demonstrate that this PR fixes the issue; that’s already what we observe with no change.

I know @zhaofengli had a somewhat reproducible test setup, but it was difficult to arrange.

[emilazy]

(For clarity: the sibling outputs thing is an interesting observation that I can imagine might have something to do with what we’re seeing here, but given the amount of times we see random stuff in staging-next crashing at startup from this bug, I’d be pretty surprised if they happened to all be builds that had failed before but still produced some outputs, and then got built again on the same machine with the output not having been cleaned up, or similar.)

[ak2k]

Thank you, @emilazy. One correction first, then to your three points.

Correction: My statement that "a failed build that left half the outputs behind would reach the same code path" was wrong. Nix's registerOutputs is atomic per-derivation; a failed build doesn't leave one of a multi-output drv's outputs valid in the store. The other trigger scenarios are correct as written: nix-store --delete of one output, or substitution of one output but not the other.

It's expected that local rebuilds of the derivations won't have any issues

Local rebuilds with the specific trigger setup (sibling output already in the store, target output absent, substitution disabled) do reliably exhibit the issue on my machine. I just ran three consecutive iterations on aarch64-darwin (macOS 26.2, unpatched system daemon Nix 2.24.10), same nix-store --delete, same --option substitute false, same nixpkgs d96b37b, same store state (fish-doc present from a prior build):

iteration 1 NAR hash: sha256:1qplch87dy4242vxwi3s5h62m6gnywn0f8z9wf659vkrh6hm4a0g
iteration 2 NAR hash: sha256:1qplch87dy4242vxwi3s5h62m6gnywn0f8z9wf659vkrh6hm4a0g
iteration 3 NAR hash: sha256:1qplch87dy4242vxwi3s5h62m6gnywn0f8z9wf659vkrh6hm4a0g

Bit-identical between runs. Both produced codesign: invalid signature, exit-137 SIGKILL, and 1/2526 mismatches at page 1872 @ 0x00750000, the same single-slot mismatch as the cache.nixos.org artifact at the same store path.

since it's apparently nondeterministic

Seemingly nondeterministic in production, yes — but the trigger setup above is at least one state configuration under which it reproduces deterministically. As I read the bug I've verified, it's state-dependent rather than nondeterministic: it requires the sibling output to be in the store at the moment the build starts. Within that state, on this machine, it fires bit-identically. Outside that state (truly cold store, no sibling present), it doesn't trigger at all. Whether @zhaofengli's reproducible setup converges with this one or describes a distinct mechanism, I haven't traced yet; if there's a pointer to where it lives, I'd be glad to look and compare.

So building a broken‐in‐the‐cache derivation locally and seeing that it doesn't exhibit the issue does not demonstrate that this PR fixes the issue; that's already what we observe with no change.

This isn't the shape of the PR body's A/B. The unpatched local rebuild does exhibit the issue under the trigger setup — the same 1/2526 mismatch and SIGKILL shown above is what the unpatched block in the PR description shows. The patched-daemon rebuild from the same starting state and the same command sequence produces 0/2526 and runs. Same store path, same nixpkgs revision, same trigger condition; the only variable I changed is the daemon. So the comparison isn't "broken cache vs incidentally-clean local rebuild" — it's "broken cache, broken unpatched local rebuild under the trigger, clean patched local rebuild under the same trigger."

If you'd like to test on your machine, I've packaged the described trigger setup as a flake at https://github.com/ak2k/nix-507531-repro:

nix run github:ak2k/nix-507531-repro

On aarch64-darwin it runs the unpatched rebuild three times (to show the bug fires deterministically across iterations), runs the patched rebuild once via a private daemon built from this PR's commit, then prints a side-by-side comparison table and a final PASS/FAIL line. ~5–7 min wall time, one sudo prompt for the patched daemon spawn. A recorded transcript of a passing run is at examples/ab-test-output.txt if you'd rather read than run.

If on your darwin machine the corruption reproduces on the unpatched daemon under this trigger, the A/B in the PR body holds and the patched daemon's fix applies under the same trigger. If it doesn't reproduce, that would point to a second state factor I haven't isolated here, distinct from the trigger above. Either way, the bug is independently visible: cache.nixos.org is serving a fish that macOS refuses to execute, the unpatched local rebuilds above reproduce the same single-slot mismatch deterministically, and #507531 / #208951 seem to collect related reports from darwin users.

One epistemic caveat I should make: the observable here — a single SHA-256 page hash slot mismatch in a linker-signed CodeDirectory — is narrow enough that two distinct mechanisms could in principle produce indistinguishable outputs, and codesign --verify plus the kernel's page-in check can't tell them apart. The claim I can make is that this PR fixes the mechanism I've reproduced; the claim I can't make is that it fixes every report regardless of mechanism, since some of those could in principle have a distinct cause that converges on the same symptom. If there's something concrete pointing at a separate mechanism in some of those reports, I'd be glad to dig in.

[ak2k]

Thanks @emilazy — your parenthetical caught a real narrowness in how I'd been framing this. Two refinements after looking more carefully at the mechanism and at Hydra's source.

1. The trigger isn't specifically "sibling present"; it's "any output being built has scratchPath != finalPath", which happens whenever that output was in the store at build start. In src/libstore/unix/build/derivation-builder.cc at this PR's base a37db9d24, scratch-path selection routes any already-present output through makeFallbackPath(status.known->path) at L798; the finish lambda at L1614 populates outputRewrites with a scratchHash → finalHash entry for every such output; and the rewriteOutput lambda at L1634 applies those rewrites via RewritingSink. Two variants:

• Sibling-reference: X is being built, sibling Y is present, X embeds Y's path. fish hits this: fish-4.2.1/bin/fish embeds its fish-4.2.1-doc sibling's share/doc/fish path as a literal string in __TEXT,__cstring.
• Self-reference: X is being built, X itself is already present at build start (forcing a fallback scratch path for X), X embeds $out-derived self-references. This fires under --rebuild / --repair-path specifically — a wanted=false already-valid output otherwise skips rewriteOutput via the AlreadyRegistered path at L1484, so the self-ref rewrite on X's own new bytes only runs when the rebuild explicitly re-enters rewriteOutput. zsh hits this under the PR's --rebuild functional test: zsh-5.9/bin/zsh has no sibling runtime references at all, but three self-references to /nix/store/s07v...zsh-5.9/{share/zsh/5.9/functions,lib/zsh/5.9,etc/zshenv}, all inside the flags=0x20002(adhoc,linker-signed) CodeDirectory.

The sibling/self split is the same distinction @Ericson2314 drew from the architectural direction above — "just not have self references in the binary" — reached here from the source side.

2. The trigger state is not rare on Hydra; it's a consequence of per-output substitution. My earlier examples (nix-store --delete of one output, substitution of one output but not the other) sounded narrow because I was thinking about end-user actions. Looking at Hydra's queue-runner source, the routes are routine:

• In subprojects/hydra-queue-runner/src/state/mod.rs at cd235f7, the master computes the set of missing outputs for a drv (L1788–L1815), then fans them out per output via substitute_output in a buffer_unordered(10) stream. A single failed substitute (transient network, partial cache state, S3 inconsistency) leaves the drv with some outputs substituted and others missing. The drv is then scheduled for a local build, with the partial state in place.
• The builder side does the same on its own store: substitute_paths loops ensure_path per path with the same partial-failure mode.
• Long-running workers: a worker that previously built package Y (where Y runtime-depends on D.out) ends up with D.out in its store. When Hydra later asks that worker to build D itself, D.out is present → fallback scratch path → rewrite fires in the newly-built bytes.
• GC asymmetry: workers GC on runtime-root reachability. For multi-output drvs where out is rooted but siblings aren't, the siblings get collected while out stays (or vice-versa for the rarer case).

None of these require "failed build that left some outputs"; they're routine consequences of Hydra's per-output substitution design plus long-running worker store state. For a given staging-next rebuild, the probability that a multi-output drv has at least one output present when its sibling needs rebuilding seems high enough that I don't think it would need to be a rare coincidence to explain the observed rate.

Quick pattern-check against the heavily-mentioned packages in the threads, verified in-situ on this machine via strings and otool -L. Sibling-reference: fish (bin/fish embeds its fish-4.2.1-doc sibling path as a cstring literal in __TEXT,__cstring), git (bin/git embeds its git-2.51.2-doc sibling path as a cstring literal), curl (-bin's bin/curl references its out sibling's libcurl.4.dylib via an LC_LOAD_DYLIB load command — same rewrite mechanism, different Mach-O section; the trigger isn't specific to cstring literals, it fires on any byte covered by a page hash). Self-reference: zsh (bin/zsh has three $out-derived self-references, no sibling refs), bash (bin/bash has one $out-derived self-reference, no sibling refs). gitFull isn't built locally so I haven't traced it, but it's the same derivation family as git.

If you or @zhaofengli have a reproduction that doesn't fit this trigger, I'd want to look at it.

[rkjnsn]

Thank you for the detailed investigation! This explains why just rebuilding fish from source locally didn't fix the problem: since the substituted version was already in the store, I was running into the exact situation of nix using a scratch path for the install prefix, breaking the resulting binary.

Mach-O parsing logic in libstore is structurally wrong, and this PR makes it a little more wrong by adding page-hash recomputation alongside the existing darwin-specific code.

It seems to me that if Nix is going to be building with an alternative install prefix, then rewrite that install prefix in the resulting output, it should know how to properly do the rewrite, at least in common cases like a regular macOS binary. I agree that ideally Nix shouldn't be doing rewriting at all, but that's not the situation today.

Moving forward, it seems like there are three main options from Nix's side of things:

1. Stick with the status quo, where Nix uses scratch paths and does the rewriting. Nix needs to know how to properly rewrite paths (including updating any hashes or signatures) in common file types including libraries and binaries. A rewrite hook could be optional for packages to handle special cases.
2. Nix still uses scratch paths, but isn't responsible for rewriting them. If the a package embeds any references to its output paths, it must provide a hook to do the rewriting. Nix only does a best-effort scan for the scratch path, and fails if found. Some affordance is made, for a while at least, for building old nixpkgs versions with Nix path rewriting enabled. Once fully removed, such versions could only be built fresh (no outputs in the store, no scratch path needed).
3. Nix never uses scratch paths. The package is always built with its true output directory as the install prefix. On Linux, for single-user installs, this would probably mean using unprivileged namespaces or proot or some such. On macOS, I don't know of any solution short of running builds in a lightweight macOS VM via the Virtualization Framework (which would help with blocking network access for builds, along with other macOS sandbox issues).

For both 1 & 2, to prevent show stoppers like NixOS/nixpkgs#507531 from popping up sporadically, it seems prudent to have a test builder that always uses a scratch path to catch issues where Nix or a package aren't rewriting the path properly.

[emilazy]

Before talking about potential solutions, let’s first ensure that the diagnosis is correct. The hypothesis is that these code signing issues with executables built by Hydra are a result of hash rewriting – makeFallbackPath picks non‐final output paths for a build, the build is performed with those output paths, and then after the build completes the paths are rewritten in the outputs and the outputs are moved to their final locations. This would indeed break the hashes in ad hoc code signatures, and is a well‐known issue; that’s why --rebuild doesn’t work properly on macOS.

However, Hydra doesn’t use --rebuild. The question is whether the (increasingly common) issues we see from users of the binary cache, such as NixOS/nixpkgs#208951, NixOS/nixpkgs#507531, and NixOS/nixpkgs#511265, are a result of the same fallback path mechanism. The rationale given for why this would be expected to happen commonly on Hydra was as follows:

• In subprojects/hydra-queue-runner/src/state/mod.rs at cd235f7, the master computes the set of missing outputs for a drv (L1788–L1815), then fans them out per output via substitute_output in a buffer_unordered(10) stream. A single failed substitute (transient network, partial cache state, S3 inconsistency) leaves the drv with some outputs substituted and others missing. The drv is then scheduled for a local build, with the partial state in place.

The first issue with this is that the Rust Hydra queue runner rewrite has not yet been deployed, so the code references aren’t relevant here. It would seem a priori surprising to me for Hydra’s connection to S3 to be so unreliable, and at least if Nix itself is being used for substitution in the current Hydra codebase (I don’t know for sure), it would retry the download several times before giving up and building.

We can check cases empirically, though. Let’s take NixOS/nixpkgs#507531 and NixOS/nixpkgs#511265, as recent examples of this issue that got a lot of attention. (It is true that all of fish, ffmpeg, and git have multiple outputs; it does seem likely that multiple outputs may be involved in some way.)

/nix/store/gngn7y9mn510mf1hkmr0l69qbpvxfbfh-fish-4.2.1/bin/fish and /nix/store/6a5nr567sb4a36lisa6gydpp3bfij1vv-ffmpeg-8.0-bin/bin/ffmpeg both get SIGKILLed on startup; Console.app shows the following logs:

2026-05-09 16:29:52.299626+0100 0x181a92   Default     0x0                  0      0    kernel: ffmpeg[86015] triggered unnest of range 0x1e8000000->0x1ec000000 of DYLD shared region in VM map 0x5ed954bb1cf76c4d. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-05-09 16:29:52.324065+0100 0x181a92   Default     0x0                  0      0    kernel: CODE SIGNING: cs_invalid_page(0x1020e0000): p=86015[ffmpeg] final status 0x23020200, denying page sending SIGKILL
2026-05-09 16:29:52.324072+0100 0x181a92   Default     0x0                  0      0    kernel: CODE SIGNING: process 86015[ffmpeg]: rejecting invalid page at address 0x1020e0000 from offset 0x8000 in file "/nix/store/hn58l3pvn5iwq87p6ddp9wsw8ai9dl93-ffmpeg-8.0-lib/lib/libavdevice.62.1.100.dylib" (cs_mtime:1.0 == mtime:1.0) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
2026-05-09 16:29:52.324114+0100 0x181a92   Default     0x0                  0      0    kernel: ffmpeg[86015] Corpse allowed 1 of 5

2026-05-09 16:29:12.948558+0100 0x181852   Default     0x0                  0      0    kernel: fish[85948] triggered unnest of range 0x1e8000000->0x1ec000000 of DYLD shared region in VM map 0xdca0c4dbf7d9f4f7. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-05-09 16:29:42.978386+0100 0x181a11   Default     0x0                  0      0    kernel: fish[86009] triggered unnest of range 0x1e8000000->0x1ec000000 of DYLD shared region in VM map 0x32aea3c9aff28bb5. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-05-09 16:30:38.340481+0100 0x181d68   Default     0x0                  0      0    kernel: fish[86064] triggered unnest of range 0x1e8000000->0x1ec000000 of DYLD shared region in VM map 0x9b87ae48aaf91acb. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-05-09 16:30:47.191288+0100 0x181de4   Default     0x0                  0      0    kernel: CODE SIGNING: cs_invalid_page(0x10340c000): p=86070[fish] final status 0x23020200, denying page sending SIGKILL
2026-05-09 16:30:47.191300+0100 0x181de4   Default     0x0                  0      0    kernel: CODE SIGNING: process 86070[fish]: rejecting invalid page at address 0x10340c000 from offset 0x750000 in file "/nix/store/gngn7y9mn510mf1hkmr0l69qbpvxfbfh-fish-4.2.1/bin/fish" (cs_mtime:1.0 == mtime:1.0) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)

In the case of FFmpeg, it’s unhappy about one of the dylibs; all of them fail codesign --verify, and they all pull in /nix/store/hn58l3pvn5iwq87p6ddp9wsw8ai9dl93-ffmpeg-8.0-lib/lib/libavutil.60.dylib, ultimately, which itself fails code signing even though its dependencies are fine. So this does point to all the Mach‐O files in the outputs of that FFmpeg derivation having been corrupted. Interestingly, though, a simple test program that links against that libavutil.60.dylib and calls avutil_version() compiles and runs fine. So let’s focus on executables. The log is unhappy with /nix/store/gngn7y9mn510mf1hkmr0l69qbpvxfbfh-fish-4.2.1/bin/fish specifically, which also fails codesign --verify, but all the dylibs pulled in by that executable are fine.

Given these store paths, we can look up the Hydra build logs for them:

• The fish build is https://hydra.nixos.org/build/325686818, raw log at https://cache.nixos.org/log/s8swwl2iva8bw1yzcpdbifskbpw8cwhl-fish-4.2.1.drv.

• The ffmpeg build is https://hydra.nixos.org/build/325686676, raw log at https://cache.nixos.org/log/gdhqs3kr69yz4l14vqb7ali2bcycnsk6-ffmpeg-8.0.drv.

(I’ve preserved these logs at https://gist.github.com/emilazy/78e5286ec92f1ec6fda6be7b23439ad2 for posterity.)

If makeFallbackPath was being used for the builds that produced these logs, we would expect that the build logs would refer to fallback paths – it would be what’s passed to ./configure scripts, where the binaries are installed from the build’s perspective, and so on.

That does not seem to be the case for fish; the correct output paths for /nix/store/s8swwl2iva8bw1yzcpdbifskbpw8cwhl-fish-4.2.1.drv are /nix/store/gngn7y9mn510mf1hkmr0l69qbpvxfbfh-fish-4.2.1 and /nix/store/62v6ki5ql5wxvgabn60aln10l2a4aacb-fish-4.2.1-doc. Both appear verbatim in the log, and no other relevant paths do.

So it seems to me like for the makeFallbackPath hypothesis to hold for this build, we would need a rebuild to have taken place after a partial substitution, but for the log itself to not have been replaced. Perhaps there could be a story where cache paths are never rewritten, but the upload of fish^out failed, so only fish^doc and the log got pushed out, with fish^out later being rebuilt and uploaded in a corrupted form on another builder, with the log being discarded? That seems like it would require a very convenient sequence of errors to be happening so often lately, though.

The ffmpeg case is more interesting. The correct output paths for /nix/store/gdhqs3kr69yz4l14vqb7ali2bcycnsk6-ffmpeg-8.0.drv are /nix/store/lspxkkbmyvzp36jbvjvy3a3d1j979iqb-ffmpeg-8.0, /nix/store/6a5nr567sb4a36lisa6gydpp3bfij1vv-ffmpeg-8.0-bin, /nix/store/g7gwxb7jinbidq6j0h0fd95rf6zc8937-ffmpeg-8.0-data, /nix/store/60hnmkly9hdsn0ajqmqf2lmd3vnf5w94-ffmpeg-8.0-dev, /nix/store/gpf5ks0x6x2ih4jjasp53cmx0cmk1bbw-ffmpeg-8.0-doc, /nix/store/hn58l3pvn5iwq87p6ddp9wsw8ai9dl93-ffmpeg-8.0-lib, and /nix/store/j6mqv1jx0pvkz3ww8j3mk65pfg5cc4pi-ffmpeg-8.0-man.

All of these appear verbatim in the build log, except for /nix/store/g7gwxb7jinbidq6j0h0fd95rf6zc8937-ffmpeg-8.0-data, which appears instead as /nix/store/6kppwm3q8g67fs44v3cnzf9vxn3k7px3-ffmpeg-8.0-data. This is indeed the fallback path for that output! Indeed, if we ensure that no other output is present, do nix build /nix/store/g7gwxb7jinbidq6j0h0fd95rf6zc8937-ffmpeg-8.0-data to fetch it from the cache, and then do nix build --substituters '' github:NixOS/nixpkgs/36643231654bc49df94fcaceb65827b1d9080d75#ffmpeg_8 -L, we get an identical configure flags: line to the Hydra log.

So something weird with partially‐present outputs is happening here, and this log does seem to be for a rebuild! If we do that exact rebuild, we get broken FFmpeg outputs, and indeed a rewritten /nix/store/g7gwxb7jinbidq6j0h0fd95rf6zc8937-ffmpeg-8.0-data path is present in /nix/store/6a5nr567sb4a36lisa6gydpp3bfij1vv-ffmpeg-8.0-bin/bin/ffmpeg and the dylibs.

I’m a bit surprised, but in this case it seems to be true: partial rebuilds of paths seem to be responsible for the issue with that FFmpeg build. Baking in Mach‐O knowledge into Nix could plausibly solve that case. But that doesn’t explain why it happened, and given that there doesn’t appear to be any similar indicator with the fish build, I’m not sure we have evidence yet that this is causing a substantial proportion of the issues we’re seeing people have.

The first time a derivation is built, all its outputs are produced. My understanding was that they are then all unconditionally pushed to the cache, and that once a path is pushed to the cache it is never rewritten, which seems like it would make it hard for any partial rebuild of the outputs to result in any corruption of the cache, unless it leads to downstream issues with codesigning executables on the same builder. However, I am not confident in this assessment, and would need someone with more experience with the Hydra codebase and deployment to confirm. I would tend to think that, rather than chronic issues with substitution failures, it’s more likely to be Hydra doing weird things splitting builds of outputs across multiple machines and those running into race conditions (it’s certainly the case that we get duplicate builds on multiple builders running sometimes), or builds failing but still registering a partial set of outputs (due to lack of disk space, perhaps, which has happened increasingly commonly on our Hydra Darwin builders recently?), but these are just guesses.

I think that before we discuss further about adding Mach‐O rewriting code to Nix or self‐references, we should make sure that the issues that have resulted in this PR and the attention it’s getting are going to actually be reliably fixed by it, and that there’s not a more immediate cause. While --rebuild and partial substitution of outputs being broken on macOS for end users is unfortunate and could use a fundamental solution, it’s far from the critical issue that the persistent Hydra ad‐hoc code signature issues we’re seeing are, and it would be unfortunate to block resolution of such a major pain point on committing to the resolution to a bunch of subtle design decisions that would be required to solve the general case, especially if we haven’t yet shown that it’s the only or main cause of these problems.

Certainly something weird is going on, and multiple outputs could very plausibly have something to do with it even in cases like fish where we don’t see anything fishy in the log. But it seems to me like something is missing from the root cause analysis here.

[emilazy]

Okay, so if you grab /nix/store/62v6ki5ql5wxvgabn60aln10l2a4aacb-fish-4.2.1-doc from the cache and then build /nix/store/gngn7y9mn510mf1hkmr0l69qbpvxfbfh-fish-4.2.1, it does indeed get the exact same SIGKILL, and the binary does contain the rewritten /nix/store/62v6ki5ql5wxvgabn60aln10l2a4aacb-fish-4.2.1-doc (though of course the fallback path is visible in the build log, unlike on Hydra). It could indeed also be what was going on with @zhaofengli’s test harness (but I don’t remember the details and can’t verify one way or the other); the build succeeds the first time and failed thereafter because only some of the store outputs were actually deleted. I might be becoming a believer, here.

I’m still not sure about how this would interact with the cache behaviour, though. It can’t just be a local issue that persists to the point where things get uploaded to the cache, because then we’d see the fallback path in the log that gets uploaded.

I guess we could observe both of these issues if we assume that store outputs and logs are only ever written to the cache once, but that any of them can fail to upload:

• ffmpeg gets built, ffmpeg^data gets pushed out to the cache/registered/whatever, nothing else does, including the build log. Something pulls in another output; ffmpeg gets rebuilt with the fallback path for ffmpeg^data, gets broken by path rewrites, everything else gets uploaded, including the log.

• fish gets built, fish^doc and the build log get pushed out to the cache, but fish^out doesn’t. Another build pulls in fish^out; fish gets rebuilt with the fallback path for fish^doc, gets broken by path rewrites, and gets uploaded, but the original log stays.

This still seems like a surprising sequence of events to be happening on such a regular basis, but it’s at least consistent. I am still very tempted to say that the disk space issues on Hydra builders could be contributing somehow to the increased prevalence of this issue recently, but I’m not quite sure how that would fit into the story.

If this story is correct, then the least invasive way to address the immediate problems we’re seeing on Hydra is probably to not publish .narinfo files fully to the cache until all outputs of a derivation have been successfully uploaded, and to ensure that Hydra never gives up on substitutions and rebuilds things instead. I think some root cause analysis on what’s happening on the infra side of things is warranted, though.

[rkjnsn]

Before talking about potential solutions, let’s first ensure that the diagnosis is correct.

Whether or not this issue is the cause of all binaries with invalid page hashes in the cache, it clearly is an issue that is reproducible and affects local builds as well.

the least invasive way to address the immediate problems we’re seeing on Hydra is probably to not publish .narinfo files fully to the cache until all outputs of a derivation have been successfully uploaded, and to ensure that Hydra never gives up on substitutions and rebuilds things instead.

If there's a quickish way to avoid Hydra poisoning the cache with such broken builds, that's definitely a good first step to fix things for most folks as soon as possible. That said, not all builds can be substituted (whether due to overrides, intentionally disabling substitution, or even using a different store path), and the problem situation can easily occur for local builds (e.g., if the user neglects to set keep-outputs and some outputs get GC'd), so I do think this should be fixed in nix somehow.

If Hydra is adjusted so as to never build with a scratch path itself, I think it becomes especially important that building nixpkgs with scratch paths is tested an some interval or that nix never uses them locally. Otherwise, it's only a matter of time until a user ends up broken building locally.

[emilazy]

Whether or not this issue is the cause of all binaries with invalid page hashes in the cache, it clearly is an issue that is reproducible and affects local builds as well.

Sure, I agree, of course, now that I’ve verified that – before the research I did for my comment, it wasn’t clear to me that we actually had evidence that it was the cause of any binaries with issues in the cache.

Disabling fallback paths seems like probably the correct way to go for now, given that they’re not used on sandboxed Linux, and making path rewriting work reliably on macOS in a way that works reasonably with the Nix model would require a non‐trivial amount of design work. --rebuild being a footgun on macOS already confuses people, and outside of that they’re relatively niche in practice; you need to have some outputs in the store but not all, and be in a situation where you can’t easily just delete those outputs – if it turns out to come up more than I expect then some tweaks to how GC handles multiple outputs on macOS could probably work out, but given that it’s already broken I expect a hard error is going to be preferable to silent corruption.

[mroi]

Maybe this is an interesting datapoint:

The behavior of macOS crashing such invalid-signature binaries also appears non-deterministic. I have two machines with the same opencode binary installed (from Nixpkgs 3d8f0f3f). Turns out this binary has an invalid signature:

> codesign -vv /nix/store/swvx6k42qawwj376byi0fxjiy3yf9yb9-opencode-1.15.7/bin/.opencode-wrapped 
/nix/store/swvx6k42qawwj376byi0fxjiy3yf9yb9-opencode-1.15.7/bin/.opencode-wrapped: invalid signature (code or signature have been modified)
In architecture: arm64

But until today, this binary was running fine on both machines. Today, I updated one machine to macOS 27 Beta and now on that machine the same binary gets killed on launch with the known invalid page error. (It still works on the macOS 26 machine.)

[ak2k]

I went through the currently-failing population in cache.nixos.org — 330 failing slices across the three darwin channels (2026-06-11 scan of ak2k/nix-507531-scope), 101 unique drvs after deduplication via narinfo Deriver — and classified each by what modifies the signed bytes. 53 of the 101 carry a direct tool signal in their cached Hydra log (install_name_tool "will invalidate" warnings, 12; bun build --compile, 24; @electron/fuses, 17). The rest I placed by package identity plus a consistent per-slice signature, by the contents of the stale page, or by the absence of any in-build step that touches the binary — called out per row below. Two mechanisms I also reproduced locally, and one I confirmed by rebuilding the package.

Mechanism	Drvs	Packages	Evidence
Hash rewriting (what this PR fixes)	6	ffmpeg-headless; agda2hs, cmdargs-browser, dhall-docs, mmsyn7ukr-array, stache	Confirmed for stache: a --rebuild with its data output present routes the Cabal Paths_ references through the fallback-path rewrite, and the daemon logs the fix-up recomputing bin/stache (12849 code slots, matching the cached artifact) and the package dylib. ffmpeg: fallback paths in the cached log, three log-referenced output paths 404 on cache.nixos.org (fallback paths are never uploaded). The other four Haskell packages: the stale page(s) carry that same Paths_ block — self- and sibling-output store paths. None show a fallback signal in the log, so the page contents and the stache rebuild are the basis, not the log
install_name_tool during the build	12	swift, libtorch	Every slice's mismatch is page 0, the load-commands region -id/-change rewrite, and all 12 logs carry the tool's own "will invalidate the code signature" warning
bun build --compile during the build	34	opencode, gitlab-duo, filen-cli	24 logs show bun build --compile directly; the 10 opencode drvs build the binary in-build through opencode's own bun script (the log shows it built and smoke-tested) and carry the same slice signature. Reproduced locally: bun build --compile rewrites its runtime in place, leaving a linker-signed-flagged CodeDirectory whose last page hash is stale — every arm64 slice in this family has that shape, one mismatch on the last hashed page
@electron/fuses via electron-builder	27	teams-for-linux, shogihome, httptoolkit	17 logs show the explicit @electron/fuses step; all 27 fail on the Electron Framework binary inside the packaged app. Reproduced locally on the electron-41.3.0 framework: flipping fuses changes 3 bytes at the fuse sentinel and invalidates the one page containing it
Upstream artifact already broken	22	tailwindcss_4, esy, vscode-extension-kilocode, avalonia-ilspy	No in-build step touches the binary (it arrives prebuilt). Checked for tailwindcss: the tailwindcss-macos-arm64 v4.2.4 binary from GitHub releases carries the same single-stale-page signature as the cached nix output (18557 slots, stale hash on page 18556) — the nix build only unpacks and wraps it

The opencode report is the bun family: opencode ships as a Bun single-executable, built during its nix build, and every cached arm64 .opencode-wrapped has one stale hash on the last page — the page where Bun embeds its payload, which I reproduced on another Bun binary. Nothing reads that page until the kernel validates it, which would explain the same binary running on one machine and being killed on the macOS 27 one.

Disabling fallback paths on macOS — the direction proposed above — turns the rewriting family into a hard error instead of silent corruption. In today's cache that family is small: one drv with direct fallback-path evidence (ffmpeg), plus five Haskell packages with the same Paths_ fingerprint, one of which (stache) I confirmed by rebuild goes through that rewrite path. But it also covers the local --rebuild and GC'd-sibling cases, which are the ones that bite with no cache involved, so it stands on its own merits. It doesn't reach the other 95: those are already broken by build tools or upstream packaging before registerOutputs runs.

For those 95, two non-exclusive paths. Per-package fixes in nixpkgs — re-sign after bun build / electron-builder, reject the broken vendored binaries at fetch time — need no Nix change and keep the failure visible at build time. On the Nix side, this PR's fix-up is gated to the rewrite, so it covers the rewriting family and nothing else. Extending it to the other 95 would mean parsing every output in the daemon — the part flagged earlier as not belonging in libstore — so those are the ones that fit the builder-context rewrite hook, or the per-package fixes above, rather than an unconditional daemon pass. The rewriting case itself could stay as this daemon-side fix-up, move behind the hook once its shape is settled, or be mooted by disabling fallback paths; I'm glad to take any of those.

Per-drv classification data and scripts are in ak2k/nix-507531-scope.

[ak2k]

Based on the classification above, a concrete proposal. The population splits by who breaks the bytes: the daemon itself (hash rewriting), the package's own build tools (73 drvs), or upstream (22 drvs). The last two are nixpkgs's to fix, by re-signing in fixupPhase or rejecting broken vendored binaries at fetch time, and need nothing from Nix. The first is Nix-caused: the builder has already exited when RewritingSink runs, so no derivation-side fix can reach it. That family covers the 6 drvs above, the local --rebuild and GC'd-sibling cases, and CA cold builds (nix#6065), whenever the rewrite touches a signed Mach-O.

For that family I'd reshape this PR in two steps:

Step 1: detect and refuse. When a hash rewrite has modified a file carrying LC_CODE_SIGNATURE, fail the build with an error naming the already-present outputs to delete. Detection only: a magic-number check and a load-command walk, no signature parsing, no writes. This is the disable-fallback-paths idea scoped down: a blanket disable would also break IA rebuilds whose self-references rewrite harmlessly (scripts, text files — the common case), and can't apply to CA, where scratch paths aren't optional. Detection turns exactly the would-be corruptions into hard errors, on both IA and CA.

Step 2: repair, outside the daemon. Move the fix-up out of libstore into a small tool shipped with Nix, which the daemon execs as the build user at the detect point (the diff-hook pattern), failing closed to the step-1 error. The isolation is the same as running it in the builder's context: a malicious Mach-O exploiting the parser gets the build user, not root. Wired as a post-rewrite-hook setting defaulting to the Nix-shipped tool, this becomes the rewrite-hook mechanism suggested above, with a default payload. For this family I think the default matters, though I may be missing something: derivation-declared hooks can't fix already-published nixpkgs revisions (older releases declare no hooks, so partial rebuilds of them stay broken — the same backward-compatibility problem as removing self-references), and the repair needs to be byte-identical to a clean build for --check and CA hashing, which a codesign re-sign inside the package can't provide. For package-caused breakage the derivation-declared form still seems right.

Does that shape address the libstore and attack-surface concerns? If so I'll reshape along these lines, as two commits here or as a follow-up, whichever is easier to review.

[xokdvium]

Tbh this seems like a bit of a whack-a-mole trying to fix the symptoms, rather than the root cause. There can be many more issues that can arise from Franken-builds than just what has been observed here.

Before introducing additional complexity we should try to attack the problem at its root – the bugs in hydra that lead frankenbuilds. The recent round of those has been caused by a combination of AWS S3 uploads not being properly retried + frankenbuilds that happened when retrying the builds with a partial upload/substitution. And if necessary, new safeguards to explicitly disallow interim paths from being used on the builders (as in a nix option) + bailing out when only some outputs could be substituted.

[mroi]

@ak2k Thanks for the in-depth analysis.

Regarding the bun build --compile category (which includes my opencode comment), I just filed an issue upstream, which their robots have started to work on. Would be best if this got fixed outside of Nixpkgs.

[ak2k]

The PR has been restructured per the two-step proposal above; the body now describes the new shape. A summary of where it landed, since the thread has been quiet for a few weeks.

The design in one paragraph: on macOS, a binary only works if its signature's page hashes match its contents, and this PR makes that a property Nix preserves when bytes enter a local store. It is checked where the NAR hash is already checked, repaired where permissions are already canonicalised, by one Nix-shipped tool that never runs as root. Three commits, one per enforcement point: the build door refuses rewrites that would break a signature (default on), a privilege-dropped hook repairs the daemon's own rewrite damage with the output registered only after the repaired signatures re-verify, and an opt-in substitution-time check covers binaries that were already broken where they were built. One invariant, enforced at each door bytes come through, rather than a patch per producer.

On the diagnosis discussion, @emilazy's verification looks right to me: the recent fish/ffmpeg cache breakage is exactly the builds-retried-with-partial-outputs mechanism (ffmpeg's cached log shows the fallback path directly). The proposed Hydra-side fixes, never publishing narinfos until all outputs upload and rebuilding instead of giving up on substitutions, are worth doing and would cut most of the cache-side incidence. What they cannot close: a worker whose GC keeps out but collects a sibling (keep-outputs defaults false), the locally-reproducible case @emilazy demonstrated, and CA cold builds, where scratch paths differ by design and nothing needs to be "in place" for the damaging rewrite to fire (#6065). That is why the guard sits in Nix at the rewrite itself.

On the bail-out shape, @xokdvium's "bail out when only some outputs could be substituted" and this PR's build door are the same safeguard at different granularity. The coarse form (at build start) is simpler, needs no Mach-O knowledge in Nix, and also catches non-signature rewrite damage. The fine form (at the rewrite) has a narrower error surface and reaches CA cold builds, which the coarse form cannot, since nothing is present at build start to bail on. The diff makes the trade concrete; I'm happy to implement the coarse form instead of, or alongside, the fine one if that's preferred.

On the privilege concern, the repair never runs in the daemon: the hook is exec'd as a build user (the diff-hook pattern), and "repaired" is verified by a second privilege-dropped --check run before registration. What remains in-daemon is a small read-only detector, bounds-checked, fuzz-tested, and ASan/UBSan-clean, in the same untrusted-parsing risk class as NAR restoration and reference scanning; something has to decide to invoke the helper. Shipping the default payload with Nix, rather than relying on derivation-declared hooks, is what lets it cover nixpkgs revisions that are already published.

One disclosure and one wording note. The disclosure: an earlier revision of this branch could repair a content-addressed output and register a ca field that no longer matched its contents (re-importing such a path fails the ca-hash check); CA paths are now never repaired anywhere and refuse loudly instead. The real CA fix, hashing modulo signature slots as it is already modulo self-ref strings, changes every CA store path and is named in the body as future work, not fought here. The wording note: the docs now state the repair's property as determinism (same input bytes, same output bytes, only stale hash slots touched, every other byte preserved) rather than as identity with a clean rebuild, since the latter additionally depends on the package build itself being reproducible, as the fish note in the body already discussed.

Review-wise this is ~2.6k lines, but the shape is three independent doors sharing one parser; each commit stands alone if splitting the review (or the PR) is easier.

[xokdvium]

So a few things:

1. As mentioned above, baking in support for MACH-O into Nix is a NACK. The early bail is literally no more than 50 lines of code and I'd recommend going with that.
2. Focusing on just one instance of the problem isn't going to help us much. What if the redirected output path ends up in a compressed blob (think Java classes or whatever else).
3. We should be doing less rewriting - not more.
4. This seems to conflate architectural issues in hydra that lead to these frankenbuilds.

The issues in hydra are real, and there certainly things we can do in Nix to make easier to ban rewriting from happening.

I don't think we'll be going with this approach though. It's too much of unreviewable code with a dubious design.

Also please, can we dial it down a bit with the agents? These walls of text are surreal.

[ak2k]

I see that you admin-closed this. For context, I was trying to work toward what I thought was a consensus direction, and maybe I was trying to address too much in one PR. I hope the diagnostic contribution was at least useful.

One question, mainly for the users on NixOS/nixpkgs#507531/NixOS/nixpkgs#208951 rather than for this PR: what is the path for people hit by the artifacts already in the cache? Today's scan shows 437 broken slices across 48 packages in the live channels, and as @mroi reported, macOS 27 beta kills binaries that macOS 26 still tolerated. Of those, 94% are ad-hoc-signed and mechanically repairable; the opt-in substitution check and repair in this PR were aimed at exactly that set. On the prevention side, the S3 retry fix (#15855) is merged, but I could not find a PR or tracking issue for the other safeguards mentioned (withholding narinfos until all outputs upload, or the bail-out itself). Is there a tracked plan for the full set of producer-side fixes, and do we know that set is complete? None of it helps anyone substituting what is already in the cache, and the bail-out also cannot reach content-addressed cold builds, which stay silently broken.

Without a tracked plan for both halves, prevention and the existing cache, this stays broken for darwin users indefinitely, and the enforcement direction in macOS 27 makes that worse, not better.

[ak2k]

For future PRs, or just for posterity: assuming all producer problems are addressed (Hydra, bun, electron, the rest), a solution to this class seems to require covering:

Requirement	Who it affects	Covered by proposed early bail?
Rescue for the 437 broken slices already in cache (94% ad-hoc-signed and mechanically repairable)	anyone substituting them today; more become fatal as macOS 27 tightens enforcement	no: prevention does not reach artifacts already published
Rescue for broken paths already registered in local stores	anyone who substituted them before any fix lands	no
Content-addressed cold builds (#6065): every cold build of a self-referential signed binary	CA users; open since 2022	no: nothing is present at build start to bail on, and CA structurally requires the placeholder rewrite, so disallowing interim paths cannot apply either
--check / --rebuild of signed darwin binaries without a spurious "may not be deterministic" failure	anyone verifying reproducibility on darwin	no: all outputs are present, so a partial-outputs bail never fires