libredirect: build fat library for x86_64, arm64, arm64e on darwin

[angerman]

libredirect currently fails to build on macOS machines. I've tried various combinations of BigSur and Monotery, it fails everywhere.

Our llvmPackages_13.clang doesn't seem to like being forced to create tiple arch files. So we'll resort to the system clang on darwin for now. This item is so fundamentally intwined with the impure state of the host system (dyld) over which we have no control, that I find this "hack" to be acceptable, while ugly.

/cc @thefloweringash @Ericson2314

[veprbl]

libredirect currently fails to build on macOS machines

What do you mean by that? It builds on Hydra:
https://hydra.nixos.org/job/nixpkgs/staging-next/libredirect.x86_64-darwin

[angerman]

libredirect currently fails to build on macOS machines

What do you mean by that? It builds on Hydra: https://hydra.nixos.org/job/nixpkgs/staging-next/libredirect.x86_64-darwin

that is x86_64-darwin. And the aarch64-darwin probably just builds by coincidence that the hydra builder use rather old BigSur versions? It appears somewhere during the BigSur updates a new dyld was introduced that didn't like the aarch64 (arm64 in apple speak), anymore and as such libredirect started failing to build on those machines.

[angerman]

$ nix-shell -p neofetch --run neofetch --argstr system aarch64-darwin                                                                       
                    'c.          angerman@aarch64-darwin-1
                 ,xNMM.          -------------------------
               .OMMMMo           OS: macOS 12.1 21C52 arm64
               OMMM0,            Host: Macmini9,1
     .;loddo:' loolloddol;.      Kernel: 21.2.0
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 16 hours, 19 mins
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 176 (nix-system)
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: bash 5.1.8
;MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua
:MMMMMMMMMMMMMMMMMMMMMMMM:       WM: Quartz Compositor
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM Theme: Blue (Light)
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    Terminal: /dev/ttys001
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   CPU: Apple M1
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   GPU: Apple G13G
    kMMMMMMMMMMMMMMMMMMMMMMd     Memory: 1434MiB / 16384MiB
     ;KMMMMMMMWXXWMMMMMMMk.
       .cooc,.    .,coo:.

Getting a cached build from hydra works (well, "works", it pulls the binary from cache...)

$ nix-build "<nixpkgs>" -A libredirect -I nixpkgs=https://github.com/nixos/nixpkgs/archive/nixpkgs-21.11-darwin.tar.gz
/nix/store/19j8my6waz6agi4247jx3q23lx3wqx69-libredirect-0

trying to force a build on the above machine:

$ nix-build "<nixpkgs>" -A libredirect -I nixpkgs=https://github.com/nixos/nixpkgs/archive/nixpkgs-21.11-darwin.tar.gz --check
checking outputs of '/nix/store/n2jajggzh9rm02dkxdh2s65x5xsdycgn-libredirect-0.drv'...
unpacking sources
patching sources
updateAutotoolsGnuConfigScriptsPhase
configuring
no configure script, doing nothing
building
libredirect.c:102:11: warning: incompatible function pointer types initializing 'int (*)(const char *, int, mode_t)' (aka 'int (*)(const char *, int, unsigned short)') with an expression of type 'int (*)(const char *, int, ...)' [-Wincompatible-function-pointer-types]
    int (*open_real) (const char *, int, mode_t) = LOOKUP_REAL(open);
          ^                                        ~~~~~~~~~~~~~~~~~
libredirect.c:107:27: warning: second argument to 'va_arg' is of promotable type 'mode_t' (aka 'unsigned short'); this va_arg has undefined behavior because arguments will be promoted to 'int' [-Wvarargs]
        mode = va_arg(ap, mode_t);
                          ^~~~~~
/nix/store/7sfdssd4p5c6gh5dr2j1fy06fxky195q-clang-wrapper-11.1.0/resource-root/include/stdarg.h:19:50: note: expanded from macro 'va_arg'
#define va_arg(ap, type)    __builtin_va_arg(ap, type)
                                                 ^~~~
libredirect.c:134:11: warning: incompatible function pointer types initializing 'int (*)(int, const char *, int, mode_t)' (aka 'int (*)(int, const char *, int, unsigned short)') with an expression of type 'int (*)(int, const char *, int, ...)' [-Wincompatible-function-pointer-types]
    int (*openat_real) (int, const char *, int, mode_t) = LOOKUP_REAL(openat);
          ^                                               ~~~~~~~~~~~~~~~~~~~
libredirect.c:139:27: warning: second argument to 'va_arg' is of promotable type 'mode_t' (aka 'unsigned short'); this va_arg has undefined behavior because arguments will be promoted to 'int' [-Wvarargs]
        mode = va_arg(ap, mode_t);
                          ^~~~~~
/nix/store/7sfdssd4p5c6gh5dr2j1fy06fxky195q-clang-wrapper-11.1.0/resource-root/include/stdarg.h:19:50: note: expanded from macro 'va_arg'
#define va_arg(ap, type)    __builtin_va_arg(ap, type)
                                                 ^~~~
4 warnings generated.
installing
install: creating directory '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0'
install: creating directory '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib'
'libredirect.dylib' -> '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib'
post-installation fixup
patching script interpreter paths in /nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0
patching script interpreter paths in /nix/store/cbzgblyik90wag4arznwrw5sx62xfqsa-libredirect-0-hook
running install tests
dyld[53552]: terminating because inserted dylib '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib' could not be loaded: tried: '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'arm64e')), '/usr/local/lib/libredirect.dylib' (no such file), '/usr/lib/libredirect.dylib' (no such file)
dyld[53552]: tried: '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'arm64e')), '/usr/local/lib/libredirect.dylib' (no such file), '/usr/lib/libredirect.dylib' (no such file)
Assertion failed: (system(TESTPATH) == 0), function test_system, file test.c, line 37.
/nix/store/i97a6fwzdp2n2mb9crzhnvdb7r15arqi-stdenv-darwin/setup: line 1355: 53550 Abort trap: 6           NIX_REDIRECTS="/foo/bar/test=/nix/store/dwhdayp98xmpp9vzzb1qp4va1pjy65gz-coreutils-9.0/bin/true" ./test
error: builder for '/nix/store/n2jajggzh9rm02dkxdh2s65x5xsdycgn-libredirect-0.drv' failed with exit code 134;
       last 10 log lines:
       > install: creating directory '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib'
       > 'libredirect.dylib' -> '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib'
       > post-installation fixup
       > patching script interpreter paths in /nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0
       > patching script interpreter paths in /nix/store/cbzgblyik90wag4arznwrw5sx62xfqsa-libredirect-0-hook
       > running install tests
       > dyld[53552]: terminating because inserted dylib '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib' could not be loaded: tried: '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'arm64e')), '/usr/local/lib/libredirect.dylib' (no such file), '/usr/lib/libredirect.dylib' (no such file)
       > dyld[53552]: tried: '/nix/store/27qmzv86dlmr9pibvmc6mb28n1f987qy-libredirect-0/lib/libredirect.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'arm64e')), '/usr/local/lib/libredirect.dylib' (no such file), '/usr/lib/libredirect.dylib' (no such file)
       > Assertion failed: (system(TESTPATH) == 0), function test_system, file test.c, line 37.
       > /nix/store/i97a6fwzdp2n2mb9crzhnvdb7r15arqi-stdenv-darwin/setup: line 1355: 53550 Abort trap: 6           NIX_REDIRECTS="/foo/bar/test=/nix/store/dwhdayp98xmpp9vzzb1qp4va1pjy65gz-coreutils-9.0/bin/true" ./test
       For full logs, run 'nix log /nix/store/n2jajggzh9rm02dkxdh2s65x5xsdycgn-libredirect-0.drv'.

[veprbl]

# $(nix-build . -A llvmPackages_13.clang.cc --no-out-link)/bin/clang -target arm64e-apple-darwin test.c -c -o test_arm64e.o
# $(nix-build . -A llvmPackages_13.clang.cc --no-out-link)/bin/clang -target arm64-apple-darwin test.c -c -o test_arm64.o  
# $(nix-build . -A llvmPackages_13.clang.cc --no-out-link)/bin/clang -target x86_64-apple-darwin test.c -c -o test_x86_64.o
# lipo -create test_arm64.o test_arm64e.o test_x86_64.o -output test.a 
# file test.a
test.a: Mach-O universal binary with 3 architectures: [x86_64:\012- Mach-O 64-bit x86_64 object, flags:<|SUBSECTIONS_VIA_SYMBOLS>] [arm64:\012- Mach-O 64-bit arm64 object, flags:<|SUBSECTIONS_VIA_SYMBOLS>] [arm64e (caps: 0x2):\012- Mach-O 64-bit arm64e 00) object, flags:<|SUBSECTIONS_VIA_SYMBOLS>]

[angerman]

# $(nix-build . -A llvmPackages_13.clang.cc --no-out-link)/bin/clang -target arm64e-apple-darwin test.c -c -o test_arm64e.o
# $(nix-build . -A llvmPackages_13.clang.cc --no-out-link)/bin/clang -target arm64-apple-darwin test.c -c -o test_arm64.o  
# $(nix-build . -A llvmPackages_13.clang.cc --no-out-link)/bin/clang -target x86_64-apple-darwin test.c -c -o test_x86_64.o
# lipo -create test_arm64.o test_arm64e.o test_x86_64.o -output test.a 
# file test.a
test.a: Mach-O universal binary with 3 architectures: [x86_64:\012- Mach-O 64-bit x86_64 object, flags:<|SUBSECTIONS_VIA_SYMBOLS>] [arm64:\012- Mach-O 64-bit arm64 object, flags:<|SUBSECTIONS_VIA_SYMBOLS>] [arm64e (caps: 0x2):\012- Mach-O 64-bit arm64e 00) object, flags:<|SUBSECTIONS_VIA_SYMBOLS>]

That's much better! I got confused why it didn't like my -arch logic.

[angerman]

@veprbl thanks! I'll run some validation on a few aarch64-darwin machines.

[angerman]

@veprbl alright, I've finally managed to find the incantation to make this build with the nixpkgs provided tools. And I've also learned that we can use -arch with the llvmPackages clang.

[angerman]

@veprbl hope this is good to go now. I did rebase onto staging, and now it appears I have to rebuild the world again. Sigh.

[angerman]

This will need a backport to 21.11

[angerman]

@veprbl ping

[veprbl]

Diff LGTM

[angerman]

@veprbl who can merge this? I don't seem to be able to.

[veprbl]

@veprbl who can merge this? I don't seem to be able to.

Currently only members of the "Nixpkgs committers" group can merge PRs. I wanted to leave this open for a day to see if there will be any additional comments. Note that this will have to go through the staging process before it lands on master, which may take another 3 to 4 weeks.

[angerman]

Yes. I'm aware that this will take quite some time. I just don't want to have to maintain my custom nixpkgs-21.11-darwin branch any longer than absolutely necessary.

[github-actions]

Successfully created backport PR #153917 for staging-21.11.

[Ericson2314]

Out of curiousity, why does this library need to be fat?

[angerman]

@Ericson2314 so depending on the macOS version you end up on, and we don't really have control over the macOS version the end user ends up using, dyld will expect arm64, arm64e or x86_64 (though we could argue for x86_64, that this is unlikely with the guard; however better safe than sorry).

So we'd better build a fat library with all slices, to insulate us from the underlying system changing requirements in dyld.

This did build flawlessly on an old BigSur revision, that nixOS builders run. It failed on more recent BigSur builder and Monterey builders that my hydra runs, as they expect different architectures (most likely arm64e; however I also believe that BigSurs dyld has a bug in which non-far dylibs are erroneously rejected).

[Ericson2314]

@angerman is that true for any library, or is this library somehow special?

[angerman]

@Ericson2314 so I haven't played much with libredirect and only ran into this as ti was a dependency. Again something just incidental to what I'm doing. I hope this is only with this one and due to the prelaod/insertion logic for dyld. The faketime package has similar issues, and it's the only other reference I could find online. (Though they didn't figure out how to solve it 🙄)

[Ericson2314]

Oh I see, the issue is that is messing with dyld itself with how it works? and due to Rosetta and other things need to cope with dyld being different arches?

[angerman]

@Ericson2314 it's more like the dyld is dependent on the operating system. And an update to it can introduce slight changes to dyld, as in wanting arm64e isntead of arm64 now.

It appears as:

• early versions of BigSur were ok with arm64 only
• early version of Monterey were ok with arm64e only
• late versions of BigSur, didn't like arm64 only anymore, but didn't like arm64e only either, they seem to be ok with a fat library containing arm64 and arm64e though (the codepaths for fat and non-fat libraries in dyld are different).

So I've opted to just go for all possible architectures to cover all our bases here.

[Ericson2314]

OK. So perhaps x86_64 doesn't matter as rosetta stuff doesn't come into play but whatever, once were are doing fat-binary stuff might as well go all in. Thanks for explaining to me.