cctools: 973.0.1 -> 1010.6

[reckenrode]

Description of changes

This PR updates cctools and ld64. It also fixes related breakage. It is currently set to draft until 24.05 is released. I will be rebasing weekly against staging, resolving any merge conflicts then. Before updating the ld64 branch, I build my configs and the Darwin channel blockers to confirm no regressions.

The following notes are copied from https://discourse.nixos.org/t/darwin-updates-news/42249/10.

Packaging Changes

• cctools, ld64, and libtapi are now built from Apple’s OSS distributions repo instead of the cctools-port and apple-libtapi repos.
• These versions (cctools 1010.6, ld64 951.9, libtapi 1500.0.12.3) correspond to the versions shipped with Xcode 15. cctools and ld64 are current as of Xcode 15.3. Packages should no longer have to patch around missing flags or feature support.
• Restoring Linux support is a WIP. I have made some changes to use non-private and non-portable APIs, but I am currently focused on getting Darwin updates first.
• The packages have been moved to the by-name hierarchy. Aliases are provided to the old names, but there are some breaking changes (see below).
• cctools is now only cctools. This is technically a breaking change (see below). ld64 has been moved to its own, separate package.
• darwin.binutils is now the package that defines the mix of cctools and LLVM tools. It has been update with additional tools. See below for the breakdown of what it contains.
• The build system has been replaced with Meson. I did this because dealing with xcbuild stinks, and these packages eventually need to support Linux for cross-compilation.
• corecrypto and CommonCrypto hashing APIs are reimplemented using OpenSSL on both Darwin (and eventually) Linux. cctools-port only does this on non-Darwin platforms.
• libtapi has been split into separate out and lib outputs. Most packages only need libtapi.dylib, which is ~500 KiB compared to ~30 MiB for the tapi binary.
• libtapi is linked against zlib. I don’t know what it uses zlib to do, but the upstream libtapi is also linked against zlib, and I want to match what upstream is doing.

Note: ld-prime is will not be packaged because it is not currently included in the source releases. While I think it’s unlikely, ld-prime will be added separately as a new package should the source be released.

New Features

• LTO support. With this update, LTO works. I had to patch ld64 to look for libLTO.dylib in the store and clang to pass the correct path (see llvmPackages_{12,13,14,15,16,17,18,git}.clang: fix libLTO.dylib path #304350). I bootstrap ld64 and link it with itself with LTO enabled to make sure that LTO actually works.
• Code directory support. cctools is built with libcodedirectory support, which allows strip and install_name_tool to identify linker-created signatures and update them automatically. This removes the need for postLinkSignHook and actually makes it harmful to continue using it by default (because it prevents automatic signature updates by replacing them with a non-linker-created signature).

Breaking Changes

• ld64 and cctools use GCD (libdispatch) for parallelism. This should improve performance, but it technically ups the required macOS version to 10.14. The APIs it uses are present under different names in 10.12, so I have attempted to patch around them. It builds with the 10.12 SDK, but it will need testing to confirm the tools actually work. This was the driver of updating the default SDK on x86_64-darwin for 24.11.
• Apple removed support for nested static archives from ld64. This happened with the release of Xcode 14, so it has been out in the ecosystem for a few years. Hopefully packages have adapted by now. For example, Kyua adds libutils.a to libcli.a using libcli_a_LIBADD in automake. This will fail to link with ld64 951.9.
• ld64 is more picking about command-line arguments. Passing -rpath when merging Mach-O object files fails.
• GNU as has been moved to the cctools^gas output. -q and -Q flags are supported to control whether it calls GNU as or the clang assembler (matching the upstream behavior of as).
• cctools only provides binaries from cctools. ld64 is a separate package now.
• darwin.cctools-apple, darwin.cctools-port, and darwin.cctools-llvm are replaced by just cctools. cctools-llvm is an alias to darwin.bintools.
• The old names can’t be used in nixpkgs because aliases are disabled when ofborg eval is run. The PR updating the tools will include changes for all packages using the old names. (Updates for LTO will be done in a separate PR or PRs.)
• Use of postLinkSignHook has been removed from bintools-wrapper. Because it removes linker-created signatures, it breaks the automatic updating of signatures by strip and install_name_tool. Unless you’re manually invoking codesign, you shouldn’t be affected. install_name_tool will do the right thing by default.
• strip and install_name_tool are no longer wrapped with sigtool. If you are modifying linker-signed binaries, which should be the typical case, they will update the signatures automatically. If you are manually invoking codesign, you will need to update the signatures manually after running strip or install_name_tool.

stdenv updates

• Stage 2 has been collapsed from two parts into one stage. The source-based CF is not coming back. It adds a lot of complexity to the bootstrap.
• As noted in breaking changes, the Darwin stdenv no longer provides bintools using postLinkSignHook. Once the bootstrap tools are updated with these releases, it may be possible to drop sigtool completely from the bootstrap. Until that happens, early stages manually implement postLinkSignHook because the bootstrap tools ld64 does not appear capable of signing binaries on its own.
• As part of this update, I have to update the stdenv to use the new paths.
• The overall stdenv size is down by ~20 MiB due to dropping tapi from the closure. It also drops signtool and signingUtils, but those binaries are small.
• curl and GNU binutils throw when used during the bootstrap. curl should never be used during the bootstrap (to allow it to be updated without triggering a whole rebuild of Darwin). GNU binutils generally shouldn’t be used on Darwin (e.g., Rust 1.77 will clash harder with GNU binutils on Darwin Nix #299606), so the bootstrap shouldn’t need to use it.

darwin.bintools changes

darwin.bintools changes

• c++filter is no longer linked from GNU binutils. It is linked from LLVM. A handful of packages are linked from cctools because LLVM does not provided drop-in replacements for them. The lists of tools are exported as attributes on darwin.bintools (llvm_cmds and cctools_cmds respectively). All binaries are symlinked to their traditional names (e.g., llvm-strip is symlinked to strip).
  • LLVM tools: addr2line, ar, c++filt, dwarfdump, dsymutil, lipo, install-name-tool, nm, objcopy, objdump, otool, size, strings, strip
  • cctools tools: codesign_allocate, gprof, ranlib (and ld from ld64)
• Darwin is still isCCTools and not isLLVM because it does not use lld by default. Maybe that could change with lld 18, but I’m doubtful.

Resolved Issues

• LTO works. Closes Clang link time optimization not working on OSX #19098.
• ld64 no longer crashes linking Haskell applications. Closes cctools ld segfaults when linking haskellPackages.Agda (2.6.2 only?) #149692.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/darwin-updates-news/42249/12

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/need-help-from-darwin-users-syntax-errors-in-library-frameworks-foundation-framework-headers/30467/17

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/darwin-ld-symbol-s-not-found-for-architecture-arm64/46177/1

[reckenrode]

Latest push is up until right before the Darwin cross work.

[reckenrode]

I’m paring down this PR to reduce the chance of conflicts and to make it easier to review in aggregate. Cherry-picking commits is going to take too much time, and sometimes they depend on other commits.

[reckenrode]

The following will be done in separate PRs:

• libunwind upgrade;
• adding cctools to darwin-aliases.nix; and
• enable LTO in packages that have it disabled on Darwin.

[reckenrode]

The only thing I have left to do before marking this ready is fix the AvailabilityVersions package to patch out newer macros on SDKs 10.12, 10.13, 10.14, and 10.15. Some packages (like Git and SDL2) check for the presence of certain macros regardless of their value to check SDK version. That feels like a misuse of the macros, but I don’t want to break anything with the switch.

[toonn]

I've kicked off a build on 10.13 hardware to see if the changes to keep <10.14 working are alright, not sure if we want to wait for that to complete though.

[reckenrode]

I've kicked off a build on 10.13 hardware to see if the changes to keep <10.14 working are alright, not sure if we want to wait for that to complete though.

It should work, so I’d be surprised if it didn’t. Anyway, I’m currently waiting on approvals from you and @sternenseemann (or at least confirmation/feedback re: the revised comments).

[sternenseemann]

(or at least confirmation/feedback re: the revised comments).

Looks good.

[toonn]

LGTM, like I said no need to block on the build.
Had a couple failures related to the LLVM manpages but I haven't been able to check whether it's not just running out of disk space or such. So let's just go ahead with the merge already.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/darwin-updates-news/42249/15

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/breaking-changes-announcement-for-unstable/17574/57