apple-sdk_15: init at 15.0#347862
Conversation
This was omitting the SDK version from the structure, probably due to refactors in response to my own reviews on the SDK rework PR. Oops!
|
Oh, I just now figured out the script output. It’s a combination of both: it actually is grouped like I originally thought, but it’s listed by Xcode version, and Xcode 16.0 is the first version that shipped the macOS 15.0 SDK. Okay, phew, mystery solved. But hopefully we can still get a automated pipeline that can download the right catalogue and map the Xcode version to the appropriate SDK version. |
toonn
left a comment
toonn
left a comment
There was a problem hiding this comment.
LGTM
Quite an amazing improvement required-effort wise.
Maybe you can slap most of the contents of your OP in that empty README so there's at least minimal documentation going forward? (Even just a link to this PR would be enough for now.)
ofborg
bot
added
8.has: package (new)
|
I’ll work on the documentation this weekend. @emilazy is correct that the script is listing Xcode version. Xcode version and SDK version are loosely coupled. Usually, the Xcode major is one higher than the SDK version, but the minor version is only loosely related. I plan to include a link to https://developer.apple.com/documentation/xcode-release-notes in the stdenv documentation and the SDK readme to help people interpret Xcode versions. The stdenv may also incorporate a table, but I don’t know yet, since it hasn’t been written (and it would need to be maintained). Regarding the |
| packageHash=$(nix --extra-experimental-features nix-command hash path "$package-$packageTag") | ||
|
|
||
| pkgsjson="{\"$package\": {\"version\": \"$packageVersion\", \"hash\": \"$packageHash\"}}" | ||
| pkgsjson="{\"$sdkVersion\": {\"$package\": {\"version\": \"$packageVersion\", \"hash\": \"$packageHash\"}}}" |
There was a problem hiding this comment.
Probably a copy-paste victim of removing cat <<-EOF.
|
The documentation will also cover |
reckenrode
left a comment
reckenrode
left a comment
There was a problem hiding this comment.
I tested adding the SDK in my branch while working on #346043, so this LGTM.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
4 participants
In Nixpkgs, we’ve become used to the idea that adding a new macOS SDK version is a major undertaking that can take weeks or months. Indeed, before July we hadn’t added any in years (though shout‐outs to @ConnorBaker for #229210). As #346043 was being worked on, I set myself a challenge in coordination with @reckenrode: after it was merged, I would try to add the new macOS 15 SDK to Nixpkgs by myself, with no assistance from Randy, and see how long it took.
The answer is 70 minutes to have it working, tested, and used in the tree, and most of that was avoidable. That’s huge. I had an unfair advantage, of course, having been following the process from the beginning and having read the PR in full, but I had never tried to run one of the scripts, there is not yet any documentation to help me when it goes wrong, and I’m also pretty tired after a long day getting the rework PR tested and merged. The bulk of my time was spent waiting on unnecessary downloads that I had to restart, fixing my own silly mistakes, and getting confused by minor bugs and infelicities in the update scripts; I think that when it comes time to add the macOS 16 SDK in a year’s time, the same process should take no more than 10 minutes. Read on if you’d like a play‐by‐play and my takeaways.
Experience report
Before I started, I knew I’d need a program that used a macOS 15 API to test with, so I assembled the following basic package that computes √2 as a half‐precision float, using the new
__sqrtf16API:I confirmed that it build and ran as expected with the Xcode toolchain. As expected, the Nix package failed to build with the default SDK version (currently 10.12 on
x86_64-darwin):So I had my shiny new software that wants fresh SDK features; my goal was to add the new SDK and get it building in as little time as possible. I started the clock and went looking in
pkgs/by-name/ap/apple-sdkfor the appropriate scripts. There’s ascriptsdirectory withget-sdks-from-catalog.sh,lock-sdk-deps.sh, andregenerate-lockfile.sh, and after a glance at all three, it seemed likeget-sdks-from-catalog.shwas a good starting point.It wanted to be fed an Apple software update catalogue file; I looked again at the other two scripts since I figured there would be something to automatically download that for me, but nope. I remembered that swscan.apple.com is famously hard to find things in and that Randy had mentioned a Gist where he found the URLs for stuff in the past. A quick search of the Matrix room logs brought up https://gist.github.com/meyer/b14c87d162366f0428a99cd2ff0d0b8b, with a comment helpfully pointing to the correct link for macOS 15. I downloaded the catalogue and gave it to the script.
It gave me the following output:
Okay, clearly the 15.3 SDK is at https://swcdn.apple.com/content/downloads/14/48/052-59890-A_I0F5YGAY0Y/p9n40hio7892gou31o1v031ng6fnm9sb3c/CLTools_macOSNMOS_SDK.pkg, that looks great. (A twinge of doubt in my head: didn’t macOS 15.0 just come out? Are we really on the 15.3 SDK already? But I could worry about that later; the clock is ticking.)
I remembered that the SDK versions were all stored in a JSON file. I looked around briefly for tooling to automatically add the new one, but couldn’t find any. No matter – a quick
nix store prefetch-fileand I had the new 15.3 SDK version manually added tometadata/versions.json. I then added theapple-sdk_15line toall-packages.nix. At this point it was about 8 minutes since I started.Okay, job done, right? Well, no; there’s those other two scripts. Looking at them revealed that they were there to handle pinning the corresponding source releases.
regenerate-lockfile.shis a wrapper aroundlock-sdk-deps.shthat automates running it over all the SDK versions and a bunch of packages, so that seemed like the thing to run. It took about ten minutes redownloading the source releases for all the previous SDK versions.At that point the script failed because apparently the
macos-153tag in https://github.com/apple-oss-distributions/distribution-macOS doesn’t exist, which indeed it doesn’t. Uh oh – I know that Apple are often slow to release corresponding source releases. There’s amacos-150tag, though, so maybe we can just use that? It seemed like the only way to override it was to adjust the version inmetadata/versions.jsonand run the script again; another ten minutes down the drain. About half‐way through, I looked at themetadata/apple-oss-lockfile.jsonfile it was writing to, and saw that it had removed the outer layer of SDK versions from the JSON structure. I remembered being an annoying pedant about Randy’sjquse in these scripts – did it break in the process of responding to that review feedback? Yep, it did. I fixed the script to add the SDK version back to the JSON structure and ran it again. Success! A quick switch of the SDK version back to 15.3 and the packaging should be done.Now for the moment of truth: I added the new SDK to the build inputs and hoped for the best.
…and it failed with an arcane message about
cp: missing destination file operand after '/nix/store/65ifh4l4h4vpiv85nsv6w77js32rhrks-macOS-SDK-15.3'. Um, a messed up glob, maybe? Consulting the derivation told me that it was extracting the package and looking for the appropriate versioned SDK inside it. I downloaded and extracted it manually to look at and… uh, it’s the 14.3 SDK?Did I misread the output of that catalogue‐reading script? Maybe that’s why it’s 15.3 instead of 15.0? The answer is yes, I misread the output, but no, that doesn’t explain the version thing. I thought the output was grouped like this:
but it’s actually like this:
I replaced the URL with the correct one and, after a slight mishap with the hash (because
nix store prefetch-filedoesn’t name its store pathsourcelike the Nixpkgs fetchers do), started building the SDK again. It failed with the same error. I downloaded and extracted the new.pkg, and… the SDK version inside is actually 15.0? I still don’t really understand where the 15.3 is coming from. I know Xcode and SDK versions aren’t the same, but it doesn’t look like Xcode 15.3 shipped with the macOS 15 SDK. So, I don’t know; I’ll need to wait for Randy to explain that one to me. But I put the version back to 15.0, and the SDK started actually building this time. Yay! I was getting impatient at this point because of this taking longer than I expected, and it made me think dangerous thoughts about how I should parallelize the.tbdconversion process. But, just over an hour in, it successfully built my test package, and I ran it on the community builder I was using to do the builds (because it already had thestdenvforx86_64-darwincached from my earlier testing of the rework PR):I’ve never been so happy to see a frowning face. This is the correct result: the community builder is running 14.7, and the SDK to build with is decoupled from the minimum deployment target we require at runtime (currently 10.12 for
x86_64-darwinand 11.0 foraarch64-darwin), so this is the runtime check I wrote being handled correctly. I downloaded the binary locally, and successfully computed the world’s most accurate approximation of the square root of two:Now I wanted to test the hook to set the deployment target, which is required for software that doesn’t do dynamic checks or unconditionally requires newer APIs. Thankfully, that’s also really easy with the new scheme:
The resulting binary continued to work on my local 15.0 machine, but correctly failed to load in spectacular fashion when run on the too‐old community builder:
At that point, the job was done, but I remembered from reviewing the PR that there was one package of an Apple source release that was currently fetching a private header version from GitHub because of not yet having the macOS 15 SDK packaged. A quick ripgrep and I edited
pkgs/os-specific/darwin/apple-source-releases/libpcap/package.nixto comment out theapple-sdk_15.sourceReleaseline that was waiting for me. I built the resulting package, and around 70 minutes from when I started, the job was done.Well, except for reorganizing the commits and writing this extremely long pull request message, of course.
Conclusion
70 minutes is incredible! Thanks to Randy, we should never again be in a position where we need a newer SDK but don’t have it. The fact that the diff here is so tiny (seriously – go look at it!) and most of the 70 minutes were spent waiting for downloads and dealing with the scripts not yet being fully polished is a massive vindication of the general approach. So I don’t at all want to seem critical here, especially given that we had to rush all of this right before the release, but I think it’s helpful to note down areas for future improvement here.
.tbdconversion process was parallelized so that the SDKs built faster. (It still only takes a few minutes, though.)Hopefully I can tackle some of these soon, and possibly just rewrite the scripts in something other than Bash. But when you consider how much toil and suffering past attempts to package new macOS SDKs have resulted in, it’s incredible how minor these nitpicks are. The process is almost entirely automated now, and as soon as I got the correct SDK URL downloading with the correct version recorded, the SDK package built and worked the very first time. The future looks bright.
I only tested this on
x86_64-darwinsince I didn’t want to build anotherstdenv. I’m counting on you to testaarch64-darwinand tell me what I messed up, Randy :)Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.