stdenv.darwin: bootstrap darwin using updated tools (resurrected)

[wegank]

Resurrection of #301252. All credits go to annalee.

Description of changes

• update the hashes and tools needed to extract the bootstrap-tools archive

• unify the x64 and aarch64 unpack process

• second try of stdenv.darwin: bootstrap darwin using updated tools #295558 but after xz downgrade to 5.4.2

• considered downgrading xz further for tools to 5.2.5 so not have to update again as 5.4.2 has commits from malicious committer but stuck with xz version in nixpkgs

• considered using zstd / gzip for compression of tarball / unpack.nar but nix doesn't support gz for nar files

tools from:

• https://hydra.nixos.org/build/255282138
• https://hydra.nixos.org/build/255281731

testing:

maintainers/scripts/bootstrap-files/refresh-tarballs.bash --targets=aarch64-apple-darwin,x86_64-apple-darwin
for sys in aarch64 x86_64; nix-build  -A stdenv --system ${sys}-darwin

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[wegank]

aarch64-apple-darwin

sha256sum of files to be uploaded:

sha256sum /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/*

6b7ece80c7531e31722ace144f1a03ef3c7a4dedc0ef3b128e16c51f36cdc50f  /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
8b267b5946822fe4037be198d3c23ee6c2b7eec7fbeb7413c36e04854bf4da25  /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/unpack.nar.xz

Suggested commands to upload files to 'tarballs.nixos.org':

nix-store --realize /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools
aws s3 cp --recursive --acl public-read /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f
aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f ./
sha256sum bootstrap-tools.tar.xz unpack.nar.xz

x86_64-apple-darwin

sha256sum of files to be uploaded:

sha256sum /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/*

dce06283395bbbf67a83cafb86c39683de474efec8270f4dbdb6a6c2fc3ef3cc  /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
53756aecf5ea3765034f39a3d8a780ae5ddc4b89570beb56eea01ffdd7c9fb96  /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/unpack.nar.xz

Suggested commands to upload files to 'tarballs.nixos.org':

nix-store --realize /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools
aws s3 cp --recursive --acl public-read /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f
aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f ./
sha256sum bootstrap-tools.tar.xz unpack.nar.xz
sha256sum /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/*

[reckenrode]

This PR is identical to the other one, so LGTM.

[wegank]

ping @lovesegfault for tools update (thanks!): info here #302387 (comment)

no xz backdoors in this round:

$ git checkout d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f
HEAD is now at d03a4482228d

$ nix-build -A freshBootstrapTools.bootstrapFiles
/nix/store/38ivab6cy7awrz8ni7csm78wyyvzly7f-unpack
$ /nix/store/38ivab6cy7awrz8ni7csm78wyyvzly7f-unpack/bin/xz --version
xz (XZ Utils) 5.4.6
liblzma 5.4.6

$ nix-build -A freshBootstrapTools.bootstrapFiles --argstr system x86_64-darwin
/nix/store/xy5swsli302v0c0qzwfbh31y24dbjpkw-unpack
$ /nix/store/xy5swsli302v0c0qzwfbh31y24dbjpkw-unpack/bin/xz --version
xz (XZ Utils) 5.4.6
liblzma 5.4.6

[lovesegfault]

Uploaded:

aarch64-apple-darwin

$ nix-store --realize /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools
/nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools

$ sha256sum /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/*
6b7ece80c7531e31722ace144f1a03ef3c7a4dedc0ef3b128e16c51f36cdc50f  /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
8b267b5946822fe4037be198d3c23ee6c2b7eec7fbeb7413c36e04854bf4da25  /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/unpack.nar.xz

$ aws s3 cp --recursive --acl public-read /nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f
upload: ../../../../../nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/unpack.nar.xz to s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/unpack.nar.xz
upload: ../../../../../nix/store/zxxcz7sza5ypy061rsg7cmr7h0a96hbb-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz to s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/bootstrap-tools.tar.xz

$ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f ./
download: s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/unpack.nar.xz to ./unpack.nar.xz
download: s3://nixpkgs-tarballs/stdenv/aarch64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/bootstrap-tools.tar.xz to ./bootstrap-tools.tar.xz

$ sha256sum bootstrap-tools.tar.xz unpack.nar.xz
6b7ece80c7531e31722ace144f1a03ef3c7a4dedc0ef3b128e16c51f36cdc50f  bootstrap-tools.tar.xz
8b267b5946822fe4037be198d3c23ee6c2b7eec7fbeb7413c36e04854bf4da25  unpack.nar.xz

x86_64-apple-darwin

$ nix-store --realize /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools
this path will be fetched (90.73 MiB download, 299.57 MiB unpacked):
  /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools
copying path '/nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools' from 'https://cache.nixos.org'...
/nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools

$ sha256sum /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/*
dce06283395bbbf67a83cafb86c39683de474efec8270f4dbdb6a6c2fc3ef3cc  /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
53756aecf5ea3765034f39a3d8a780ae5ddc4b89570beb56eea01ffdd7c9fb96  /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/unpack.nar.xz

$ aws s3 cp --recursive --acl public-read /nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f
upload: ../../../../../nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/unpack.nar.xz to s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/unpack.nar.xz
upload: ../../../../../nix/store/lsl9rl3zj9nr318w471vvmlvxzj21b2k-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz to s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/bootstrap-tools.tar.xz

$ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f ./
download: s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/unpack.nar.xz to ./unpack.nar.xz
download: s3://nixpkgs-tarballs/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/bootstrap-tools.tar.xz to ./bootstrap-tools.tar.xz

$ sha256sum bootstrap-tools.tar.xz unpack.nar.xz
dce06283395bbbf67a83cafb86c39683de474efec8270f4dbdb6a6c2fc3ef3cc  bootstrap-tools.tar.xz
53756aecf5ea3765034f39a3d8a780ae5ddc4b89570beb56eea01ffdd7c9fb96  unpack.nar.xz