aarch64: update bootstrap-files by dguibert · Pull Request #151399 · NixOS/nixpkgs

dguibert · 2021-12-20T09:18:56Z

This PR fixes #112086.

The bootstrap files have been updated from these jobs on hydra:

https://hydra.nixos.org/job/nixpkgs/trunk/stdenvBootstrapTools.aarch64-linux.dist ->
https://hydra.nixos.org/build/160330465#tabs-buildinputs

These commands have been run to get the output hashes as FODs:

nix store add-file --name bootstrap-tools.tar.xz  $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/bootstrap-tools.tar.xz
nix store add-path --name busybox                 $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/busybox

Motivation for this change

Things done

vcunat · 2021-12-20T09:38:11Z

Well, for aarch64-linux we have native builders on Hydra, so I expect we'd prefer to take the native bootstrap tools. (I didn't realize this difference when pointing you to risc-v.) https://hydra.nixos.org/job/nixpkgs/trunk/stdenvBootstrapTools.aarch64-linux.dist probably

fixes NixOS#112086 Fetch bootstrap files built by Hydra and re-add them as FODs to skip currently dangling URLs. They are already available on https://cache.nixos.org https://hydra.nixos.org/job/nixpkgs/trunk/stdenvBootstrapTools.aarch64-linux.dist -> https://hydra.nixos.org/build/160330465#tabs-buildinputs These commands have been run to get the output hashes as FODs: nix store add-file --name bootstrap-tools.tar.xz $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/bootstrap-tools.tar.xz nix store add-path --name busybox $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/busybox

dguibert · 2021-12-20T09:57:35Z

Ok. I've updated the PR with the native hydra job.

tpwrules · 2021-12-24T03:06:07Z

I was able to use this PR to build a minimal system on an aarch64 machine with 16k pages. Before it was impossible to change the stdenv because the bootstrap tools would segfault. Thanks for fixing this.

vcunat

I re-checked the URL and hashes, and also building some stuff (though it's only 4k-page machine).

@lovesegfault: we only need to upload this pair of files, again.

dguibert · 2021-12-24T12:39:06Z

I re-checked the URL and hashes, and also building some stuff (though it's only 4k-page machine).

Great.
I knew about aarch64 build box for long time but I took time just now to ask access: NixOS/aarch64-build-box#122

lovesegfault · 2022-01-03T18:16:50Z

Happy to help with uploading the new bootstrap tarball!

I need:

The commit hash that was used to generate the bootstrap tools
The store path of the bootstrap tools
The sha256sum of all the on-server components, e.g.:

$ sha256sum /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/*
6fae5d5db226ea8eace94f2d0221a947a48c7ef7e7fd515c6604c705eb49662b  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/bootstrap-tools.cpio.bz2
78df72595b11e55ab198a6508cd9fa6702baee33e9631d6f0614cf08052f88f3  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/bzip2
467a81602423f7f8e73b59b5ae080235795aebbbcb19a4fb81c9706153615394  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/cpio
6f9673b2e44bfa524068965da0ee5520f2c27f02f7dc3585ffe828356fb46eb1  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/mkdir
865d209fe8fb0fb457983f0208bb5febe87990b800a378ecf5f1a1cd5574b65b  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/sh

(I use this last bit to double check that the intended thing is being uploaded)

dguibert · 2022-01-04T13:54:51Z

Happy to help with uploading the new bootstrap tarball!

Great.

I need:

1. The commit hash that was used to generate the bootstrap tools

c7c997a0662bf88264db52cbc41e67884eb7a1ff

2. The store path of the bootstrap tools

/nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools (from /nix/store/0azgpcnws27pva4m5spp1kfmmghf40gx-stdenv-bootstrap-tools/nix-support/hydra-build-products)

3. The `sha256sum` of all the `on-server` components, e.g.:

$ sha256sum /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/*
0233af99a5bc24555968149150c2ae7ebf64268ce0fedb19afb3ef504050ca2e  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
9c9eba5f65dfb12b0ee2f97a34bcda159d4e521593c323e70e95dd6b4dbfe49d  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/busybox

lovesegfault · 2022-01-04T14:30:45Z

Done!

$ pushd ~/src/nixpkgs; git show c7c997a0662bf88264db52cbc41e67884eb7a1ff; popd                    
~/src/nixpkgs ~/tmp/bootstrap
commit c7c997a0662bf88264db52cbc41e67884eb7a1ff
Merge: 571cbf3d1db e2738be1365
Author: maralorn <malte.brandy@maralorn.de>
Date:   Fri Dec 3 01:27:14 2021 +0100

    Merge pull request #148357 from maralorn/nom-update
    
    nix-output-monitor: 1.0.3.3 -> 1.0.4.0

~/tmp/bootstrap

$ nix build /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools                                 

$ sha256sum /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/*
0233af99a5bc24555968149150c2ae7ebf64268ce0fedb19afb3ef504050ca2e  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
9c9eba5f65dfb12b0ee2f97a34bcda159d4e521593c323e70e95dd6b4dbfe49d  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/busybox

$ aws s3 cp --recursive --acl public-read /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff                          
upload: ../../../../nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/busybox to s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/busybox
upload: ../../../../nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz to s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/bootstrap-tools.tar.xz

$ aws s3 ls s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/
2022-01-04 11:27:11   16156444 bootstrap-tools.tar.xz
2022-01-04 11:27:11     177608 busybox

$ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/ ./                                                                                            
download: s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/busybox to ./busybox
download: s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/bootstrap-tools.tar.xz to ./bootstrap-tools.tar.xz

$ sha256sum ./bootstrap-tools.tar.xz ./busybox                                            
0233af99a5bc24555968149150c2ae7ebf64268ce0fedb19afb3ef504050ca2e  ./bootstrap-tools.tar.xz
9c9eba5f65dfb12b0ee2f97a34bcda159d4e521593c323e70e95dd6b4dbfe49d  ./busybox

vcunat · 2022-01-04T19:09:39Z

I tried to build some stuff. No use waiting anymore.

tpwrules · 2022-01-04T19:28:08Z

I cleaned up my 16k page aarch64 machine and was able to run nixpkgs-review pr 151399 -p hello. Looks like everything was uploaded correctly and the full stdenv bootstrap works (along with hello of course).

Looking forward to seeing this land in master! Thanks everyone for tackling this.

@lovesegfault

This PR updates the bootstrap tarballs for riscv64-linux with new Hydra-generated ones. Fixes NixOS#275848 (bootstrap assembler too old to build gcc 13). I'll be following the script used in NixOS#151399, NixOS#168199, NixOS#183487, and NixOS#188334. Files came from [this](https://hydra.nixos.org/build/246376732#tabs-summary) Hydra build, which used nixpkgs revision 160cedc to instantiate: ``` /nix/store/cpiajh4l83b08pynwiwkpxj53d78pcxr-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu.drv ``` and then built: ``` /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu ``` I downloaded these files from Hydra and prefetched them into the nix store with the following commands: ``` STOREPATH=8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" nix store prefetch-file \ file://$(nix store add-file --name bootstrap-tools.tar.xz $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz) nix store prefetch-file --executable \ file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox) ``` These commands produced the following output: ``` Downloaded 'file:///nix/store/xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' to '/nix/store/3fal4gikp92013kac6rdmfbrch2s859b-xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' (hash 'sha256-0LxRd7fdafQezNJ+N2tuOfm0KEwgfRSts5fhP0e0r0s='). Downloaded 'file:///nix/store/9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' to '/nix/store/kb7wyy30y1gxcmdajljr26kxxac606qa-9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' (hash 'sha256-OGO96QUzs2n5pGipn/V87AxzUY9OWKZl417nE8HdZIE='). ``` I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit. I then started the bootstrap with the following command: ``` nix build -L -f . --arg localSystem '(import ./lib).systems.examples.riscv64' hello ``` As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification: ``` sha256sum /nix/store/${STOREPATH}/on-server/* ``` which produced the following output: ``` d0bc5177b7dd69f41eccd27e376b6e39f9b4284c207d14adb397e13f47b4af4b /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz 65f9433abb598f63c932d33351b14f686551512b1cece1e64c2d0e76aa0ec52e /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/busybox ```

github-actions bot added the 6.topic: stdenv Standard environment label Dec 20, 2021

dguibert mentioned this pull request Dec 20, 2021

fix bootstrap tools on aarch64 64k page size system #150659

Closed

13 tasks

dguibert requested a review from vcunat December 20, 2021 09:20

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Dec 20, 2021

dguibert force-pushed the dg/new-bootstrap-tools-aarch64 branch from dd9010a to 8e7acfc Compare December 20, 2021 09:56

vcunat approved these changes Dec 24, 2021

View reviewed changes

vcunat merged commit 0b7de2a into NixOS:staging Jan 4, 2022

dguibert deleted the dg/new-bootstrap-tools-aarch64 branch January 4, 2022 21:51

dguibert restored the dg/new-bootstrap-tools-aarch64 branch January 4, 2022 21:51

dguibert deleted the dg/new-bootstrap-tools-aarch64 branch January 4, 2022 21:51

dguibert restored the dg/new-bootstrap-tools-aarch64 branch January 12, 2022 09:44

ghost mentioned this pull request Apr 11, 2022

pkgs/stdenv/linux: add powerpc64le bootstrap-files #168199

Merged

4 tasks

ghost mentioned this pull request Jul 5, 2022

pkgs/stdenv/linux: add mips64el bootstrap-files #180223

Merged

3 tasks

reckenrode mentioned this pull request Jul 15, 2022

make-bootstrap-tools: add pbzx and tbd tools on x86_64-darwin #181550

Merged

13 tasks

ghost mentioned this pull request Jul 28, 2022

stdenv/linux/bootstrap-files: add mipsel-linux (mips32) #183487

Merged

3 tasks

ghost mentioned this pull request Aug 5, 2022

pkgs/stdenv/linux: update powerpc64le bootstrap-files #185182

Merged

4 tasks

ghost mentioned this pull request Aug 25, 2022

pkgs/stdenv/linux: bootstrap-files for third (and final) mips ABI: n32 #188334

Merged

3 tasks

tpwrules mentioned this pull request Dec 22, 2022

stdenv: aarch64-linux: gcc9 -> gcc12 #207135

Merged

13 tasks

Artturin mentioned this pull request Sep 9, 2023

bootstrapFiles should be updated periodically to avoid arcane bugs in stale binaries #253713

Closed

fgaz mentioned this pull request Jan 21, 2024

pkgs/stdenv/linux: update riscv64 bootstrap-files #282517

Merged

3 tasks

alyssais mentioned this pull request Jan 24, 2024

stdenvBootstrapTools: update aarch64 musl #283492

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

aarch64: update bootstrap-files#151399

aarch64: update bootstrap-files#151399
vcunat merged 1 commit intoNixOS:stagingfrom
dguibert:dg/new-bootstrap-tools-aarch64

dguibert commented Dec 20, 2021 •

edited

Loading

Uh oh!

vcunat commented Dec 20, 2021

Uh oh!

dguibert commented Dec 20, 2021

Uh oh!

tpwrules commented Dec 24, 2021

Uh oh!

vcunat left a comment

Uh oh!

dguibert commented Dec 24, 2021

Uh oh!

lovesegfault commented Jan 3, 2022

Uh oh!

dguibert commented Jan 4, 2022

Uh oh!

lovesegfault commented Jan 4, 2022

Uh oh!

vcunat commented Jan 4, 2022

Uh oh!

tpwrules commented Jan 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

dguibert commented Dec 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for this change

Things done

Uh oh!

vcunat commented Dec 20, 2021

Uh oh!

dguibert commented Dec 20, 2021

Uh oh!

tpwrules commented Dec 24, 2021

Uh oh!

vcunat left a comment

Choose a reason for hiding this comment

Uh oh!

dguibert commented Dec 24, 2021

Uh oh!

lovesegfault commented Jan 3, 2022

Uh oh!

dguibert commented Jan 4, 2022

Uh oh!

lovesegfault commented Jan 4, 2022

Uh oh!

vcunat commented Jan 4, 2022

Uh oh!

tpwrules commented Jan 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dguibert commented Dec 20, 2021 •

edited

Loading