stdenv: bootstrap riscv64-linux (Step 1)

[oxalica]

Motivation for this change

Fix bootstrap packages and bootstrap stdenv on native riscv64-linux platform.

Related: #101651

UPDATE 2021.11.27

Rebased and fixed. Currently it doesn't contain bootstrap binary hashes. As suggested by @r-burns #115406 (comment), we should build it on Hydra for trusted binaries.

UPDATE 2021.10.23
Rebased and re-bootstrapped with QEMU VM. Thanks @zhaofengli.

Previous attempt

Files ready to be uploaded to tarballs.nixos.org: (Currently on my cachix https://riscv64.cachix.org)

• /nix/store/yffnjz4sny3ygbm59wk69yk56ja0lyk8-busybox
  Nar hash: sha256:d67bc5ef5c550bc1e2b86b27fe20b9ffadca42b9d104ae0f27957794be62ac5f

• /nix/store/gaa783s4y4w14wyc94lh5pvbpyszbvhc-bootstrap-tools.tar.xz
  File hash: sha256:541b51c0dcc4e9fb04652664ff03581c73c16e24198e4030c8ff205fc1114c3d

Steps I did

1. Cross build bootstrap tools for riscv64-linux, with build platform x86_64-linux.

nix build -f pkgs/stdenv/linux/make-bootstrap-tools-cross.nix riscv64.build -o result-rv64-bootstrap-tools-stage0

2. Get hashes to fill pkgs/stdenv/linux/bootstrap-files/riscv64.nix.
   Then add fixed-output paths into store since our bootstrap file URLs are fake.

nix hash path --type sha256 --base16 ./result-rv64-bootstrap-tools-stage0/on-server/busybox
nix hash file --type sha256 --base16 ./result-rv64-bootstrap-tools-stage0/on-server/bootstrap-tools.tar.xz
# ... hashes to fill

nix store add-path ./result-rv64-bootstrap-tools-stage0/on-server/busybox
nix store add-file ./result-rv64-bootstrap-tools-stage0/on-server/bootstrap-tools.tar.xz

3. Setup a native riscv64-linux platform.
   For me, HiFive-Unleashed is used.
   Cross build nix for riscv64-linux, copy closures to the riscv64 platform, and setup /nix and PATH correctly.
   I currently don't have any native machine, so I setup an QEMU VM for the bootstrap build.
   Note that some packages like coreutils must be built on native machine or QEMU VM, instead of binfmt (QEMU user mode), to pass all tests . See Notes 3 below.

4. Remote build native bootstrap tools on riscv64-linux with previous cross bootstrap tools.

nix build -f pkgs/stdenv/linux/make-bootstrap-tools.nix build --arg localSystem '{ system = "riscv64-linux"; }' -L --keep-going --builders 'ssh://rv64_machine_or_vm riscv64-linux - 4' -j0 -o result-rv64-bootstrap-tools-stage1

5. Update bootstrap file hashes to native ones in pkgs/stdenv/linux/bootstrap-files/riscv64.nix.

nix hash path --type sha256 --base16 ./result-rv64-bootstrap-tools-stage1/on-server/busybox
nix hash file --type sha256 --base16 ./result-rv64-bootstrap-tools-stage1/on-server/bootstrap-tools.tar.xz
# ... hashes to fill

nix store add-path ./result-rv64-bootstrap-tools-stage1/on-server/busybox
nix store add-file ./result-rv64-bootstrap-tools-stage1/on-server/bootstrap-tools.tar.xz

6. Bootstrap again to check if it works. But not update hashes this time.

nix build -f pkgs/stdenv/linux/make-bootstrap-tools.nix build --arg localSystem '{ system = "riscv64-linux"; }' -L --keep-going --builders 'ssh://rv64_machine_or_vm riscv64-linux - 4' -j0 -o result-rv64-bootstrap-tools-stage2

Notes

1. Bootstrap tools are currently using fake URLs. They should be uploaded first and updated to tarballs.nixos.org before merging this PR.

2. All intermediate and final store paths are available on https://riscv64.cachix.org.

3. Using binfmt instead of native riscv64 platform (or QEMU VM) will make few derivations like coreutils fail the checkPhase because of changed argv[0] and emulator behavior. Some tests expects a specific failure like "invalid cwd" from binary, but binfmt qemu fails before it which produces different error message.

4. gcc10 uses PGO and therefore not deterministic (gcc10: build is not deterministic #108475). So bootstrap file hashs may also be not reproducible.

5. libatomic is required for packages linking pthread or using <atomic> (like xz) during the bootstrap. But there is another way to avoid producing libatomic in bootstrap tools, but re-configure these packages to disable multi-threading before new gcc is built (like what we do with perl)
   I'm not sure which way is preferred.
   I just added libatomic to bootstrap tools.

6. bootstrap-tools-stage2 differs from bootstrap-tools-stage1. The differences are from glibc and gcc (cc1/cc1plus).
   I believe it's caused by that -frandom-seed is based on $out (stdenv: introduce -frandom-seed #102251), which is hashed from all drv inputs. Here between two bootstraps, the bootstrap tools should be functionally identical but have different hash, which changes random-seed of downstream packages.
   It should not be an issue though.

diff -r ./result-rv64-bootstrap-tools-stage{1,2}

Binary files ./result-rv64-bootstrap-tools-stage1/on-server/bootstrap-tools.tar.xz and ./result-rv64-bootstrap-tools-stage2/on-server/bootstrap-tools.tar.xz differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/bin/make and ./result-rv64-bootstrap-tools-stage2/pack/bin/make differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/crt1.o and ./result-rv64-bootstrap-tools-stage2/pack/lib/crt1.o differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/ld-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/ld-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/ld-linux-riscv64-lp64d.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/ld-linux-riscv64-lp64d.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libc-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libc-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libcrypt-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libcrypt-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libcrypt.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libcrypt.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libcrypt.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libcrypt.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libc.so.6 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libc.so.6 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libdl-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libdl-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libdl.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libdl.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libdl.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libdl.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libm-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libm-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libmemusage.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libmemusage.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libm.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libm.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libm.so.6 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libm.so.6 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnsl-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnsl-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnsl.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnsl.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_compat-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_compat-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_compat.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_compat.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_compat.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_compat.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_db-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_db-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_db.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_db.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_db.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_db.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_dns-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_dns-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_dns.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_dns.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_dns.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_dns.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_files-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_files-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_files.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_files.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_files.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_files.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_hesiod-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_hesiod-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_hesiod.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_hesiod.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_hesiod.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_hesiod.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libpthread-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libpthread-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libpthread.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libpthread.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libpthread.so.0 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libpthread.so.0 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libresolv-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libresolv-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libresolv.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libresolv.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libresolv.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libresolv.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/librt-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/librt-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/librt.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/librt.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/librt.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/librt.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libutil-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libutil-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libutil.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libutil.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libutil.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libutil.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1 and ./result-rv64-bootstrap-tools-stage2/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1plus and ./result-rv64-bootstrap-tools-stage2/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1plus differ

[oxalica]

cc @r-burns

[zhaofengli]

Any progress on this? Would be great if a trusted member could produce an official bootstrap tools tarball on https://tarballs.nixos.org

[oxalica]

@Ericson2314
What do you think about issue 2 about libatomic from gcc-lib? Currently I simply copy it into bootstrap tools. We can also not touching bootstrap tools, but disable multi-threading pkgs like xz before the new gcc is built (in bootstrap-stage4).

[Ericson2314]

@oxalica I don't mind adding to the the bootstrap tools. Long term, I am hoping the bootstrap tools can just be the tools, statically linked, and we bootstrap all the run-time libraries up from libgcc/compiler-rt. But that is a big refactor, and will start with the better-behaved LLVM stuff first, so I don't care if we add more one libatomic.a` in the meantime.

[oxalica]

Rebased and rebuilt the seed.

[zhaofengli]

Gentle poke. Now that a basic NixOS system is running on riscv64, I'm really looking forward to seeing this PR merged in the near future. This script starts a simple qemu-system-riscv64 VM and should help with natively building the bootstrap tools for maintainers without access to RISC-V hardware.

[zhaofengli]

Hmm, I have a problem. The GCC in stdenv doesn't seem to be able to find spec files:

$ echo "int main() {}" > test.c
$ /nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/gcc -fopenmp test.c
gcc: fatal error: cannot read spec file ‘libgomp.spec’: No such file or directory
compilation terminated.

libgomp.spec is indeed in the lib of the wrapped compiler, and I can link libgomp directly myself:

$ ls /nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib
gcc        libasan_preinit.o  libcc1.a   libgomp.spec       libssp.a            libstdc++.a    libsupc++.a
libasan.a  libatomic.a        libgomp.a  libsanitizer.spec  libssp_nonshared.a  libstdc++fs.a  libubsan.a
$ cat /nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/libgomp.spec
# This spec file is read by gcc when linking.  It is used to specify the
# standard libraries we need in order to link with libgomp.
*link_gomp: -lgomp %{static: -ldl -lpthread }
$ /nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/gcc -lgomp test.c
$

The same commands work fine on an x86-64 host.

[zhaofengli]

Still kind of confused though there's a solution. Here is the list of paths that gcc on x86-64 searched:

/nix/store/54klr10i53jdfgn7322mzgza6wsai0q8-gcc-10.3.0-lib/lib/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/54klr10i53jdfgn7322mzgza6wsai0q8-gcc-10.3.0-lib/lib/libgomp.spec
/nix/store/sbbifs2ykc05inws26203h0xwcadnf0l-glibc-2.32-46/lib/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/sbbifs2ykc05inws26203h0xwcadnf0l-glibc-2.32-46/lib/libgomp.spec
/nix/store/35pnk5kwi26m3ph2bc7dxwjnavpzl8cn-gcc-wrapper-10.3.0/bin/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/35pnk5kwi26m3ph2bc7dxwjnavpzl8cn-gcc-wrapper-10.3.0/bin/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../../x86_64-unknown-linux-gnu/lib/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../../x86_64-unknown-linux-gnu/lib/../lib64/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../../lib64/libgomp.spec
  -> hurray, we symlink $out/lib64 to $out/lib - success

The same applies on aarch64. However, on riscv64 it tries the following before giving up:

/nix/store/zw9qnha64sygasja7am4fhmlx5jx4yhb-gcc-10.3.0-lib/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/zw9qnha64sygasja7am4fhmlx5jx4yhb-gcc-10.3.0-lib/lib/libgomp.spec
/nix/store/m1jbr8vqnpgnpjfdzx13da897pd3aa6f-glibc-2.32-46/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/m1jbr8vqnpgnpjfdzx13da897pd3aa6f-glibc-2.32-46/lib/libgomp.spec
/nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/libgomp.spec
/nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/gcc/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/gcc/riscv64-unknown-linux-gnu/10.3.0/../../../../riscv64-unknown-linux-gnu/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/gcc/riscv64-unknown-linux-gnu/10.3.0/../../../../riscv64-unknown-linux-gnu/lib/libgomp.spec
/lib64/lp64d/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/lib64/lp64d/libgomp.spec
/usr/lib64/lp64d/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/usr/lib64/lp64d/libgomp.spec
/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/lib/libgomp.spec
/usr/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/usr/lib/libgomp.spec

At first glance we may want to link $out/[triplet]/lib to $out/lib as well, but I feel uneasy adopting a solution without knowing why. gcc/config/riscv/linux.h has these lines that may explain the lib64/lp64d paths, but not the others. This seems to be handled in set_up_specs which follows the startfile prefixes, but I'm pretty unfamiliar with the code.

[zhaofengli]

I've tested a naive fix which does make -fopenmp work in the final native toolchain. Would like to hear whether there is a more proper fix to the issue.

cc @Ericson2314 @oxalica

[oxalica]

Rebased and re-bootstrapped with QEMU VM now. PR description is also updated.

I hope bootstrap tools can be uploaded and this PR can be merged soon to make fundamental stdenv available. It makes everything, including fixing other packages, way much easier for native platform.

PS: I'm not sure who is the right person to ask for uploading bootstrap tools. Please cc them if anyone knows.

[r-burns]

I left a comment in #141448 (comment) about this, basically I think non-reproducible bootstrap tools might be a no-go, and we're best off trying to generate bootstrap tools on hydra so there's no leap of faith for the tarball uploader.

This could mean either:

• Build exotic platforms' native bootstrap tools on hydra using an emulator
• Fix or work around Cross bootstrap tools have broken stackprotector support #113977 so we can use the existing cross-compiled bootstrap tools

[oxalica]

@r-burns

I think non-reproducible bootstrap tools might be a no-go

I don't state that bootstrap tools are non-reproducible. They should, since the gcc issue is already resolved. I give the steps and final hash mainly for others to reproduce the same result.

Build exotic platforms' native bootstrap tools on hydra using an emulator

I assume you mean binfmt + qemu user-mode. I tried it and it works for all packages except some coreutils tests. If possible, we could disable these tests, especially, the tests running programs in non-existing CWD and expecting a specific error message, and then everything would work. I'm not sure this is a good way since we need to a workaround for an emulator rather than a platform, which is quite strange. But yeah there are already tons of workarounds of coreutils tests.

Fix or work around "Cross bootstrap tools have broken stackprotector support #113977" so we can use the existing cross-compiled bootstrap tools

Do you mean to just use cross-compiled bootstrap tools without another bootstrapping? This means I just need to revert the last commit here. It absolutely works. I doesn't run into the issue you mentioned, since I use the cross-compiled bootstrap tools (stage0) to get native bootstrap tools (stage1). And stage1 is still able to build stage2 bootstrap tools. Thus I built gcc four times in the chain without any issue.

If we need hydra to perform the build, what should I do here?

[Ericson2314]

Should be "shared library loader" like the others.

[r-burns]

I don't state that bootstrap tools are non-reproducible. They should, since the gcc issue is already resolved. I give the steps and final hash mainly for others to reproduce the same result.

Right I misspoke, I just meant it might be easier to get someone to mirror a hydra-built trusted-by-default artifact than something that is trusted-if-verified.

This means I just need to revert the last commit here. It absolutely works. I doesn't run into the issue you mentioned, since I use the cross-compiled bootstrap tools (stage0) to get native bootstrap tools (stage1).

Oh, that's great! I would just say to revert the last commit then.

If we need hydra to perform the build, what should I do here?

I don't think we need to do anything else then, we just need to get someone with infra permissions to mirror the artifact from nixpkgs:cross-trunk:bootstrapTools.riscv64.build to tarballs.nixos.org

[oxalica]

I'm wait for the coreutils 9.0 in staging-next #141684. Hope we can have it for linux.
I'll try to patch coreutils and test bootstrapping from cross compiled seed with binfmt after that.

[SuperSandro2000]

Suggested change

	extraNativeBuildInputs =
	lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;
	extraNativeBuildInputs = lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;

[zhaofengli]

#143060 enables argv[0] preservation in binfmt and fixes all the argv[0]-related test failures in coreutils, however the full test suite still doesn't pass (see linked logs).

[oxalica]

#143060 enables argv[0] preservation in binfmt and fixes all the argv[0]-related test failures in coreutils, however the full test suite still doesn't pass (see linked logs).

Almost forget it. I always have my patch #118926 applied. Besides the argv[0] issue, some other failures are:

1. rm/rm-readdir-fail.sh, misc/csplit-io-err.sh
LD_PRELOAD is set. libc tries to preload the riscv64 .so when executing the intermediate shell script and QEMU, resulting two lines of ERROR: ld.so: object '...' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored..
2. misc/date.pl, misc/sort-debug-warn.sh
LC_ALL is set. It cause the intermediate shell script to raise
   warning: setlocale: LC_ALL: cannot change locale.
3. ls/removed-directory.sh
   It tries to execute a binary in non-existing CWD, causing the intermediate shell script to raise
   shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory.

(1), (2) and (3) are all related to the intermediate shell script. We can fix them by,

1. Use a static compiled C binary to forward arguments, replacing the shell script. So environment variables and CWD doesn't affect us.
2. Unset LD_* and pass -E LD_*=value to set environments only for target ELF, not for QEMU itself.

@zhaofengli Are you interested in implementing this? I think we could enhance #143060 to fix all of these.

[zhaofengli]

@oxalica I didn't look deeper into the remaining failures - Thanks for doing that! I think for 1, one way is to build the wrappers alongside qemu, packaged as a separate output (e.g., pkgs.qemu.binfmtShims). I'll look more and give it a try tomorrow.

[vcunat]

I'm wait for the coreutils 9.0 in staging-next #141684. Hope we can have it for linux.

It is in master now, though there were complications because of a ZFS bug.

[oxalica]

Rebased and removed commits containing binary hashes.

@r-burns
I'd request a Hydra job to build cross compiled bootstrap tools on this PR, basically,
nix build -f pkgs/stdenv/linux/make-bootstrap-tools-cross.nix riscv64.build
And upload two result binaries as other bootstrap tools did.

[vcunat]

So we simply merge this and it will appear there, right? https://hydra.nixos.org/job/nixpkgs/cross-trunk/bootstrapTools.riscv64.bootstrapTools

(And continue in followup PRs.)

[vcunat]

I expect the builds to finish within a few hours: https://hydra.nixos.org/eval/1726433?filter=riscv64

[oxalica]

It's done building and I can match result hashes with my own build.

But how can we reference these tools by tarball.nixos.org like others?

nixpkgs/pkgs/stdenv/linux/bootstrap-files/aarch64.nix

Line 3 in 33da2dc

url = "http://tarballs.nixos.org/stdenv-linux/aarch64/bb3ef8a95c9659596b8a34d27881cd30ffea2f9f/busybox";

Do we need someone manually upload build results of busybox and bootstrap-tools.tar.xz?

[vcunat]

For the commit in URL it's customary to use the one that generated the bootstrap, i.e. 9bd3cf0063b80428bd85a286205adab4b6ffcbd6 here. The rest is probably obvious?

I'd suggest preparing the PR and test that the bootstrap works locally (if you prefetch the files, nix won't attempt download). Then ping someone who can upload it manually; I'd suggest @grahamc. After upload, perhaps re-test and then merge.

[oxalica]

I've opened the following PR #147942.

I'd suggest preparing the PR and test that the bootstrap works locally (if you prefetch the files, nix won't attempt download).

I had tested building stdenv with my previous built bootstrap files. And Hydra reproduced my bootstrap files. So I could simply change URLs of FODs, which doesn't change any downstream derivations.