pkgs/stdenv/linux: add powerpc64le bootstrap-files

[ghost]

Description of changes

This PR adds Hydra-generated bootstrap tarballs for powerpc64le-linux. I'll be following the script established in #151399.

Files came from this Hydra build, which used nixpkgs revision 49a8344 to instantiate:

/nix/store/gj272sd56gsj6qpyzh4njpfzwdhviliz-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv

and then built:

/nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu

I downloaded these files from Hydra with the following commands:

STOREPATH=n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org"
nix store add-file \
  --name bootstrap-tools.tar.xz  \
  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
  --name busybox \
  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox

As @lovesegfault requested in #151399, here are the the sha256sums of all the on-server components for extra verification:

$ sha256sum /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
036d062869f7accf0ad89714d12029469dfe6af504f9b226d61eb7d808ad4735  /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
6bc05832180f5075f4458c32eb0a5e2b673f605884dce01822be474f4e0a63ee  /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox

Things done

• Built on platform(s)
  • powerpc64le-linux (in progress)
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

[ghost]

$ uname -m
ppc64le
$ NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix-shell -I nixpkgs=. -p hello --command hello
Hello, world!

Note that to use the bootstrap you also need this commit, which I have not yet submitted because I'm not totally clear on why it is necessary.

[ghost]

Squashed. Holy cow, @SuperSandro2000 is extra-super today!

[SuperSandro2000]

@lovesegfault can you upload the tarballs?

[sternenseemann]

NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1

I wonder what causes the eval failure without this? Extending the package-in-question's meta.platforms would probably be prudent.

[ghost]

NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1

I wonder what causes the eval failure without this? Extending the package-in-question's meta.platforms would probably be prudent.

Apparently setting this wasn't necessary. I just tried the same command again without that environment variable and it evaluated correctly (I didn't wait for the build to finish though).

Edit: ah, I needed it for my ghc PR, and just got in the habit of adding it to invocations of nix on my powerpc64le machine. It looks like the ghc package is the only one that needs it.

In general I in no particular hurry to remove the need for things like NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1. Getting back to our earlier discussion about the notion of "support" (and my complaint that English uses the same word for both "code that handles this case" and "the burden of helping users"), I agree that this kind of "support" should be changed with great caution and careful consideration, because removing a flag like this does advertise to users that "you should expect this to work", which can reflect badly on nixpkgs if it doesn't.

On the other hand, the kind of "support" that involves merging one-liners to add bootstrap hashes and or an extra line to a Rosetta Stone somewhere -- I think that adding that kind of "support" shouldn't be held to any higher standard than any other PR submitted to nixpkgs.

[ghost]

Ping @lovesegfault

Are there other people with the ability to upload to tarballs.nixos.org? We'll be repeating this process for mips64el as soon as staging reopens and #168413 merges (it fixes the problem that is preventing Hydra from building a bootstrap tarball for mips64el).

[lovesegfault]

Hi, I'm the person responsible for that right now. It's a busy time in my life, I'll be doing this right now.

[lovesegfault]

@a-m-joseph Can you hop on Matrix and DM me?

I need a bit of help to understand where these should go, since we already have some powerpc bootstrapping files.

[ghost]

@a-m-joseph Can you hop on Matrix and DM me?

Believe it or not I've never used Matrix; I've been kinda procrastinating waiting for the IRC bridge to be enabled. I guess now would be a good time for me to learn Matrix, but that's probably not likely to happen in the next ten minutes...

I need a bit of help to understand where these should go, since we already have some powerpc bootstrapping files.

There are both 32-bit and 64-bit variations of PowerPC. In addition, the chip can operate in either big-endian or little-endian mode. So we have:

powerpc
powerpcle
powerpc64
powerpc64le

Some people use Debian's naming scheme, where the powerpc has been abbreviated to ppc. This should be avoided, unless tarballs.nixos.org is already using that naming scheme, in which case I guess we could stick with it.

In any event, this PR contains the URL where it expects to find the tarball; is there a reason why you can't make the tarball appear at that specific URL?

Thanks,

[lovesegfault]

@a-m-joseph Thanks for clarifying!

The files are there:

$ aws s3 ls s3://nixpkgs-tarballs/stdenv-linux/powerpc64le/49a83445c28c4ffb8a1a90a1f68e6150ea48893b/
2022-04-16 14:17:30   22964768 bootstrap-tools.tar.xz
2022-04-16 14:17:30     299176 busybox

I've verified that the hashes match the ones you've given me. You should be all set.

Sorry for the wait :)

[lovesegfault]

In any event, this PR contains the URL where it expects to find the tarball; is there a reason why you can't make the tarball appear at that specific URL?

Just for future reference, I have the power to add bootstrap tarballs, but I do not have the power to remove, move, or modify them once they are there.

This is why I quadruple-check everything before uploading, since if I make a mistake, only two other people can undo it, and they are quite busy so that can take weeks to happen.

IOW: I do not upload a bootstrap tarball unless I understand exactly:

1. How it was generated
2. What is in it
3. Where it's going
4. What it's for

[ghost]

Just for future reference, I have the power to add bootstrap tarballs, but I do not have the power to remove, move, or modify them once they are there.

Yeah, for a while I wondered why a manual process was needed in order to get stuff on tarballs.nixos.org, then I realized that once things are put there, they pretty much need to stay there forever to avoid breaking things.

Thank you so much for your help here, and for your careful guardianship of this duty!

I'm verifying that everything works right now. Edit: verified, see next comment.

[ghost]

Alright, using a fresh install of Debian bullseye on a powerpc64le Talos Raptor II, with no /nix/store:

# adduser nix
# mkdir /nix
# chown nix:nix /nix
### install the bazillion .deb's needed to build nix-2.3.10
# su nix
nix@talos:~$ mkdir prefix
nix@talos:~$ git clone -b 2.3.10 https://github.com/NixOS/nix
nix@talos:~$ cd nix
nix@talos:~/nix$ ./bootstrap.sh && ./configure --prefix ~/prefix --disable-doc-gen && make -j32 && make install
### clone nixpkgs into /nix/nixpkgs
nix@talos:/nix/nixpkgs$ git fetch https://github.com/NixOS/nixpkgs/ pull/168199/head
From https://github.com/NixOS/nixpkgs
 * branch                    refs/pull/168199/head -> FETCH_HEAD
nix@talos:/nix/nixpkgs$ git rev-parse FETCH_HEAD
9d46a1025a36e2e301a9782a2ff6ba080c0bd7fe
nix@talos:/nix/nixpkgs$ git checkout -b pr168199 FETCH_HEAD
Switched to a new branch 'pr168199'
nix@talos:/nix/nixpkgs$ git fetch https://github.com/NixOS/nixpkgs/ pull/168983/head
From https://github.com/NixOS/nixpkgs
 * branch                    refs/pull/168983/head -> FETCH_HEAD
nix@talos:/nix/nixpkgs$ git rev-parse FETCH_HEAD
f5e5e78a3afd00b6c11190ad26f9f9bf170c982c
nix@talos:/nix/nixpkgs$ git cherry-pick FETCH_HEAD
[pr168199 c06478d720d] stdenv/adapters.nix: disable stackprotector hardening on 64-bit powerpc during static-only stages of stdenv bootstrap
 Author: Adam Joseph <adam@westernsemico.com>
 Date: Sat Apr 16 16:32:08 2022 -0700
 1 file changed, 1 insertion(+)
nix@talos:/nix/nixpkgs$ export PATH=/home/nix/prefix/bin:$PATH
nix@talos:/nix/nixpkgs$ export NIX_LIBEXEC=/home/nix/prefix/libexec
nix@talos:/nix/nixpkgs$ nix build --debug --option substituters "" --option substitute false --max-jobs 8 -L -f . hello |& tee ~/log.txt
### ...
downloading 'http://tarballs.nixos.org/stdenv-linux/powerpc64le/49a83445c28c4ffb8a1a90a1f68e6150ea48893b/busybox'...
starting download of http://tarballs.nixos.org/stdenv-linux/powerpc64le/49a83445c28c4ffb8a1a90a1f68e6150ea48893b/busybox
downloading 'http://tarballs.nixos.org/stdenv-linux/powerpc64le/49a83445c28c4ffb8a1a90a1f68e6150ea48893b/bootstrap-tools.tar.xz'...
starting download of http://tarballs.nixos.org/stdenv-linux/powerpc64le/49a83445c28c4ffb8a1a90a1f68e6150ea48893b/bootstrap-tools.tar.xz
finished download of 'http://tarballs.nixos.org/stdenv-linux/powerpc64le/49a83445c28c4ffb8a1a90a1f68e6150ea48893b/busybox'; curl status = 0, HTTP status = 200, body = 299176 bytes
### ...
### crunch crunch crunch
hello> checking for references to /tmp/nix-build-hello-2.12.drv-0/ in /nix/store/z90c016l9116sbq5sx9yi37vs8107za8-hello-2.12...
nix@talos:/nix/nixpkgs$ nix-shell --max-jobs 8 -I nixpkgs=. -p hello --command hello
Hello, world!

Note that #168983 is needed to build stdenv. That PR is still marked draft because I don't fully understand why it is necessary. I suspect that the unhardened binaries are not part of the final stdenv, but I have not verified this.

@SuperSandro2000 @sternenseemann this PR is ready to merge.

I'm working on pkgsCross.powernv.nixStatic next, to eliminate the "bazillion" step above.