nixos/binfmt: Add support for using statically-linked QEMU

[zhaofengli]

Motivation for this change

This PR adds support for using a statically-linked build of QEMU without any intermediate wrapper to the binfmt-misc integration in NixOS. It also enables the "F" (fix binary) flag, allowing chroot into a foreign root filesystem without first having to make the interpreter available inside the rootfs. This fixes #160300 so foreign chroots "just work."

To use the statically-linked QEMU, set boot.binfmt.preferStaticEmulators = true;.

Currently, qemu-user-static is built with musl and has a couple of hard-to-debug problems. When building coreutils for aarch64-linux on x86_64-linux with it (see #143060 for background), all tests pass except for test-free where it segfaults the tests become stuck. Other distros ship qemu-user-static with static glibc. To prevent introducing new breakages, this PR keeps the dynamically-linked QEMU as the default.

How To Test

1. Run nix-build -A nixosTests.systemd-binfmt
   • The chroot test is added which makes use of the new qemu-user-static
2. Activate a config with this PR applied
3. Confirm that nix-build --system aarch64-linux -A hello --check works
4. Activate a config with boot.binfmt.preferStaticEmulators = true;
5. Open /etc/nix/nix.conf and observe that extra-sandbox-paths is now empty
   • Yay purity!
6. Confirm that nix-build --system aarch64-linux -A hello --check still works
   • Sadly nix-build --system aarch64-linux -A coreutils --check doesn't work now
7. Try chrooting into ArchLinuxARM-aarch64-latest.tar.gz

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[eddyb]

What's the status of this?

I ran into the need for aarch64 containers (to try and reproduce some weird ar archive corruption/format mismatch), and ended up rebasing this PR (of note is guestAgentSupport that was added after this PR, and I also ended up adding a commit for numaSupport = false; - not sure why that's only a problem now?).

For the limited testing needs I have I can probably just keep rebasing if I need to update, and just disable the binfmt config once I no longer need it - but it would be great to land this functionality in nixpkgs.

EDIT: heh, didn't really need this, turns out the real issue was a much weirder "high UID/GID" thing, not specific to any architecture - but I'll keep my branch around if it helps anyone.

[SuperSandro2000]

Maybe we should combine this into a minimal option or package?

[tartavull]

This is great work, how can we contribute to finalize it?

[zhaofengli]

I guess we can first get the support for building QEMU statically as well as qemu-user-static in first. After that, we can iterate on it and finally replace the binfmt implementation to use qemu-user-static in another PR.

[YisuiDenghua]

Error occurred when I was trying to build NixOS from github:zhaofengli/nixpkgs/binfmt-qemu-static. Adding numaSupport = false to user-binfmt.nix may solve the problem.

[Detegr]

This looks great, I'd love to see this getting merged.

[zhaofengli]

Hi all, glad to see this finally getting some attention. I'm currently busy with something else at the moment but will get back to this PR in the coming days.

[Detegr]

Is there anything I can do to help getting this forward?

[ImBearChild]

This is a friendly ping. Since the allocation failure is intentional, would it be possible to merge the pull request now? I believe that this feature would be very useful for developers who work on embedded Linux products.

[Detegr]

I ran into this limitation today once again. +1 for merging this.

[muellerbernd]

I also ran into this problem. +1 for merging

[ck3d]

Could you please solve the merge conflicts. I will merge afterwards.

[TECHNOFAB11]

Ran into this the last few days and took me a while to figure out why binfmt was working on Debian but not on NixOS, +1 for merging, would allow me to run different arch images in Kubernetes

[bltavares]

I'm not familiar with the etiquette on updating the PR of others, but I did attempt to update it to fix merge conflicts and new compilation issues on #300070

I did not find yet the right git commands to keep #160802 authorship during the merge I'd like to thank @zhaofengli for the original PR and I'll investigate how to keep the authorship intact meanwhile

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/docker-ignoring-platform-when-run-in-nixos/21120/17

[SuperSandro2000]

@bltavares see https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

[zhaofengli]

Superseded by #334859 (and previously #300070)

[oxalica]

Superseded by #160802 (and previously #300070).

Do you mean #334859? Your link is a self-reference to this one.

[zhaofengli]

Do you mean #334859? Your link is a self-reference to this one.

Oops, you’re right. I fixed the link.