nixos/qemu-vm: use persistent block device names

[nikstur]

Description of changes

This change removes the bespoke logic around identifying block devices. Instead of trying to find the right device by iterating over qemu.drives and guessing the right partition number (e.g. /dev/vda{1,2}), devices are now identified by persistent names provided by udev in /dev/disk/by-*.

Before this change, the root device was formatted on demand in the initrd. However, this makes it impossible to use filesystem identifiers to identify devices. Now, the formatting step is performed before the VM is started. Because some tests, however, rely on this behaviour, a utility function to replace this behaviour in added in /nixos/tests/common/auto-format-root-device.nix.

Devices that contain neither a partition table nor a filesystem are identified by their hardware serial number which is injecetd via QEMU (and is thus persistent and predictable). PCI paths are not a reliably way to identify devices because their availability and numbering depends on the QEMU machine type.

This change makes the module more robust against changes in QEMU and the kernel (non-persistent device naming) and by decoupling abstractions (i.e. rootDevice, bootPartition, and bootLoaderDevice) enables further improvement down the line.

For all the tests to succeed, depends on #234030.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[RaitoBezarius]

This is beautiful. :)

[nikstur]

Could this solve the issue trying to be solved in #233847 ?

[RaitoBezarius]

Could this solve the issue trying to be solved in #233847 ?

Only partially, here's the issue is related to Nix closures I believe

[roberth]

I'm not familiar with this part and I can't do a deep dive now, but the idea seems good and I don't see any problems besides breaking tests that depend on these details, which you've already identified and I think is worth breaking.

[RaitoBezarius]

Once we are done with directBoot, I can push your branch to our Hydra jobset.

[RaitoBezarius]

Pushing 532ea30 as a baseline evaluation.

[RaitoBezarius]

Hello hello, I bring the failures: https://hydra.nixos.org/eval/1796336#tabs-now-fail — :D

[nikstur]

I think I fixed all deterministic failures. @RaitoBezarius can you push another eval?

[RaitoBezarius]

Pushed ee60d30

[nikstur]

Looks like only unrelated failures are left: https://hydra.nixos.org/eval/1796429?compare=-184800&full=0

[RaitoBezarius]

nods

[RaitoBezarius]

Please write a release note for this, maybe introduce a nixpkgs internal change section in 23.11 based on what we have done for 23.05.

[nikstur]

Please write a release note for this

Done

[emilazy]

This broke darwin.builder; mkfs.ext4 cannot be run on Darwin hosts. I don't see any way to make it work with this approach off the top of my head (other than maybe making it not depend on disk images at all?).

cc @Gabriella439

[emilazy]

Ah, after reading the commit more, maybe it's as simple as turning off useDefaultFilesystems and using autoFormat? I thought the mkfs.ext4 was being added unconditionally.

[alyssais]

Why can't mkfs.ext4 run on Darwin?

[emilazy]

Hmm, I wasn't expecting e2fsprogs to be portable enough to build on Darwin but apparently it is, except that all the tests fail horribly. Not sure if it's just a matter of skipping the tests on Darwin or if something is more fundamentally broken there.

The actual observed issue is that the VM run script references a Linux mkfs.ext4 (but Darwin versions of all other tools), I guess because pkgs here is for the guest system?

[emilazy]

Unfortunately it is not that simple:

emily@yuyuko ~> /nix/store/66wby349s40sp4pyzj40s01kzrxhpc7f-e2fsprogs-1.47.0-bin/bin/mkfs.ext4 test.img
mke2fs 1.47.0 (5-Feb-2023)
dyld[13680]: missing symbol called
fish: Job 1, '/nix/store/66wby349s40sp4pyzj40…' terminated by signal SIGABRT (Abort)

I don't really have any idea where to start debugging it. Homebrew has a package that apparently works, so it should be possible to fix. I'll try to take a look tomorrow.

[nikstur]

I opened #238596 which should at least use the right e2fsprogs package from the host pkgs. Can't really help with mkfs.ext4 not working properly on darwin since I don't have anything to test.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/darwin-builder-on-an-m1-mac-problems-with-mkfs-ext4/29523/3

[alyssais]

Apparently e2fsprogs built for Darwin until the recent upgrade, so bisecting might be an option?

[emilazy]

I tracked it down to #238791 which will be fixed with the upcoming Darwin stdenv rework; sorry for not linking back.

[symphorien]

This broke nixos-rebuild build-vm: #292156

[symphorien]

My bad, I need to remove the old nixos.qcow2 image in the current directory. The new code does not handle old images.