nixos/iso-image: Support systemd initrd by ElvishJerricco · Pull Request #291750 · NixOS/nixpkgs

ElvishJerricco · 2024-02-27T06:47:55Z

Description of changes

This modifies the ISO configuration to be able to use systemd initrd. It should still work without systemd initrd as well.

Something to note is the removal of the awkward use of the root= kernel param. In scripted stage 1, the value of this parameter ends up being symlinked to /dev/root, and previously the ISO used this to mount /iso. For one thing, this is an incorrect usage of root=. But, I do not know how this will affect other tools flashing the image to a bootable drive. I know we only really support directly writing the image to disk, but I don't want to degrade the experience from people's historical expectations. Also, the findiso= mechanism will not be supported with systemd initrd. This is a niche feature to begin with, and we intend to stop using grub for the ISO in the near future anyway.

Things done

Add a 👍 reaction to pull requests you find important.

K900

Diff LGTM, will test on weird setups later

ElvishJerricco · 2024-02-27T06:54:40Z

Oh, forgot to mention the weird thing with findiso=. Added that to the todo list

K900 · 2024-02-27T08:09:23Z

Things I found so far:

can't find rootfs with ventoy
when failing to find rootfs, can't enter recovery shell because root account is locked

ElvishJerricco · 2024-02-27T08:17:07Z

@K900 Fixed the recovery shell thing; though not in a way compliant with the boot.shell_on_fail thing, which is why that's in the todo list

misuzu · 2024-02-27T20:52:03Z

Related: #217173

nikstur · 2024-03-10T23:24:47Z

I think the overlay thing doesn't create the same depends values. Though, see #233707 or #273642

Afaik the depends are just a no-op right now even in the scripted initrd.

ElvishJerricco · 2024-03-16T19:43:36Z

@nikstur depends currently exclusively works for file systems mounted by scripted stage 1. File systems are toposorted by path and depends and then traversed by stage-1-init.sh in that order. But depends currently does nothing for systemd stage 1 or for stage 2, which is what the linked prs are about.

RaitoBezarius · 2024-03-18T20:56:45Z

@K900 Fixed the recovery shell thing; though not in a way compliant with the boot.shell_on_fail thing, which is why that's in the todo list

What does that mean? From my understanding, the boot.shell_on_fail does not exist under systemd initrd, it's just systemd emergency access and that requires manual upgrades, I'd say?

ElvishJerricco · 2024-03-19T02:47:42Z

@RaitoBezarius well that's what I mean. Ideally, systemd stage 1 would still support (most of) the same cmdline parameters (see: #170125). The boot.debug1 one for instance equates to (sort of) rd.systemd.unit=emergency.target ((sort of)), so it would be nice if we could somehow make that an alias. But we don't really have a way to say "if stage 1 fails, drop straight into a shell", which is the currently expected behavior of the ISO.

nikstur · 2024-03-28T09:20:13Z

we don't really have a way to say "if stage 1 fails, drop straight into a shell", which is the currently expected behavior of the ISO.

IMO systemd's emergency shell fulfills this use case adequately, even if it's not the same exact behaviour as before.

ElvishJerricco · 2024-03-28T10:03:45Z

@nikstur Yea, I've come to largely the same conclusion that we don't need one to one kernel param compatibility. I'm also adding systemd's debug shell service in a separate PR (#299717), so that provides even more functionality similar to what we have in scripted stage 1.

We'll just have to document the differences in the cmdline options.

nixos-discourse · 2025-03-17T03:47:11Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/custom-nixos-installer-plug-install-play-how-to-achieve-this/61710/13

ElvishJerricco · 2025-08-01T04:58:22Z

With #429845 done, I think we're ready to get this one done now. We plan to switch systemd initrd on by default in the near future, so this PR simply prepares for that by adding support for systemd initrd, rather than switching the ISO to using it directly.

Also, I've decided not to try and support the findiso= feature, as we're going to lose that feature when we switch off grub anyway.

flokli · 2025-08-01T08:18:28Z

can't find rootfs with ventoy

@K900 Is this still broken? Still being able to load a NixOS ISO to a ventoy stick seems important, as that's how NixOS frequently gets installed.

ElvishJerricco · 2025-08-01T14:03:50Z

@flokli I've researched that pretty extensively and concluded that that was already a problem for the ISO without systemd initrd, it was just less likely. The "hook" it uses just searches the initrd for a directory it thinks it should inject a udev rule file into, but whether or not it guesses the right one correctly basically depends on the order of the hash-addressed paths in the store, and the larger number of store paths in systemd initrd just makes it more likely for that to fail.

The only way to fix this is to adopt ventoy's custom protocol that lets the ISO implement ventoy support itself instead of relying on this broken "hook". I have a branch where I've implemented this but it requires a bunch of new code and I'm not happy about that

EDIT: To be clear, since researching this, I have encountered countless users who have the same problem with ventoy not working for this reason with the current ISO builds.

emilazy · 2025-08-01T14:36:15Z

Yeah I don’t think we should block on something that’s already broken and inherently fragile. We don’t recommend Ventoy anywhere and one of the first Google results for “Ventoy NixOS” is a Reddit thread full of people talking about how Ventoy doesn’t work with NixOS ISOs.

emilazy

Code LGTM; haven’t tested this myself but I expect you have done more thorough testing than I would. (This PR doesn’t flip the default, anyway.)

I personally don’t think it makes sense to support both systemd stage 1 and scripted stage 1 in something we ship as a self‐contained appliance, but since we’re adding support before changing the default that’s somewhat unavoidable for now anyway.

emilazy · 2025-08-01T14:39:10Z

nixos/modules/installer/cd-dvd/iso-image.nix

+      # Most of util-linux is not included by default.
+      initrdBin = [ config.boot.initrd.systemd.package.util-linux ];
+      services.copytoram = {
+        description = "Copy ISO contents to RAM";
+        requiredBy = [ "initrd.target" ];
+        before = [
+          "${utils.escapeSystemdPath "/sysroot/nix/.ro-store"}.mount"
+          "initrd-switch-root.target"
+        ];
+        unitConfig = {
+          RequiresMountsFor = "/sysroot/iso";
+          ConditionKernelCommandLine = "copytoram";
+        };
+        serviceConfig = {
+          Type = "oneshot";
+          RemainAfterExit = true;
+        };
+        path = [
+          pkgs.coreutils
+          config.boot.initrd.systemd.package.util-linux
+        ];
+        script = ''
+          device=$(findmnt -n -o SOURCE --target /sysroot/iso)
+          fsSize=$(blockdev --getsize64 "$device" || stat -Lc '%s' "$device")
+          mkdir -p /tmp-iso
+          mount --bind --make-private /sysroot/iso /tmp-iso
+          umount /sysroot/iso
+          mount -t tmpfs -o size="$fsSize" tmpfs /sysroot/iso
+          cp -r /tmp-iso/* /sysroot/iso/
+          umount /tmp-iso
+          rm -r /tmp-iso
+        '';
+      };
+    };


(This is kind of gross, but I realize that doing anything about it is out of scope for this PR.)

bjornfor · 2025-08-07T14:33:14Z

EDIT: To be clear, since researching this, I have encountered countless users who have the same problem with ventoy not working for this reason with the current ISO builds.

When did NixOS ISOs stop working with Ventoy? I tested NixOS 25.05 in Ventoy this week, and that worked. I will be sad if it stops working.

ElvishJerricco · 2025-08-08T21:15:30Z

When did NixOS ISOs stop working with Ventoy? I tested NixOS 25.05 in Ventoy this week, and that worked. I will be sad if it stops working.

@bjornfor They never worked reliably with Ventoy. The hook it uses to inject its software in our initrd often fails, seemingly depending on the specific build of the ISO you use rather than any runtime details. IIRC it basically ends up putting its udev rules (which call its small OS's-worth of binaries to set things up in a cursed way) into the wrong directory, probably because the hashes in nix store path names effectively randomizes their order on every build.

systemd stage 1 just makes this problem more likely to occur, probably because we don't do the same extraUtils thing that scripted stage 1 did, so we end up with a lot more top level paths in the initrd's nix store.

I can fix this by using Ventoy's custom boot protocol. Tbh it's a relatively sane protocol, except that by default it wants us to scan /dev/mem on legacy BIOS platforms because Windows 7 ISOs break when it passes info in as an ACPI table; so that part's pretty cursed. Overall though it's much more sane than the hook it uses for ISOs not employing the protocol. But AFAICT no other distro implements this protocol in their ISOs, and it's a lot of added code to nixpkgs, so I'm not exactly eager to actually add it.

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Feb 27, 2024

ElvishJerricco requested review from a team, K900 and RaitoBezarius February 27, 2024 06:48

K900 approved these changes Feb 27, 2024

View reviewed changes

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Feb 27, 2024

ElvishJerricco force-pushed the systemd-initrd-iso branch from 9510f63 to 8690124 Compare February 27, 2024 08:17

ElvishJerricco force-pushed the systemd-initrd-iso branch from 8690124 to c4d9e6a Compare March 2, 2024 08:02

ElvishJerricco mentioned this pull request Mar 3, 2024

[Tracking Issue] Sunset scripted initrd and scripted networking #287308

Open

12 tasks

ElvishJerricco mentioned this pull request Apr 15, 2024

Support all cmdline options in systemd stage 1 #170125

Open

15 tasks

ElvishJerricco mentioned this pull request May 5, 2024

GDM + GNOME crashes after 20 seconds with autologin, systemd initrd, and plymouth enabled. #309190

Closed

ElvishJerricco mentioned this pull request Aug 2, 2024

iso image with systemd stage1 fails to boot due to nonexistent /dev/root #327187

Closed

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 27, 2024

nvmd mentioned this pull request Oct 1, 2024

resumeDevice for swap on btrfs nix-community/disko#651

Open

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 2, 2025

arianvp added this to Replace scripted networking with networkd Mar 3, 2025

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 17, 2025

ElvishJerricco force-pushed the systemd-initrd-iso branch from c4d9e6a to c551ce5 Compare August 1, 2025 04:28

ElvishJerricco changed the title ~~nixos/iso-image: Use systemd initrd~~ nixos/iso-image: Support systemd initrd Aug 1, 2025

nixpkgs-ci bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 1, 2025

nixos/iso-image: Support systemd initrd

db42eeb

ElvishJerricco force-pushed the systemd-initrd-iso branch from c551ce5 to db42eeb Compare August 1, 2025 04:36

ElvishJerricco marked this pull request as ready for review August 1, 2025 04:54

emilazy approved these changes Aug 1, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Aug 1, 2025

nikstur approved these changes Aug 7, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 3+ This PR was reviewed and approved by three or more persons. and removed 12.approvals: 2 This PR was reviewed and approved by two persons. labels Aug 7, 2025

emilazy merged commit 46aad76 into NixOS:master Aug 10, 2025
31 of 33 checks passed

github-project-automation bot moved this to Done in Replace scripted networking with networkd Aug 10, 2025

Enzime mentioned this pull request Aug 13, 2025

Question: Why do these images work on Ventoy nix-community/nixos-images#372

Closed

ElvishJerricco mentioned this pull request Aug 16, 2025

Systemd Stage 1 ISO Boot #217173

Closed

BrainWart mentioned this pull request Oct 11, 2025

Install-iso: waiting for /dev/root on boot then fails nix-community/nixos-generators#432

Open

Uh oh!

Conversation

ElvishJerricco commented Feb 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

K900 left a comment

Choose a reason for hiding this comment

Uh oh!

ElvishJerricco commented Feb 27, 2024

Uh oh!

K900 commented Feb 27, 2024

Uh oh!

ElvishJerricco commented Feb 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

misuzu commented Feb 27, 2024

Uh oh!

nikstur commented Mar 10, 2024

Uh oh!

ElvishJerricco commented Mar 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RaitoBezarius commented Mar 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ElvishJerricco commented Mar 19, 2024

Uh oh!

nikstur commented Mar 28, 2024

Uh oh!

ElvishJerricco commented Mar 28, 2024

Uh oh!

nixos-discourse commented Mar 17, 2025

Uh oh!

ElvishJerricco commented Aug 1, 2025

Uh oh!

flokli commented Aug 1, 2025

Uh oh!

ElvishJerricco commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilazy commented Aug 1, 2025

Uh oh!

emilazy left a comment

Choose a reason for hiding this comment

Uh oh!

emilazy Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

bjornfor commented Aug 7, 2025

Uh oh!

ElvishJerricco commented Aug 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

ElvishJerricco commented Feb 27, 2024 •

edited

Loading

ElvishJerricco commented Feb 27, 2024 •

edited

Loading

ElvishJerricco commented Mar 16, 2024 •

edited

Loading

RaitoBezarius commented Mar 18, 2024 •

edited

Loading

ElvishJerricco commented Aug 1, 2025 •

edited

Loading