Move in-tree kernel modules to separate derivation output by jmbaur · Pull Request #423933 · NixOS/nixpkgs

jmbaur · 2025-07-10T01:31:19Z

The kernel image takes up a significant amount of space in the derivation where kernel modules are installed (even more so on aarch64 where we install the uncompressed Image file). Since the kernel image is not functionally needed in this derivation (by the time kernel modules would be of any use, we've already booted into our kernel), we can save some space by splitting out the modules into their own output. This change doesn't really do a lot for pure NixOS closures today, since the kernel image is still in the closure of config.system.build.toplevel, however the linux kernel build recipe in nixpkgs is super useful for more than just NixOS use-cases anyways. Plus, it's not out of the realm of possibility for NixOS's toplevel to remove the reference to the kernel image, which would be especially helpful in saving space on image-based systems where the ESP and/or boot partition is built and populated ahead of time and not part of the switch-to-configuration/bootloader-installation step.

Things done

Add a 👍 reaction to pull requests you find important.

K900 · 2025-07-10T05:21:41Z

CC @emilazy

jmbaur · 2025-07-10T05:25:11Z

I've tested this on a minimal kernel configuration, but this does not yet work with the full-blown NixOS kernel configs yet, still working through this issue:

make[2]: /nix/store/00zrahbb32nzawrmv9sjxn36h7qk9vrs-bash-5.2p37/bin/bash: Argument list too long
make[2]: *** [../scripts/Makefile.modfinal:31: arch/x86/events/amd/power.mod.o] Error 127
make[1]: *** [/build/linux-6.12.36/Makefile:1878: modules] Error 2

emilazy · 2025-07-10T11:42:48Z

This seems like a sensible enough direction. Even better would be if we could if we could have one output per module, or even one derivation per module. But that might not be easy to achieve.

I plan to split the outputs of the kernel images as part of my ongoing refactoring, but that shouldn’t conflict with this (semantically – diff‐wise, it probably will).

jmbaur · 2025-07-10T17:04:21Z

I plan to split the outputs of the kernel images as part of my ongoing refactoring.

That would be great! Would be amazing to build as many kernel image types for a given architecture as possible (like the ideas in #324419), to allow for the user to pick the one they want later without having to incur a rebuild.

alyssais

Seems sensible enough to me but haven't tested. I guess there's not anything we can really do to make it backwards compatible without defeating the purpose by adding a reference…

emilazy · 2025-07-10T18:44:58Z

Yeah I think it makes sense to batch this with my other target change since that will be (deliberately) breaking.

That would be great! Would be amazing to build as many kernel image types for a given architecture as possible (like the ideas in #324419), to allow for the user to pick the one they want later without having to incur a rebuild.

Right, yeah, that’s the idea, especially as it means we can more safely prefer compressed UEFI stub images on AArch64. Though uImage I am dropping entirely.

jmbaur · 2025-07-10T21:10:04Z

I've tested this on a minimal kernel configuration, but this does not yet work with the full-blown NixOS kernel configs yet, still working through this issue:
make[2]: /nix/store/00zrahbb32nzawrmv9sjxn36h7qk9vrs-bash-5.2p37/bin/bash: Argument list too long
make[2]: *** [../scripts/Makefile.modfinal:31: arch/x86/events/amd/power.mod.o] Error 127
make[1]: *** [/build/linux-6.12.36/Makefile:1878: modules] Error 2

So it appears somewhere in the kernel build system, the $modules environment variable is read, which was causing this issue. I've changed the output name to mod, which works, but definitely open to other ideas as well.

jmbaur · 2025-07-10T22:06:36Z

@ofborg test systemd-initrd-modprobe

jmbaur · 2025-07-12T06:46:51Z

This is ready. I've tested locally with the systemd-initrd-modprobe test, though ofborg seems to be chugging on it.

emilazy · 2025-07-12T10:13:17Z

We can avoid contorting the output name by just unsetting modules and using ${outputs[modules]}, I believe. Might require __structuredAttrs but would be as good a time as any to migrate to that if we're not already on it anyway.

jmbaur · 2025-07-12T21:38:14Z

We can avoid contorting the output name by just unsetting modules and using ${outputs[modules]}, I believe. Might require __structuredAttrs but would be as good a time as any to migrate to that if we're not already on it anyway.

Should be good now!

pkgs/os-specific/linux/kernel/manual-config.nix

emilazy · 2025-07-13T12:34:46Z

BTW, this should be release noted as a breaking change.

vcunat · 2025-08-19T16:59:08Z

I appreciate that the fixes come very fast after reporting them, at least 😄

Add a hack to fix Rust support broken by NixOS/nixpkgs#423933 See https://github.com/NixOS/nixpkgs/pulls/436245 for Nixpkgs upstream fix

Add a hack to fix Rust support broken by NixOS/nixpkgs#423933 See NixOS/nixpkgs#436245 for Nixpkgs upstream fix

The changes in NixOS#423933 caused previously working calls to `pkgs.makeModulesClosure` to produce empty outputs, when paired with `allowMissing = true;`. In turn, this broke boot in those systems, as instead of having kernel modules (as one would expect), instead there was nothing to load. This is not only a weird edge case, as due to how the options compose in the NixOS modules system, `allowMissing` is de-facto required in some use-cases. (See e.g. NixOS#154163 (comment) ) The choice of `allowEmpty` being set to `false` by default matches the default assumptions from `allowMissing`: something should be produced. This *may* break some builds, but it will break them at build time, rather than breaking boot. In this situation setting `allowEmpty` will be the solution to fix this issue. This is a tradeoff: break the usage of `makeModulesClosure` at build time for those that wanted an empty output, or break the usage of `makeModulesClosure` at boot for the others. The latter is much much much preferrable, since it prevents problematic situations where a remote system is stuck not booting, without remote hands. The former situation will only be groan-inducing. This change was tested with Mobile NixOS. Between NixOS#423933 and this change, it *will* produce broken stage-1 systems for modular kernels. With this change, it breaks the builds. ***This is as expected.*** This could be worked around within NixOS outright, by having the `makeModulesClosure` function check for `modules` output, but that might be a worse workaround, and may cause weirder breakage. It especially will make it much weirder to wrap one's head around `makeModulesClosure`. Signed-off-by: Samuel Dionne-Riel <samuel@dionne-riel.com>

r-vdp · 2025-08-25T20:11:08Z

FYI, this broke nixosTests.systemd-boot.no-bootspec, since the synthesize command of the bootspec package expects to find $toplevel/kernel-modules/bzImage.
It should probably look at $toplevel/kernel instead, but that will require patching bootspec.

Instead of `$toplevel/kernel-modules/bzImage`, we use `$toplevel/kernel` to find the kernel image of a generation.

r-vdp · 2025-08-26T15:01:34Z

FYI, this broke nixosTests.systemd-boot.no-bootspec, since the synthesize command of the bootspec package expects to find $toplevel/kernel-modules/bzImage. It should probably look at $toplevel/kernel instead, but that will require patching bootspec.

Fix in #437139

Nixpkgs changed the location of the kernel image in NixOS/nixpkgs#423933.

Since NixOS#423933 nixpkgs kernel have a seperate "modules" output for kernel. This check now acts similar to lib.getBin/lib.getLib etc.

Mic92 · 2025-09-01T13:04:30Z

Here is a proposal to make aggregateModules behave the same as before:

#439178

nixos-discourse · 2025-11-26T11:52:29Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/patching-an-in-tree-linux-kernel-module/22137/3

nix-owners bot requested review from K900, Ma27, NeQuissimus, TredwellGit and alyssais July 10, 2025 01:33

nixpkgs-ci bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: kernel The Linux kernel 8.has: module (update) This PR changes an existing module in `nixos/` labels Jul 10, 2025

nix-owners bot requested a review from thoughtpolice July 10, 2025 01:38

alyssais approved these changes Jul 10, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. labels Jul 10, 2025

jmbaur force-pushed the linux-outputs branch from 51c1224 to 1a44fe7 Compare July 10, 2025 21:08

jmbaur force-pushed the linux-outputs branch 2 times, most recently from 98829e9 to 59aba1f Compare July 12, 2025 21:36

emilazy requested changes Jul 13, 2025

View reviewed changes

pkgs/os-specific/linux/kernel/manual-config.nix Outdated Show resolved Hide resolved

pkgs/os-specific/linux/kernel/manual-config.nix Outdated Show resolved Hide resolved

SaltyKitkat mentioned this pull request Aug 20, 2025

linuxManualConfig: kernelPatches are not applied in patchPhase #435248

Closed

3 tasks

ghost mentioned this pull request Aug 22, 2025

remove mesa-asahi-edge nix-community/nixos-apple-silicon#303

Merged

PedroHLC mentioned this pull request Aug 23, 2025

[Bug] linuxPackages_cachyos broken with stable nixos chaotic-cx/nyx#1161

Closed

This was referenced Aug 23, 2025

GPU drivers not building on latest nixpkgs nix-community/nixos-apple-silicon#342

Closed

linuxManualConfig: reinstate Rust #436245

Merged

dramforever mentioned this pull request Aug 23, 2025

linux-asahi: Fix Rust support nix-community/nixos-apple-silicon#343

Merged

dramforever added a commit to dramforever/nixos-apple-silicon that referenced this pull request Aug 23, 2025

linux-asahi: Fix Rust support

2465c8f

Add a hack to fix Rust support broken by NixOS/nixpkgs#423933 See NixOS/nixpkgs#436245 for Nixpkgs upstream fix

samueldr mentioned this pull request Aug 23, 2025

makeModulesClosure: Add allowEmpty to harden against modules vanishing #436304

Open

13 tasks

r-vdp mentioned this pull request Aug 25, 2025

nixos/tests/systemd-boot: switch to runTest #436897

Merged

13 tasks

r-vdp added a commit to r-vdp/lanzaboote that referenced this pull request Aug 26, 2025

fix: bootspec synthesis was broken because of NixOS/nixpkgs#423933

f276850

Instead of `$toplevel/kernel-modules/bzImage`, we use `$toplevel/kernel` to find the kernel image of a generation.

r-vdp mentioned this pull request Aug 26, 2025

bootspec: 1.0.1 -> 1.1.0 #437139

Merged

13 tasks

r-vdp added a commit to r-vdp/bootspec that referenced this pull request Aug 26, 2025

bootspec: fix synthesis, which was broken since NixOS/nixpkgs#423933

9a33dd8

cole-h pushed a commit to DeterminateSystems/bootspec that referenced this pull request Aug 26, 2025

bootspec: fix synthesis with new Nixpkgs (#204)

3d5bc64

Nixpkgs changed the location of the kernel image in NixOS/nixpkgs#423933.

emilazy mentioned this pull request Aug 26, 2025

CI labelling and branch for PRs that rebuild all NixOS tests #437208

Closed

digitalrane mentioned this pull request Aug 27, 2025

gpio-utils: Build failure #437404

Closed

3 tasks

frantisekhanzlikbl mentioned this pull request Aug 29, 2025

Building vmWithDisko fails in kernel-modules-shrunk nix-community/disko#1114

Closed

aanderse mentioned this pull request Aug 30, 2025

modules/kernel: in-tree kernel modules are now in a separate derivati… finix-community/finix#23

Merged

Mic92 mentioned this pull request Sep 1, 2025

aggregateModules: auto-extract modules output from kernel derivations #439178

Closed

13 tasks

Diamondtroller mentioned this pull request Sep 8, 2025

Flake update action breaks flake nabam/nixos-rockchip#58

Closed

Uh oh!

Conversation

jmbaur commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

K900 commented Jul 10, 2025

Uh oh!

jmbaur commented Jul 10, 2025

Uh oh!

emilazy commented Jul 10, 2025

Uh oh!

jmbaur commented Jul 10, 2025

Uh oh!

alyssais left a comment

Choose a reason for hiding this comment

Uh oh!

emilazy commented Jul 10, 2025

Uh oh!

jmbaur commented Jul 10, 2025

Uh oh!

jmbaur commented Jul 10, 2025

Uh oh!

jmbaur commented Jul 12, 2025

Uh oh!

emilazy commented Jul 12, 2025

Uh oh!

jmbaur commented Jul 12, 2025

Uh oh!

Uh oh!

Uh oh!

emilazy commented Jul 13, 2025

Uh oh!

vcunat commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r-vdp commented Aug 25, 2025

Uh oh!

r-vdp commented Aug 26, 2025

Uh oh!

Mic92 commented Sep 1, 2025

Uh oh!

nixos-discourse commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Comments

jmbaur commented Jul 10, 2025 •

edited

Loading

vcunat commented Aug 19, 2025 •

edited

Loading