Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/release-notes/rl-2511.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## Highlights {#sec-nixpkgs-release-25.11-highlights}
<!-- To avoid merge conflicts, consider adding your item at an arbitrary place in the list instead. -->

- Create the first release note entry in this section!
- Added `allowVariants` to gate availability of package sets like `pkgsLLVM`, `pkgsMusl`, `pkgsZig`, etc.

## Backward Incompatibilities {#sec-nixpkgs-release-25.11-incompatibilities}

Expand Down
12 changes: 12 additions & 0 deletions pkgs/top-level/config.nix
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,18 @@ let
'';
};

allowVariants = mkOption {
type = types.bool;
default = true;
description = ''
Whether to expose the nixpkgs variants.

Variants are instances of the current nixpkgs instance with different stdenvs or other applied options.
This allows for using different toolchains, libcs, or global build changes across nixpkgs.
Disabling can ensure nixpkgs is only building for the platform which you specified.
'';
};

cudaSupport = mkMassRebuild {
type = types.bool;
default = false;
Expand Down
114 changes: 12 additions & 102 deletions pkgs/top-level/stage.nix
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,16 @@ let

aliases = self: super: lib.optionalAttrs config.allowAliases (import ./aliases.nix lib self super);

variants = import ./variants.nix {
inherit
lib
nixpkgsFun
stdenv
overlays
makeMuslParsedPlatform
;
};
Comment on lines +194 to +202
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we now have an allowVariants config option - but it doesn't do anything. Its value is not used anywhere. All the variants are still available when setting it to false.

What was the intent to introduce this again, when apparently nobody has used it, yet?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to set it to true in CI once we have a competent replacement. #410056 is one that I've worked on. We're kinda blocked on that with GCC NG, @Ericson2314 and I are working on it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to set it to true in CI

I think this is just a typo and you mean false.

I think you missed my point: When I use allowVariants = false, then it doesn't work. It doesn't do anything. The PR body mentions this:

Also add mkOptionalOverlay, this allows us to throw an error when an attribute from an overlay isn't available unless enabled.

But apparently this has been lost during later refactors of the PR. Now all variants are unconditionally included all the time.

We currently have a prominently placed release note (including a backport) which mentions that we have a new config option allowVariants, but this config option is disfunctional.

I hope that clarifies it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a typo and you mean false.

Yes, I meant false. It was a typo.

I think you missed my point: When I use allowVariants = false, then it doesn't work. It doesn't do anything. The PR body mentions this:

Also add mkOptionalOverlay, this allows us to throw an error when an attribute from an overlay isn't available unless enabled.

But apparently this has been lost during later refactors of the PR. Now all variants are unconditionally included all the time.

mkOptionalOverlay was removed due to a review.

We currently have a prominently placed release note (including a backport) which mentions that we have a new config option allowVariants, but this config option is disfunctional.

I hope that clarifies it.

Huh, lemme check if this works. I remember testing it and it did work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's easy to confirm that it can't work. Just grep for allowVariants. It's only used in top-level/config and the release notes :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh now I see what's going on, after #400351 (comment) I didn't add in lib.optionalAttrs for the variants attribute in pkgs/top-level/stage.nix.


# stdenvOverrides is used to avoid having multiple of versions
# of certain dependencies that were used in bootstrapping the
# standard environment.
Expand All @@ -214,6 +224,7 @@ let
# - pkgsCross.<system> where system is a member of lib.systems.examples
# - pkgsMusl
# - pkgsi686Linux
# NOTE: add new non-critical package sets to "pkgs/top-level/variants.nix"
otherPackageSets = self: super: {
# This maps each entry in lib.systems.examples to its own package
# set. Each of these will contain all packages cross compiled for
Expand All @@ -222,69 +233,6 @@ let
# Raspberry Pi.
pkgsCross = lib.mapAttrs (n: crossSystem: nixpkgsFun { inherit crossSystem; }) lib.systems.examples;

pkgsLLVM = nixpkgsFun {
overlays = [
(self': super': {
pkgsLLVM = super';
})
] ++ overlays;
# Bootstrap a cross stdenv using the LLVM toolchain.
# This is currently not possible when compiling natively,
# so we don't need to check hostPlatform != buildPlatform.
crossSystem = stdenv.hostPlatform // {
useLLVM = true;
linker = "lld";
};
};

pkgsArocc = nixpkgsFun {
overlays = [
(self': super': {
pkgsArocc = super';
})
] ++ overlays;
# Bootstrap a cross stdenv using the Aro C compiler.
# This is currently not possible when compiling natively,
# so we don't need to check hostPlatform != buildPlatform.
crossSystem = stdenv.hostPlatform // {
useArocc = true;
linker = "lld";
};
};

pkgsZig = nixpkgsFun {
overlays = [
(self': super': {
pkgsZig = super';
})
] ++ overlays;
# Bootstrap a cross stdenv using the Zig toolchain.
# This is currently not possible when compiling natively,
# so we don't need to check hostPlatform != buildPlatform.
crossSystem = stdenv.hostPlatform // {
useZig = true;
linker = "lld";
};
};

# All packages built with the Musl libc. This will override the
# default GNU libc on Linux systems. Non-Linux systems are not
# supported. 32-bit is also not supported.
pkgsMusl =
if stdenv.hostPlatform.isLinux && stdenv.buildPlatform.is64bit then
nixpkgsFun {
overlays = [
(self': super': {
pkgsMusl = super';
})
] ++ overlays;
${if stdenv.hostPlatform == stdenv.buildPlatform then "localSystem" else "crossSystem"} = {
config = lib.systems.parse.tripleFromSystem (makeMuslParsedPlatform stdenv.hostPlatform.parsed);
};
}
else
throw "Musl libc only supports 64-bit Linux systems.";

# All packages built for i686 Linux.
# Used by wine, firefox with debugging version of Flash, ...
pkgsi686Linux =
Expand Down Expand Up @@ -376,45 +324,6 @@ let
// stdenv.hostPlatform.gcc or { };
};
});

# Full package set with rocm on cuda off
# Mostly useful for asserting pkgs.pkgsRocm.torchWithRocm == pkgs.torchWithRocm and similar
pkgsRocm = nixpkgsFun ({
config = super.config // {
cudaSupport = false;
rocmSupport = true;
};
});

pkgsExtraHardening = nixpkgsFun {
overlays = [
(
self': super':
{
pkgsExtraHardening = super';
stdenv = super'.withDefaultHardeningFlags (
super'.stdenv.cc.defaultHardeningFlags
++ [
"shadowstack"
"nostrictaliasing"
"pacret"
"trivialautovarinit"
]
) super'.stdenv;
glibc = super'.glibc.override rec {
enableCET = if self'.stdenv.hostPlatform.isx86_64 then "permissive" else false;
enableCETRuntimeDefault = enableCET != false;
};
}
// lib.optionalAttrs (with super'.stdenv.hostPlatform; isx86_64 && isLinux) {
# causes shadowstack disablement
pcre = super'.pcre.override { enableJit = false; };
pcre-cpp = super'.pcre-cpp.override { enableJit = false; };
pcre16 = super'.pcre16.override { enableJit = false; };
}
)
] ++ overlays;
};
};

# The complete chain of package set builders, applied from top to bottom.
Expand All @@ -430,6 +339,7 @@ let
allPackages
otherPackageSets
aliases
variants
configOverrides
]
++ overlays
Expand Down
127 changes: 127 additions & 0 deletions pkgs/top-level/variants.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
/*
This file contains all of the different variants of nixpkgs instances.

Unlike the other package sets like pkgsCross, pkgsi686Linux, etc., this
contains non-critical package sets. The intent is to be a shorthand
for things like using different toolchains in every package in nixpkgs.
*/
{
lib,
stdenv,
nixpkgsFun,
overlays,
makeMuslParsedPlatform,
}:
let
makeLLVMParsedPlatform =
parsed:
(
parsed
// {
abi = lib.systems.parse.abis.llvm;
}
);
in
self: super: {
pkgsLLVM = nixpkgsFun {
overlays = [
(self': super': {
pkgsLLVM = super';
})
] ++ overlays;
# Bootstrap a cross stdenv using the LLVM toolchain.
# This is currently not possible when compiling natively,
# so we don't need to check hostPlatform != buildPlatform.
crossSystem = stdenv.hostPlatform // {
useLLVM = true;
linker = "lld";
};
};

pkgsArocc = nixpkgsFun {
overlays = [
(self': super': {
pkgsArocc = super';
})
] ++ overlays;
# Bootstrap a cross stdenv using the Aro C compiler.
# This is currently not possible when compiling natively,
# so we don't need to check hostPlatform != buildPlatform.
crossSystem = stdenv.hostPlatform // {
useArocc = true;
linker = "lld";
};
};

pkgsZig = nixpkgsFun {
overlays = [
(self': super': {
pkgsZig = super';
})
] ++ overlays;
# Bootstrap a cross stdenv using the Zig toolchain.
# This is currently not possible when compiling natively,
# so we don't need to check hostPlatform != buildPlatform.
crossSystem = stdenv.hostPlatform // {
useZig = true;
linker = "lld";
};
};

# All packages built with the Musl libc. This will override the
# default GNU libc on Linux systems. Non-Linux systems are not
# supported. 32-bit is also not supported.
pkgsMusl =
if stdenv.hostPlatform.isLinux && stdenv.buildPlatform.is64bit then
nixpkgsFun {
overlays = [
(self': super': {
pkgsMusl = super';
})
] ++ overlays;
${if stdenv.hostPlatform == stdenv.buildPlatform then "localSystem" else "crossSystem"} = {
config = lib.systems.parse.tripleFromSystem (makeMuslParsedPlatform stdenv.hostPlatform.parsed);
};
}
else
throw "Musl libc only supports 64-bit Linux systems.";

# Full package set with rocm on cuda off
# Mostly useful for asserting pkgs.pkgsRocm.torchWithRocm == pkgs.torchWithRocm and similar
pkgsRocm = nixpkgsFun ({
config = super.config // {
cudaSupport = false;
rocmSupport = true;
};
});

pkgsExtraHardening = nixpkgsFun {
overlays = [
(
self': super':
{
pkgsExtraHardening = super';
stdenv = super'.withDefaultHardeningFlags (
super'.stdenv.cc.defaultHardeningFlags
++ [
"shadowstack"
"nostrictaliasing"
"pacret"
"trivialautovarinit"
]
) super'.stdenv;
glibc = super'.glibc.override rec {
enableCET = if self'.stdenv.hostPlatform.isx86_64 then "permissive" else false;
enableCETRuntimeDefault = enableCET != false;
};
}
// lib.optionalAttrs (with super'.stdenv.hostPlatform; isx86_64 && isLinux) {
# causes shadowstack disablement
pcre = super'.pcre.override { enableJit = false; };
pcre-cpp = super'.pcre-cpp.override { enableJit = false; };
pcre16 = super'.pcre16.override { enableJit = false; };
}
)
] ++ overlays;
};
}