Skip to content

Commit

Permalink
RFC 127: Initial implementation
Browse files Browse the repository at this point in the history
  • Loading branch information
@piegamesde
piegamesde committed Sep 23, 2023
1 parent 1fb5607 commit 08d0536
Show file tree
Hide file tree
Showing 5 changed files with 472 additions and 44 deletions.
53 changes: 53 additions & 0 deletions doc/using/configuration.chapter.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ Nix comes with certain defaults about what packages can and cannot be installed,

- The package has known security vulnerabilities but has not or can not be updated for some reason, and a list of issues has been entered in to the package's `meta.knownVulnerabilities`.

- There are problems for packages which must be acknowledged, e.g. deprecation notices.

Note that all this is checked during evaluation already, and the check includes any package that is evaluated. In particular, all build-time dependencies are checked. `nix-env -qa` will (attempt to) hide any packages that would be refused.

Each of these criteria can be altered in the nixpkgs configuration.
Expand Down Expand Up @@ -169,6 +171,57 @@ There are several ways to tweak how Nix handles a package which has been marked

Note that `permittedInsecurePackages` is only checked if `allowInsecurePredicate` is not specified.

## Packages with problems {#sec-problems}

A package may have several problems associated with it.
These can be either manually declared in `meta.problems`, or automatically generated from its other `meta` attributes.
Each problem has a name, a "kind", a message, and optionally a list of URLs.
Not all kinds can be manually specified in `meta.problems`, and some kinds can exist only up to once per package.
Currently, the following problem kinds are known (with more reserved to be added in the future):

- "removal": The package is planned to be removed some time in the future. Unique.
- "deprecated": The package relies on software which has reached its end of life.
- "maintainerless": Automatically generated for packages with `meta.maintainers == []`. Unique, not manually specifiable.

Each problem has a handler that deals with it, which can be one of "error", "warn" or "ignore".
"error" will disallow evaluating a package, while "warn" will simply print a message to the log.

The handler for problems can be specified using `config.problems.handlers.${packageName}.${problemName} = "${handler}";`.

There is also the possibility to specify some generic matchers, which can set a handler for more than a specific problem of a specific package.
This works through the `config.problems.matchers` option:

```nix
{
problems.matchers = [
# Fail to build any packages which are about to be removed anyway
{
kind = "removal";
handler = "error";
}

# Get warnings when using packages with no declared maintainers
{
kind = "maintainerless";
handler = "warn";
}

# You deeply care about this package and want to absolutely know when it has any problems
{
package = "hello";
handler = "error";
}
];
}
```

Matchers can match one or more of package name, problem name or problem kind.
If multiple conditions are present, all must be met to match.
If multiple matchers match a problem, then the highest severity handler will be chosen.
The current default value contains `{ kind = "removal"; handler = "error"; }`, meaning that people will be notified about package removals in advance.

Package names for both `problems.handlers` and `problems.matchers` are taken from `lib.getName`, which looks at the `pname` first and falls back to extracting the "pname" part from the `name` attribute.

## Modify packages via `packageOverrides` {#sec-modify-via-packageOverrides}

You can define a function called `packageOverrides` in your local `~/.config/nixpkgs/config.nix` to override Nix packages. It must be a function that takes pkgs as an argument and returns a modified set of packages.
Expand Down
216 changes: 188 additions & 28 deletions pkgs/stdenv/generic/check-meta.nix
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,20 @@
{ lib, config, hostPlatform }:

let
inherit (import ./problems.nix { inherit lib; })
problemKindsManual
problemKindsUnique'
;
generateProblems =
(import ./problems.nix { inherit lib; }).generateProblems
{ inherit hasNoMaintainers;}
config.problems;

# If we're in hydra, we can dispense with the more verbose error
# messages and make problems easier to spot.
inHydra = config.inHydra or false;
# Allow the user to opt-into additional warnings, e.g.
# import <nixpkgs> { config = { showDerivationWarnings = [ "maintainerless" ]; }; }
showWarnings = config.showDerivationWarnings;

# Do not use this in new code, use `lib.getName` instead
getName = attrs: attrs.name or ("${attrs.pname or "«name-missing»"}-${attrs.version or "«version-missing»"}");

allowUnfree = config.allowUnfree
Expand Down Expand Up @@ -52,8 +59,7 @@ let
hasLicense attrs &&
isUnfree (lib.lists.toList attrs.meta.license);

hasNoMaintainers = attrs:
attrs ? meta.maintainers && (lib.length attrs.meta.maintainers) == 0;
hasNoMaintainers = pkg: lib.length (pkg.meta.maintainers or []) == 0;

isMarkedBroken = attrs: attrs.meta.broken or false;

Expand Down Expand Up @@ -87,7 +93,6 @@ let
allowInsecurePredicate attrs ||
builtins.getEnv "NIXPKGS_ALLOW_INSECURE" == "1";


isNonSource = sourceTypes: lib.lists.any (t: !t.isSource) sourceTypes;

hasNonSourceProvenance = attrs:
Expand Down Expand Up @@ -124,22 +129,26 @@ let
unsupported = remediate_allowlist "UnsupportedSystem" (x: "");
blocklisted = x: "";
insecure = remediate_insecure;
problem = remediate_problem;
broken-outputs = remediateOutputsToInstall;
unknown-meta = x: "";
maintainerless = x: "";
};

remediation_env_var = allow_attr: {
Unfree = "NIXPKGS_ALLOW_UNFREE";
Broken = "NIXPKGS_ALLOW_BROKEN";
UnsupportedSystem = "NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM";
NonSource = "NIXPKGS_ALLOW_NONSOURCE";
}.${allow_attr};

remediation_phrase = allow_attr: {
Unfree = "unfree packages";
Broken = "broken packages";
UnsupportedSystem = "packages that are unsupported for this system";
NonSource = "packages not built from source";
}.${allow_attr};

remediate_predicate = predicateConfigAttr: attrs:
''
Expand All @@ -150,8 +159,8 @@ let
}
'';

# flakeNote will be printed in the remediation messages below.
flakeNote = "
# flakeNote will be printed in the remediation messages below.
flakeNote = "
Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
(Flake) command, `--impure` must be passed in order to read this
environment variable.
Expand Down Expand Up @@ -209,6 +218,59 @@ let
'';

remediate_problem = pkg: let
pkgName = lib.getName pkg;

fullMessage = problem: (if problem.kind != problem.name then problem.kind + ": " else "") + problem.name + ": " + problem.message;
urlMessage = problem: let
urls = problem.urls or [];
in
lib.concatMapStrings (url: "\n - " + url) urls;

printProblem = problem: "- " + fullMessage problem + urlMessage problem;

problems = generateProblems pkg;

ignorePattern = indentation: lib.pipe problems [
(lib.filter ({handler, ...}: handler == "error"))
(builtins.map ({name, ...}: ''"${pkgName}"."${name}" = "warn";''))
(lib.concatStringsSep ("\n" + indentation))
];
in ''
Package problems:
${lib.pipe problems [
(lib.filter ({handler, ...}: handler == "error"))
(builtins.map printProblem)
(lib.concatStringsSep "\n")
]}
You can use it anyway by ignoring its problems, using one of the
following methods:
a) For `nixos-rebuild` you can add "warn" or "ignore" entries to
`nixpkgs.config.problems.handlers` inside configuration.nix,
like this:
{
nixpkgs.config.problems.handlers = {
${ignorePattern " "}
};
}
b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
"warn" or "ignore" to `problems.handlers` in
~/.config/nixpkgs/config.nix, like this:
{
problems.handlers = {
${ignorePattern " "}
};
}
'';

remediateOutputsToInstall = attrs: let
expectedOutputs = attrs.meta.outputsToInstall or [];
actualOutputs = attrs.outputs or [ "out" ];
Expand Down Expand Up @@ -237,14 +299,16 @@ let
else throw;
in handler msg;

handleEvalWarning = { meta, attrs }: { reason , errormsg ? "" }:
handleEvalWarning = { meta, attrs }: { reason, errormsg ? "" }:
let
remediationMsg = (builtins.getAttr reason remediation) attrs;
msg = if inHydra then "Warning while evaluating ${getName attrs}: «${reason}»: ${errormsg}"
else "Package ${getName attrs} in ${pos_str meta} ${errormsg}, continuing anyway."
+ (lib.optionalString (remediationMsg != "") "\n${remediationMsg}");
isEnabled = lib.findFirst (x: x == reason) null showWarnings;
in if isEnabled != null then builtins.trace msg true else true;
msg =
if inHydra then
"Warning while evaluating ${getName attrs}: «${reason}»: ${errormsg}"
else
"Package ${getName attrs} in ${pos_str meta} ${errormsg}, continuing anyway."
;
in
builtins.trace msg true;

# Deep type-checking. Note that calling `type.check` is not enough: see `lib.mkOptionType`'s documentation.
# We don't include this in lib for now because this function is flawed: it accepts things like `mkIf true 42`.
Expand All @@ -256,6 +320,7 @@ let
in eval.success;

# TODO make this into a proper module and use the generic option documentation generation?
# Also, this would allow specifying default values instead of having to drag null checks across the entire file
metaTypes = with lib.types; rec {
# These keys are documented
description = str;
Expand All @@ -278,6 +343,8 @@ let
unfree = bool;
unsupported = bool;
insecure = bool;
# This is checked in more detail further down
problems = attrsOf (attrsOf anything);
# TODO: refactor once something like Profpatsch's types-simple will land
# This is currently dead code due to https://github.com/NixOS/nix/issues/2532
tests = attrsOf (mkOptionType {
Expand Down Expand Up @@ -312,17 +379,84 @@ let
badPlatforms = platforms;
};

checkMetaAttr = k: v:
if metaTypes?${k} then
if typeCheck metaTypes.${k} v then
# Type of a meta.problems.* value
problemTypes = with lib.types; {
# Only allow some problem kinds to be used here in `meta`
kind = enum problemKindsManual;
message = str;
urls = listOf str;
};

# Check that a value matches a specific type. Returns an error message, or null if the check passed
checkMetaAttr = namespace: types: key: val:
if types ? ${key} then
if typeCheck types.${key} val then
null
else
"key 'meta.${k}' has invalid value; expected ${metaTypes.${k}.description}, got\n ${
lib.generators.toPretty { indent = " "; } v
"key '${namespace}.${key}' has invalid value; expected ${types.${key}.description}, got\n ${
lib.generators.toPretty { indent = " "; } val
}"
else
"key 'meta.${k}' is unrecognized; expected one of: \n [${lib.concatMapStringsSep ", " (x: "'${x}'") (lib.attrNames metaTypes)}]";
checkMeta = meta: lib.optionals config.checkMeta (lib.remove null (lib.mapAttrsToList checkMetaAttr meta));
"key '${namespace}.${key}' is unrecognized; expected one of: \n [${lib.concatMapStringsSep ", " (x: "'${x}'") (lib.attrNames types)}]"
;

checkMeta =
meta:
# Basic attribute checks
lib.optionals config.checkMeta (
lib.remove null (
lib.mapAttrsToList (checkMetaAttr "meta" metaTypes) meta
)
)

# Extended checks for problems, as they do not fit the module system
++ lib.optionals (meta ? problems) (

# Check problem kinds are correct
lib.pipe meta.problems [
(lib.mapAttrs (name: problem: { kind = name; } // problem))
(lib.mapAttrsToList (
name: problem:
lib.mapAttrsToList (checkMetaAttr "meta.problems.${name}" problemTypes) problem
))
(lib.concatLists)
(lib.remove null)
]

# Check that problem has a message (required field)
++ lib.pipe meta.problems [
(lib.mapAttrsToList (
name: problem:
if problem ? message then
null
else
"key 'meta.problems.${name}.message' is missing"
))
(lib.remove null)
]

# Check that some problem kinds are unqiue
++ lib.pipe meta.problems [
(lib.mapAttrs (name: problem: { kind = name; } // problem))
# Count the number of instances of each problem kind,
# returns an attrset mapping kind -> count
(lib.foldlAttrs
(count: name: problem:
count // { ${problem.kind} = (count.${problem.kind} or []) ++ [ name ]; }
)
{}
)
# We only care about those that must be unique
(builtins.intersectAttrs problemKindsUnique')
(lib.filterAttrs (_: names: builtins.length names > 1))
(lib.mapAttrsToList (
kind: names:
"keys [ ${
lib.concatMapStringsSep " " (name: "'meta.problems.${name}'") names
} ] all have the same problem kind, which is not allowed for kind '${kind}'"
))
]
);

checkOutputsToInstall = attrs: let
expectedOutputs = attrs.meta.outputsToInstall or [];
Expand Down Expand Up @@ -354,8 +488,12 @@ let
unsupported = hasUnsupportedPlatform attrs;
insecure = isMarkedInsecure attrs;
} // (
# Check meta attribute types first, to make sure it is always called even when there are other issues
# Note that this is not a full type check and functions below still need to by careful about their inputs!
let res = checkMeta (attrs.meta or {}); in if res != [] then
{ valid = "no"; reason = "unknown-meta"; errormsg = "has malformed metadata:${lib.concatMapStrings (x: "\n\t - " + x) res}"; }
# --- Put checks that cannot be ignored here ---
if checkOutputsToInstall attrs then
else if checkOutputsToInstall attrs then
{ valid = "no"; reason = "broken-outputs"; errormsg = "has invalid meta.outputsToInstall"; }

# --- Put checks that can be ignored here ---
Expand All @@ -365,6 +503,8 @@ let
{ valid = "no"; reason = "blocklisted"; errormsg = "has a blocklisted license (‘${showLicense attrs.meta.license}’)"; }
else if hasDeniedNonSourceProvenance attrs then
{ valid = "no"; reason = "non-source"; errormsg = "contains elements not built from source (‘${showSourceType attrs.meta.sourceProvenance}’)"; }

# --- Put checks that can be ignored here ---
else if !allowBroken && attrs.meta.broken or false then
{ valid = "no"; reason = "broken"; errormsg = "is marked as broken"; }
else if !allowUnsupportedSystem && hasUnsupportedPlatform attrs then
Expand All @@ -382,11 +522,31 @@ let
}
else if !(hasAllowedInsecure attrs) then
{ valid = "no"; reason = "insecure"; errormsg = "is marked as insecure"; }

# --- warnings ---
# Please also update the type in /pkgs/top-level/config.nix alongside this.
else if hasNoMaintainers attrs then
{ valid = "warn"; reason = "maintainerless"; errormsg = "has no maintainers"; }
# --- RFC 127 problems ---
# Please also update the type in pkgs/top-level/config.nix alongside this.
else
let
problems = (generateProblems attrs);
hasUnignoredProblems = lib.any ({handler, ...}: handler != "ignore") problems;
hasErrorProblems = lib.any ({handler, ...}: handler == "error") problems;
in
if hasUnignoredProblems then
{
valid = if hasErrorProblems then "no" else "warn";
reason = "problem";
errormsg =
if hasErrorProblems then
"has some problem that must be acknowledged"
else
"has the following problems: [ ${
lib.pipe problems [
(builtins.map (lib.getAttr "name"))
(builtins.map (x: ''"${x}"''))
(lib.concatStringsSep " ")
]
} ]"
;
}
# -----
else { valid = "yes"; });

Expand Down
Loading

0 comments on commit 08d0536

Please sign in to comment.