From fddb8ee84ebbb2bae4454b642e473c78af815330 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Sun, 7 May 2023 18:18:33 -0500 Subject: [PATCH 01/17] use-cases: deeply replace package --- use-cases/deep-replace.org | 68 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 use-cases/deep-replace.org diff --git a/use-cases/deep-replace.org b/use-cases/deep-replace.org new file mode 100644 index 0000000..4a05d24 --- /dev/null +++ b/use-cases/deep-replace.org @@ -0,0 +1,68 @@ +#+TITLE: Deeply Replacing Packages in Nixpkgs + +* Goal + +Replace all "instances" of a package across a dependency graph. +This may be replacing all usage in a package set, collection of package sets, +a collection of ad-hoc recipes, or a collection of flakes. + +A possible motivation for _deep replacement_ may be to ensure that a security +fix provided by a new release of a piece of software is used "everywhere" in +the dependency graph. + + +** Approach without Nixpkgs Modules + +While the precise organization of packages will effect the complexity and +effort required to perform _deep replacement_, in general we say that this +is accomplished using helper functions such as =callPackageWith=, +=makeScope=, =override=, or =extend=. + +#+NAME: Trivial Example ( Single Package Set, Shallow Replacement ) +#+BEGIN_SRC nix +let + nixpkgs = builtins.getFlake "nixpkgs"; + pkgsFor = builtins.getAttr builtins.currentSystem nixpkgs.legacyPackages; + patchFoo = final: prev: { foo = final.callPackage ./my-pkgs/foo {}; }; +in ( pkgsFor.extend patchFoo ).bar +#+END_SRC + + +#+NAME: Deep Replacement Example ( Single Package Set ) +#+BEGIN_SRC nix +let + nixpkgs = builtins.getFlake "nixpkgs"; + pkgsFor = builtins.getAttr builtins.currentSystem nixpkgs.legacyPackages; + patchFoo = final: prev: { foo = final.callPackage ./my-pkgs/foo {}; }; +in ( pkgsFor.extend patchFoo ).bar +#+END_SRC + + +** Approach with Nixpkgs Modules + + +* Concrete Examples + +** Approach without Nixpkgs Modules + + +** Approach with Nixpkgs Modules + + +* Current Problems + + +** Approach without Nixpkgs Modules + +With this approach we have three main sources of complexity, none of which +truly prevent a user from accomplishing their goal; but we might suffice to +say that it may be worthwhile to provide a more straightforward mechanism +for handling this use case. + +1. [[https://github.com/NixOS/nixpkgs/blob/master/lib/customization.nix][github:NixOS/nixpkgs://lib/customization.nix]] routines aren't intuitively understood by many users. +2. Nested scopes are difficult to locate, and the relationship between + parent scopes and child scopes is not opaque to users. +3. With ad-hoc recipes and flakes there isn't standardized usage of + =overlays= that allow deep overriding of packages transitively. + - Improved guidance on the use of =overlays= and =follows= in =flakes= + could help a bit here. From 9c5ac53e1b70180b34d06437580f62537a2d9aa7 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Mon, 8 May 2023 06:56:38 -0500 Subject: [PATCH 02/17] focus on existing solution --- use-cases/deep-replace.org | 32 ++++---------------------------- 1 file changed, 4 insertions(+), 28 deletions(-) diff --git a/use-cases/deep-replace.org b/use-cases/deep-replace.org index 4a05d24..bd0ec1c 100644 --- a/use-cases/deep-replace.org +++ b/use-cases/deep-replace.org @@ -1,6 +1,6 @@ #+TITLE: Deeply Replacing Packages in Nixpkgs -* Goal +* General Information Replace all "instances" of a package across a dependency graph. This may be replacing all usage in a package set, collection of package sets, @@ -11,24 +11,13 @@ fix provided by a new release of a piece of software is used "everywhere" in the dependency graph. -** Approach without Nixpkgs Modules +* Concrete Examples While the precise organization of packages will effect the complexity and effort required to perform _deep replacement_, in general we say that this is accomplished using helper functions such as =callPackageWith=, =makeScope=, =override=, or =extend=. -#+NAME: Trivial Example ( Single Package Set, Shallow Replacement ) -#+BEGIN_SRC nix -let - nixpkgs = builtins.getFlake "nixpkgs"; - pkgsFor = builtins.getAttr builtins.currentSystem nixpkgs.legacyPackages; - patchFoo = final: prev: { foo = final.callPackage ./my-pkgs/foo {}; }; -in ( pkgsFor.extend patchFoo ).bar -#+END_SRC - - -#+NAME: Deep Replacement Example ( Single Package Set ) #+BEGIN_SRC nix let nixpkgs = builtins.getFlake "nixpkgs"; @@ -37,31 +26,18 @@ let in ( pkgsFor.extend patchFoo ).bar #+END_SRC - -** Approach with Nixpkgs Modules - - -* Concrete Examples - -** Approach without Nixpkgs Modules - - -** Approach with Nixpkgs Modules - - * Current Problems - -** Approach without Nixpkgs Modules - With this approach we have three main sources of complexity, none of which truly prevent a user from accomplishing their goal; but we might suffice to say that it may be worthwhile to provide a more straightforward mechanism for handling this use case. 1. [[https://github.com/NixOS/nixpkgs/blob/master/lib/customization.nix][github:NixOS/nixpkgs://lib/customization.nix]] routines aren't intuitively understood by many users. + 2. Nested scopes are difficult to locate, and the relationship between parent scopes and child scopes is not opaque to users. + 3. With ad-hoc recipes and flakes there isn't standardized usage of =overlays= that allow deep overriding of packages transitively. - Improved guidance on the use of =overlays= and =follows= in =flakes= From b0f8ad9baa4da214606fc9e4cdcfe36ca926b564 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 08:50:28 -0500 Subject: [PATCH 03/17] various changes and rename --- use-cases/deep-replace.md | 61 ++++++++++++++++++++++++++++++++++++++ use-cases/deep-replace.org | 44 --------------------------- 2 files changed, 61 insertions(+), 44 deletions(-) create mode 100644 use-cases/deep-replace.md delete mode 100644 use-cases/deep-replace.org diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md new file mode 100644 index 0000000..a806288 --- /dev/null +++ b/use-cases/deep-replace.md @@ -0,0 +1,61 @@ +# Deeply Replacing Packages in Nixpkgs + +## General Information + +Replace all "instances" of a package across a dependency graph. +This may be replacing all usage in a package set, collection of package sets, +a collection of ad-hoc recipes, or a collection of flakes. + +A possible motivation for _deep replacement_ may be to ensure that a security +fix provided by a new release of a piece of software is used "everywhere" in +the dependency graph. + + +## Concrete Examples + +While the precise organization of packages will effect the complexity and +effort required to perform _deep replacement_, in general we say that this +is accomplished using helper functions such as `extend`, `overrideScope'`, and +`appendOverlays`, as well as the configuration field `overlays`. + +### Simple Overlay +```nix +pkgs.extend ( final: prev: { foo = final.callPackage ./my-pkgs/foo {}; } ) +``` + + +### Nested Overlay + +```nix +pkgs.extend ( final: prev: { + nodejs = prev.nodejs-14_x; + nodePackages = prev.nodePackages.extend ( nfinal: nprev: { + bar = nfinal.callPackage ./my-pkgs/bar {}; + } ); +} ) +``` + + +## Current Problems + +With this approach we have three main sources of complexity, none of which +truly prevent a user from accomplishing their goal; but we might suffice to +say that it may be worthwhile to provide a more straightforward mechanism +for handling this use case. + +1. [github:NixOS/nixpkgs://lib/customisation.nix](https://github.com/NixOS/nixpkgs/blob/master/lib/customisation.nix) +routines aren't intuitively understood by many users. + - `self`/`super` and `final`/`prev` are difficult for new users to understand. + - Easy to accidentally trigger infinite recursion. + - Some packages are not _truly_ overridable, which advanced users will only + discover after a lengthy debugging session. + +2. Nested scopes are difficult to locate, and the relationship between + parent scopes and child scopes is not opaque to users. + - See [[Nested Overlays]] example. + +3. With ad-hoc recipes and flakes there isn't standardized usage of + `overlays` that allow deep overriding of packages transitively. + - Improved guidance on the use of `overlays` and `follows` in `flakes` + could help a bit here. + diff --git a/use-cases/deep-replace.org b/use-cases/deep-replace.org deleted file mode 100644 index bd0ec1c..0000000 --- a/use-cases/deep-replace.org +++ /dev/null @@ -1,44 +0,0 @@ -#+TITLE: Deeply Replacing Packages in Nixpkgs - -* General Information - -Replace all "instances" of a package across a dependency graph. -This may be replacing all usage in a package set, collection of package sets, -a collection of ad-hoc recipes, or a collection of flakes. - -A possible motivation for _deep replacement_ may be to ensure that a security -fix provided by a new release of a piece of software is used "everywhere" in -the dependency graph. - - -* Concrete Examples - -While the precise organization of packages will effect the complexity and -effort required to perform _deep replacement_, in general we say that this -is accomplished using helper functions such as =callPackageWith=, -=makeScope=, =override=, or =extend=. - -#+BEGIN_SRC nix -let - nixpkgs = builtins.getFlake "nixpkgs"; - pkgsFor = builtins.getAttr builtins.currentSystem nixpkgs.legacyPackages; - patchFoo = final: prev: { foo = final.callPackage ./my-pkgs/foo {}; }; -in ( pkgsFor.extend patchFoo ).bar -#+END_SRC - -* Current Problems - -With this approach we have three main sources of complexity, none of which -truly prevent a user from accomplishing their goal; but we might suffice to -say that it may be worthwhile to provide a more straightforward mechanism -for handling this use case. - -1. [[https://github.com/NixOS/nixpkgs/blob/master/lib/customization.nix][github:NixOS/nixpkgs://lib/customization.nix]] routines aren't intuitively understood by many users. - -2. Nested scopes are difficult to locate, and the relationship between - parent scopes and child scopes is not opaque to users. - -3. With ad-hoc recipes and flakes there isn't standardized usage of - =overlays= that allow deep overriding of packages transitively. - - Improved guidance on the use of =overlays= and =follows= in =flakes= - could help a bit here. From ba160d9d2a7db6da00ab7015613db0fdce9079e1 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 08:51:51 -0500 Subject: [PATCH 04/17] fmt --- use-cases/deep-replace.md | 1 + 1 file changed, 1 insertion(+) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index a806288..2b1a388 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -19,6 +19,7 @@ is accomplished using helper functions such as `extend`, `overrideScope'`, and `appendOverlays`, as well as the configuration field `overlays`. ### Simple Overlay + ```nix pkgs.extend ( final: prev: { foo = final.callPackage ./my-pkgs/foo {}; } ) ``` From ec1a36df4ec7b94a6b742049df50e598134060bb Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 08:52:35 -0500 Subject: [PATCH 05/17] fix bullets --- use-cases/deep-replace.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index 2b1a388..b3af80d 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -46,10 +46,10 @@ for handling this use case. 1. [github:NixOS/nixpkgs://lib/customisation.nix](https://github.com/NixOS/nixpkgs/blob/master/lib/customisation.nix) routines aren't intuitively understood by many users. - - `self`/`super` and `final`/`prev` are difficult for new users to understand. - - Easy to accidentally trigger infinite recursion. - - Some packages are not _truly_ overridable, which advanced users will only - discover after a lengthy debugging session. + - `self`/`super` and `final`/`prev` are difficult for new users to understand. + - Easy to accidentally trigger infinite recursion. + - Some packages are not _truly_ overridable, which advanced users will only + discover after a lengthy debugging session. 2. Nested scopes are difficult to locate, and the relationship between parent scopes and child scopes is not opaque to users. From 3f80244bc0cffed58c9d38010f1f9b3fca95cd15 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 08:57:33 -0500 Subject: [PATCH 06/17] expand nested overlay example --- use-cases/deep-replace.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index b3af80d..6c1a862 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -27,7 +27,10 @@ pkgs.extend ( final: prev: { foo = final.callPackage ./my-pkgs/foo {}; } ) ### Nested Overlay +This fails to use Node.js v14 in `nodePackages`, which is unexpected to +most users. ```nix +# XXX: Incorrect usage pkgs.extend ( final: prev: { nodejs = prev.nodejs-14_x; nodePackages = prev.nodePackages.extend ( nfinal: nprev: { @@ -36,6 +39,18 @@ pkgs.extend ( final: prev: { } ) ``` +To accomplish use Node.js v14 here the user would need two overlays. +```nix +let + pkgsN14 = pkgs.extend ( final: prev: { nodejs = prev.nodejs-14_x; } ); +in pkgsN14.extend ( final: prev: { + nodePackages = prev.nodePackages.extend ( nfinal: nprev: { + bar = nfinal.callPackage ./my-pkgs/bar {}; + } ); +} ) +``` + + ## Current Problems From 3bcbab3003d9fd7dbeaee52a6228eea6cb61c3b1 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 09:01:33 -0500 Subject: [PATCH 07/17] fix link --- use-cases/deep-replace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index 6c1a862..f236fbc 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -68,7 +68,7 @@ routines aren't intuitively understood by many users. 2. Nested scopes are difficult to locate, and the relationship between parent scopes and child scopes is not opaque to users. - - See [[Nested Overlays]] example. + - See [Nested Overlays](#Nested Overlays) example. 3. With ad-hoc recipes and flakes there isn't standardized usage of `overlays` that allow deep overriding of packages transitively. From 10f8e5b45885fb6f8a74c2ec4a9c9cd69c00dc72 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 09:02:03 -0500 Subject: [PATCH 08/17] fix link --- use-cases/deep-replace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index f236fbc..bb5de5e 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -68,7 +68,7 @@ routines aren't intuitively understood by many users. 2. Nested scopes are difficult to locate, and the relationship between parent scopes and child scopes is not opaque to users. - - See [Nested Overlays](#Nested Overlays) example. + - See [Nested Overlays](Nested Overlays) example. 3. With ad-hoc recipes and flakes there isn't standardized usage of `overlays` that allow deep overriding of packages transitively. From 3062ca3fb849a49a8e3682e091c9ea0f53ac56f5 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 09:03:53 -0500 Subject: [PATCH 09/17] fix link --- use-cases/deep-replace.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index bb5de5e..68913ee 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -25,7 +25,7 @@ pkgs.extend ( final: prev: { foo = final.callPackage ./my-pkgs/foo {}; } ) ``` -### Nested Overlay +### Nested Overlays This fails to use Node.js v14 in `nodePackages`, which is unexpected to most users. @@ -68,7 +68,7 @@ routines aren't intuitively understood by many users. 2. Nested scopes are difficult to locate, and the relationship between parent scopes and child scopes is not opaque to users. - - See [Nested Overlays](Nested Overlays) example. + - See [Nested Overlays](Nested-Overlays) example. 3. With ad-hoc recipes and flakes there isn't standardized usage of `overlays` that allow deep overriding of packages transitively. From 957ade6e1b3db6a0324152776cfb020bcee03c20 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 09:04:18 -0500 Subject: [PATCH 10/17] fix link --- use-cases/deep-replace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index 68913ee..f422f66 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -68,7 +68,7 @@ routines aren't intuitively understood by many users. 2. Nested scopes are difficult to locate, and the relationship between parent scopes and child scopes is not opaque to users. - - See [Nested Overlays](Nested-Overlays) example. + - See [Nested Overlays](#Nested-Overlays) example. 3. With ad-hoc recipes and flakes there isn't standardized usage of `overlays` that allow deep overriding of packages transitively. From 5a44d928100174f873064a4c3e2a08039fb16aef Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 12 May 2023 09:05:59 -0500 Subject: [PATCH 11/17] fix link --- use-cases/deep-replace.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index f422f66..c97e3a1 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -25,6 +25,7 @@ pkgs.extend ( final: prev: { foo = final.callPackage ./my-pkgs/foo {}; } ) ``` + ### Nested Overlays This fails to use Node.js v14 in `nodePackages`, which is unexpected to @@ -68,7 +69,7 @@ routines aren't intuitively understood by many users. 2. Nested scopes are difficult to locate, and the relationship between parent scopes and child scopes is not opaque to users. - - See [Nested Overlays](#Nested-Overlays) example. + - See [Nested Overlays](#Nested-Ex) example. 3. With ad-hoc recipes and flakes there isn't standardized usage of `overlays` that allow deep overriding of packages transitively. From 37c35ffed4ad215c8e3189dae2984199cf9b12f3 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 26 May 2023 09:21:32 -0500 Subject: [PATCH 12/17] add large example --- scratch/nested-extend.nix | 264 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 264 insertions(+) create mode 100644 scratch/nested-extend.nix diff --git a/scratch/nested-extend.nix b/scratch/nested-extend.nix new file mode 100644 index 0000000..d86d79c --- /dev/null +++ b/scratch/nested-extend.nix @@ -0,0 +1,264 @@ +# ============================================================================ # +# +# +# +# ---------------------------------------------------------------------------- # + +let + +# ---------------------------------------------------------------------------- # + + nixpkgs = builtins.getFlake "nixpkgs"; + system = builtins.currentSystem; + pkgsFor = builtins.getAttr system nixpkgs.legacyPackages; + + +# ---------------------------------------------------------------------------- # + + # Causes `nodePackages' to use `node@20' + n20Overlay = final: prev: { nodejs = prev.nodejs-slim_20; }; + + # Add a package that requires `node@20' + myOverlay = final: prev: { + nodePackages = prev.nodePackages.extend ( nfinal: nprev: let + pkg-fun = { runCommand, nodejs, ... }: runCommand "bar" { + meta.nodeVersion = nodejs.version; + } '' + case "$( ${nodejs}/bin/node --version; )" in + v20.*) touch "$out"; ;; + v*) echo "v20 is required" >&2; exit 1; ;; + esac + ''; + in { + bar0 = final.callPackage pkg-fun {}; + bar1 = prev.callPackage pkg-fun {}; + bar2 = pkg-fun { inherit (prev) nodejs; inherit (final) runCommand; }; + bar3 = pkg-fun { inherit (final) nodejs; inherit (prev) runCommand; }; + bar4 = pkg-fun { inherit (prev) nodejs runCommand; }; + bar5 = pkg-fun { inherit (final) nodejs runCommand; }; + } ); + }; + + +# ---------------------------------------------------------------------------- # + + # Causes `nodePackages' to use `node@18' + n18Overlay = final: prev: { nodejs = prev.nodejs-18_x; }; + + # Add a package that requires `node@18' + friendOverlay = final: prev: { + nodePackages = prev.nodePackages.extend ( nfinal: nprev: let + pkg-fun = { runCommand, nodejs, ... }: runCommand "quux" { + meta.nodeVersion = nodejs.version; + } '' + case "$( ${nodejs}/bin/node --version; )" in + v18.*) touch "$out"; ;; + v*) echo "v18 is required" >&2; exit 1; ;; + esac + ''; + in { + quux0 = final.callPackage pkg-fun {}; + quux1 = prev.callPackage pkg-fun {}; + quux2 = pkg-fun { inherit (prev) nodejs; inherit (final) runCommand; }; + quux3 = pkg-fun { inherit (final) nodejs; inherit (prev) runCommand; }; + quux4 = pkg-fun { inherit (prev) nodejs runCommand; }; + quux5 = pkg-fun { inherit (final) nodejs runCommand; }; + } ); + }; + + +# ---------------------------------------------------------------------------- # + + extractNodeVersions = let + targets = [ + "bar0" "bar1" "bar2" "bar3" "bar4" "bar5" + "quux0" "quux1" "quux2" "quux3" "quux4" "quux5" + ]; + in pkgs: let + get = name: { + inherit name; + value = ( builtins.getAttr name pkgs.nodePackages ).meta.nodeVersion; + }; + in builtins.listToAttrs ( map get targets ); + + +# ---------------------------------------------------------------------------- # + + showNodeVersions = pkgs: let + extracted = extractNodeVersions pkgs; + showOne = name: version: let + spaces = if ( builtins.match "bar.*" name ) == null then " " else " "; + in name + ":" + spaces + "nodejs@" + version; + strs = builtins.mapAttrs showOne extracted; + in builtins.concatStringsSep "\n" ( builtins.attrValues strs ); + + +# ---------------------------------------------------------------------------- # + + chainOverlays = builtins.foldl' ( pkgs: pkgs.extend ) pkgsFor; + composeOverlays = overlays: + pkgsFor.extend ( nixpkgs.lib.composeManyExtensions overlays ); + +# ---------------------------------------------------------------------------- # + + pkgSets = { + chained0 = chainOverlays [n20Overlay myOverlay n18Overlay friendOverlay]; + chained1 = chainOverlays [n18Overlay friendOverlay n20Overlay myOverlay]; + chained2 = chainOverlays [friendOverlay n18Overlay myOverlay n20Overlay]; + chained3 = chainOverlays [myOverlay n20Overlay friendOverlay n18Overlay]; + composed0 = composeOverlays [n20Overlay myOverlay n18Overlay friendOverlay]; + composed1 = composeOverlays [n18Overlay friendOverlay n20Overlay myOverlay]; + composed2 = composeOverlays [friendOverlay n18Overlay myOverlay n20Overlay]; + composed3 = composeOverlays [myOverlay n20Overlay friendOverlay n18Overlay]; + }; + + versions = { + nix = builtins.mapAttrs ( _: extractNodeVersions ) pkgSets; + str = builtins.mapAttrs ( _: showNodeVersions ) pkgSets; + }; + + +# ---------------------------------------------------------------------------- # + +in { + + inherit (nixpkgs) lib; + inherit pkgsFor pkgSets versions; + + overlays = { + inherit n20Overlay n18Overlay myOverlay friendOverlay; + }; + util = { + inherit chainOverlays extractNodeVersions showNodeVersions; + }; + + show = str: builtins.trace ( "\n" + str ) null; + + report = let + reportOne = name: str: '' + * ${name} + - ${builtins.replaceStrings ["\n"] ["\n - "] str} + ''; + strs = builtins.mapAttrs reportOne versions.str; + in builtins.concatStringsSep "\n" ( builtins.attrValues strs ); + +} + +# ---------------------------------------------------------------------------- # +# +# Results +# ------- +# * chained0 +# - bar0: nodejs@18.16.0 +# - bar1: nodejs@18.16.0 +# - bar2: nodejs@20.2.0 +# - bar3: nodejs@18.16.0 +# - bar4: nodejs@20.2.0 +# - bar5: nodejs@18.16.0 +# - quux0: nodejs@18.16.0 +# - quux1: nodejs@18.16.0 +# - quux2: nodejs@18.16.0 +# - quux3: nodejs@18.16.0 +# - quux4: nodejs@18.16.0 +# - quux5: nodejs@18.16.0 +# +# * chained1 +# - bar0: nodejs@20.2.0 +# - bar1: nodejs@20.2.0 +# - bar2: nodejs@20.2.0 +# - bar3: nodejs@20.2.0 +# - bar4: nodejs@20.2.0 +# - bar5: nodejs@20.2.0 +# - quux0: nodejs@20.2.0 +# - quux1: nodejs@20.2.0 +# - quux2: nodejs@18.16.0 +# - quux3: nodejs@20.2.0 +# - quux4: nodejs@18.16.0 +# - quux5: nodejs@20.2.0 +# +# * chained2 +# - bar0: nodejs@20.2.0 +# - bar1: nodejs@20.2.0 +# - bar2: nodejs@18.16.0 +# - bar3: nodejs@20.2.0 +# - bar4: nodejs@18.16.0 +# - bar5: nodejs@20.2.0 +# - quux0: nodejs@20.2.0 +# - quux1: nodejs@20.2.0 +# - quux2: nodejs@18.16.0 +# - quux3: nodejs@20.2.0 +# - quux4: nodejs@18.16.0 +# - quux5: nodejs@20.2.0 +# +# * chained3 +# - bar0: nodejs@18.16.0 +# - bar1: nodejs@18.16.0 +# - bar2: nodejs@18.16.0 +# - bar3: nodejs@18.16.0 +# - bar4: nodejs@18.16.0 +# - bar5: nodejs@18.16.0 +# - quux0: nodejs@18.16.0 +# - quux1: nodejs@18.16.0 +# - quux2: nodejs@20.2.0 +# - quux3: nodejs@18.16.0 +# - quux4: nodejs@20.2.0 +# - quux5: nodejs@18.16.0 +# +# * composed0 +# - bar0: nodejs@18.16.0 +# - bar1: nodejs@18.16.0 +# - bar2: nodejs@20.2.0 +# - bar3: nodejs@18.16.0 +# - bar4: nodejs@20.2.0 +# - bar5: nodejs@18.16.0 +# - quux0: nodejs@18.16.0 +# - quux1: nodejs@18.16.0 +# - quux2: nodejs@18.16.0 +# - quux3: nodejs@18.16.0 +# - quux4: nodejs@18.16.0 +# - quux5: nodejs@18.16.0 +# +# * composed1 +# - bar0: nodejs@20.2.0 +# - bar1: nodejs@20.2.0 +# - bar2: nodejs@20.2.0 +# - bar3: nodejs@20.2.0 +# - bar4: nodejs@20.2.0 +# - bar5: nodejs@20.2.0 +# - quux0: nodejs@20.2.0 +# - quux1: nodejs@20.2.0 +# - quux2: nodejs@18.16.0 +# - quux3: nodejs@20.2.0 +# - quux4: nodejs@18.16.0 +# - quux5: nodejs@20.2.0 +# +# * composed2 +# - bar0: nodejs@20.2.0 +# - bar1: nodejs@20.2.0 +# - bar2: nodejs@18.16.0 +# - bar3: nodejs@20.2.0 +# - bar4: nodejs@18.16.0 +# - bar5: nodejs@20.2.0 +# - quux0: nodejs@20.2.0 +# - quux1: nodejs@20.2.0 +# - quux2: nodejs@18.16.0 +# - quux3: nodejs@20.2.0 +# - quux4: nodejs@18.16.0 +# - quux5: nodejs@20.2.0 +# +# * composed3 +# - bar0: nodejs@18.16.0 +# - bar1: nodejs@18.16.0 +# - bar2: nodejs@18.16.0 +# - bar3: nodejs@18.16.0 +# - bar4: nodejs@18.16.0 +# - bar5: nodejs@18.16.0 +# - quux0: nodejs@18.16.0 +# - quux1: nodejs@18.16.0 +# - quux2: nodejs@20.2.0 +# - quux3: nodejs@18.16.0 +# - quux4: nodejs@20.2.0 +# - quux5: nodejs@18.16.0 +# +# +# ============================================================================ # From c7714b7ddc0c591482f93402d624b54b14049782 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 26 May 2023 09:36:06 -0500 Subject: [PATCH 13/17] Update use-cases/deep-replace.md Co-authored-by: Silvan Mosberger --- use-cases/deep-replace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index c97e3a1..e91079b 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -56,7 +56,7 @@ in pkgsN14.extend ( final: prev: { ## Current Problems With this approach we have three main sources of complexity, none of which -truly prevent a user from accomplishing their goal; but we might suffice to +truly prevent a user from accomplishing their goal, but we might suffice to say that it may be worthwhile to provide a more straightforward mechanism for handling this use case. From 40ebafb3b1aa97b83f9c8876bd5349e056ac1be9 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Fri, 26 May 2023 09:39:24 -0500 Subject: [PATCH 14/17] add composeOverlays to util --- scratch/nested-extend.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scratch/nested-extend.nix b/scratch/nested-extend.nix index d86d79c..85c4dd7 100644 --- a/scratch/nested-extend.nix +++ b/scratch/nested-extend.nix @@ -129,7 +129,7 @@ in { inherit n20Overlay n18Overlay myOverlay friendOverlay; }; util = { - inherit chainOverlays extractNodeVersions showNodeVersions; + inherit chainOverlays composeOverlays extractNodeVersions showNodeVersions; }; show = str: builtins.trace ( "\n" + str ) null; From 778029e38bae92a7a0fb0ee51812d54980995255 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Tue, 13 Jun 2023 10:03:29 -0500 Subject: [PATCH 15/17] improve example --- use-cases/deep-replace.md | 82 ++++++++++++++++++++++++++++++--------- 1 file changed, 64 insertions(+), 18 deletions(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index e91079b..20a45cf 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -28,29 +28,75 @@ pkgs.extend ( final: prev: { foo = final.callPackage ./my-pkgs/foo {}; } ) ### Nested Overlays -This fails to use Node.js v14 in `nodePackages`, which is unexpected to -most users. +Nested overlays and "chained" vs. "composed" overlays, as well as when `prev` +and `final` should be used is a common source of confusion. +This can be particularly problematic when consuming overlays defined externally. + +There is an extended example [here](../scratch/nested-extend.nix) that shows a +variety of common pitfalls, but the snippet below shows an abbreviated case. +Here the use of `nodejs` is contrived, what's important is that `nodejs` is +used to populate a nested scope `nodePackages`. +For the purposes of this example, we'll imagine the user is unaware of how +`nodePackages` attrsets should be used and naively use `nodePackages` +to add their own package definitions. +The core of this issue is that we attempt to merge an external overlay which +uses `nodejs@18` with an overlay using `nodejs@20`. +The user's goal here is really to use `nodejs@20` for the packages they define +and for their dependencies, but doing so inadvertently modifies executables +defined in our external overlay. + +External Flake: ```nix -# XXX: Incorrect usage -pkgs.extend ( final: prev: { - nodejs = prev.nodejs-14_x; - nodePackages = prev.nodePackages.extend ( nfinal: nprev: { - bar = nfinal.callPackage ./my-pkgs/bar {}; - } ); -} ) +{ + outputs = { nixpkgs, ... }: let + # We must split these into two overlays so `prev` holds the correct `nodejs` + overlays.node18 = final: prev: { nodejs = prev.nodejs-18_x; }; + overlays.theirPkgs = final: prev: { + nodePackages = prev.nodePackages.extend ( nfinal: nprev: { + # Requires `nodejs@18' to avoid runtime errors. + someExecutable = nfinal.callPackage ./. {}; + } ); + }; + overlays.default = + nixpkgs.lib.composeExtensions overlays.node18 overlays.theirPkgs; + in { inherit overlays; } +} ``` -To accomplish use Node.js v14 here the user would need two overlays. -```nix -let - pkgsN14 = pkgs.extend ( final: prev: { nodejs = prev.nodejs-14_x; } ); -in pkgsN14.extend ( final: prev: { - nodePackages = prev.nodePackages.extend ( nfinal: nprev: { - bar = nfinal.callPackage ./my-pkgs/bar {}; - } ); -} ) +Our Flake: +``` +{ + outputs = { nixpkgs, other, ... }: let + overlays.deps = other.overlays.default; + overlays.node20 = final: prev: { nodejs = prev.nodejs-20_x; }; + overlays.myPkgs = final: prev: { + nodePackages = prev.nodePackages.extend ( nfinal: nprev: { + # Doesn't use any node modules from previous overlay, but requires + # `nodejs@20' to avoid runtime errors. + myModule = nfinal.callPackage ./module {}; + } ); + # Uses `someExecutable' from previous overlay. + someTool = final.callPackage ./tool {}; + }; + overlays.default = nixpkgs.lib.composeManyExtensions [ + overlays.deps overlays.node20 overlays.myPkgs + ]; + in { + inherit overlays; + packages.x86_64-linux = let + pkgsFor = nixpkgs.legacyPackages.x86_64-linux.extend overlays.default; + in { inherit (pkgsFor) nodePackages someTool; } + # packages. = ...; + } +} ``` +In this example the user will encounter runtime crashes in `myTool` caused by +accidentally overriding `nodejs = nodejs-20_x;` +in `nodePackages.someExecutable` defined externally. +While we certainly have ways to avoid this category of issue, the process seen +above is already a challenge to understand - ideally a more intuitive pattern +could be offered to users. ## Current Problems From 324a322af455416180f3b149ed76c909938626cf Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Tue, 13 Jun 2023 10:03:56 -0500 Subject: [PATCH 16/17] Update use-cases/deep-replace.md Co-authored-by: Benoit de Chezelles --- use-cases/deep-replace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index 20a45cf..d421639 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -114,7 +114,7 @@ routines aren't intuitively understood by many users. discover after a lengthy debugging session. 2. Nested scopes are difficult to locate, and the relationship between - parent scopes and child scopes is not opaque to users. + parent scopes and child scopes is opaque to users. - See [Nested Overlays](#Nested-Ex) example. 3. With ad-hoc recipes and flakes there isn't standardized usage of From 4adf08f680f56d76ced2fbb00548b6e9c1ae77f9 Mon Sep 17 00:00:00 2001 From: Alex Ameen Date: Sat, 24 Jun 2023 19:27:03 -0500 Subject: [PATCH 17/17] Update use-cases/deep-replace.md Co-authored-by: Paul Haerle --- use-cases/deep-replace.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/use-cases/deep-replace.md b/use-cases/deep-replace.md index d421639..6448ccd 100644 --- a/use-cases/deep-replace.md +++ b/use-cases/deep-replace.md @@ -28,8 +28,8 @@ pkgs.extend ( final: prev: { foo = final.callPackage ./my-pkgs/foo {}; } ) ### Nested Overlays -Nested overlays and "chained" vs. "composed" overlays, as well as when `prev` -and `final` should be used is a common source of confusion. +Nested overlays, "chained" vs. "composed" overlays, as and when to use `prev` +and `final` are common sources of confusion. This can be particularly problematic when consuming overlays defined externally. There is an extended example [here](../scratch/nested-extend.nix) that shows a