Skip to content

Add overlays mechanism to Nixpkgs. #21243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nbp merged 13 commits into from
Jan 16, 2017
Merged

Add overlays mechanism to Nixpkgs. #21243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b28d698
Add overlays mechanism to Nixpkgs.
nbp Dec 17, 2016
3490df6
Add NixOS option 'nixpkgs.overlays' to set the argument of Nixpkgs.
nbp Dec 25, 2016
4e3c511
Add missing line break in the release notes.
nbp Dec 26, 2016
0d5a8e4
Throw an error if NIXPKGS_OVERLAYS is invalid and improve documentation.
nbp Dec 26, 2016
4eb8d9c
Replace 'phases' by 'layers' in overlays documentation.
nbp Dec 26, 2016
9cedda5
Improve the realse notes with the upcoming documentation links, and a…
nbp Jan 14, 2017
4205b8a
Update overlay documentation by following nits from aneeshusa.
nbp Jan 15, 2017
7fda136
Fix typo in nixpkgs.nix module.
nbp Jan 15, 2017
5b9131c
Fix missing overlays argument in stdenv/linux/default.nix
nbp Jan 15, 2017
120a603
Fix extra nits from aneeshusa
nbp Jan 15, 2017
17efce4
Fix typo, lib.listOf --> types.listOf
nbp Jan 15, 2017
8798264
Fix nixpkgs manual generation, missing para closing tag.
nbp Jan 15, 2017
97e75f2
Fix release-notes compilation.
nbp Jan 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 2 additions & 62 deletions doc/functions.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,66 +17,6 @@
derivations or even the whole package set.
</para>

<section xml:id="sec-pkgs-overridePackages">
<title>pkgs.overridePackages</title>

<para>
This function inside the nixpkgs expression (<varname>pkgs</varname>)
can be used to override the set of packages itself.
</para>
<para>
Warning: this function is expensive and must not be used from within
the nixpkgs repository.
</para>
<para>
Example usage:

<programlisting>let
pkgs = import &lt;nixpkgs&gt; {};
newpkgs = pkgs.overridePackages (self: super: {
foo = super.foo.override { ... };
});
in ...</programlisting>
</para>

<para>
The resulting <varname>newpkgs</varname> will have the new <varname>foo</varname>
expression, and all other expressions depending on <varname>foo</varname> will also
use the new <varname>foo</varname> expression.
</para>

<para>
The behavior of this function is similar to <link
linkend="sec-modify-via-packageOverrides">config.packageOverrides</link>.
</para>

<para>
The <varname>self</varname> parameter refers to the final package set with the
applied overrides. Using this parameter may lead to infinite recursion if not
used consciously.
</para>

<para>
The <varname>super</varname> parameter refers to the old package set.
It's equivalent to <varname>pkgs</varname> in the above example.
</para>

<para>
Note that in previous versions of nixpkgs, this method replaced any changes from <link
linkend="sec-modify-via-packageOverrides">config.packageOverrides</link>,
along with that from previous calls if this function was called repeatedly.
Now those previous changes will be preserved so this function can be "chained" meaningfully.
To recover the old behavior, make sure <varname>config.packageOverrides</varname> is unset,
and call this only once off a "freshly" imported nixpkgs:

<programlisting>let
pkgs = import &lt;nixpkgs&gt; { config: {}; };
newpkgs = pkgs.overridePackages ...;
in ...</programlisting>
</para>

</section>

<section xml:id="sec-pkg-override">
<title>&lt;pkg&gt;.override</title>

Expand All @@ -91,9 +31,9 @@
Example usages:

<programlisting>pkgs.foo.override { arg1 = val1; arg2 = val2; ... }</programlisting>
<programlisting>pkgs.overridePackages (self: super: {
<programlisting>import pkgs.path { overlays = [ (self: super: {
foo = super.foo.override { barSupport = true ; };
})</programlisting>
})]};</programlisting>
<programlisting>mypkg = pkgs.callPackage ./mypkg.nix {
mydep = pkgs.mydep.override { ... };
}</programlisting>
Expand Down
6 changes: 3 additions & 3 deletions doc/languages-frameworks/python.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -737,18 +737,18 @@ in (pkgs.python35.override {inherit packageOverrides;}).withPackages (ps: [ps.bl
```
The requested package `blaze` depends on `pandas` which itself depends on `scipy`.

If you want the whole of Nixpkgs to use your modifications, then you can use `pkgs.overridePackages`
If you want the whole of Nixpkgs to use your modifications, then you can use `overlays`
as explained in this manual. In the following example we build a `inkscape` using a different version of `numpy`.
```
let
pkgs = import <nixpkgs> {};
newpkgs = pkgs.overridePackages ( pkgsself: pkgssuper: {
newpkgs = import pkgs.path { overlays = [ (pkgsself: pkgssuper: {
python27 = let
packageOverrides = self: super: {
numpy = super.numpy_1_10;
};
in pkgssuper.python27.override {inherit packageOverrides;};
} );
} ) ]; };
in newpkgs.inkscape
```

Expand Down
1 change: 1 addition & 0 deletions doc/manual.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
<xi:include href="meta.xml" />
<xi:include href="languages-frameworks/index.xml" />
<xi:include href="package-notes.xml" />
<xi:include href="overlays.xml" />
<xi:include href="coding-conventions.xml" />
<xi:include href="submitting-changes.xml" />
<xi:include href="reviewing-contributions.xml" />
Expand Down
99 changes: 99 additions & 0 deletions doc/overlays.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xml:id="chap-overlays">

<title>Overlays</title>

<para>This chapter describes how to extend and change Nixpkgs packages using
overlays. Overlays are used to add layers in the fix-point used by Nixpkgs
to compose the set of all packages.</para>

<!--============================================================-->

<section xml:id="sec-overlays-install">
<title>Installing Overlays</title>

<para>The set of overlays is looked for in the following places. The
first one present is considered, and all the rest are ignored:

<orderedlist>

<listitem>

<para>As an argument of the imported attribute set. When importing Nixpkgs,
the <varname>overlays</varname> attribute argument can be set to a list of
functions, which is described in <xref linkend="sec-overlays-layout"/>.</para>

</listitem>

<listitem>

<para>In the directory pointed by the environment variable
<varname>NIXPKGS_OVERLAYS</varname>.</para>
</listitem>

<listitem>

<para>In the directory <filename>~/.nixpkgs/overlays/</filename>.</para>
</listitem>

</orderedlist>
</para>

<para>For the second and third options, the directory should contain Nix expressions defining the
overlays. Each overlay can be a file, a directory containing a
<filename>default.nix</filename>, or a symlink to one of those. The expressions should follow
the syntax described in <xref linkend="sec-overlays-layout"/>.</para>

<para>The order of the overlay layers can influence the recipe of packages if multiple layers override
the same recipe. In the case where overlays are loaded from a directory, they are loaded in
alphabetical order.</para>

<para>To install an overlay using the last option, you can clone the overlay's repository and add
a symbolic link to it in <filename>~/.nixpkgs/overlays/</filename> directory.</para>

</section>

<!--============================================================-->

<section xml:id="sec-overlays-layout">
<title>Overlays Layout</title>

<para>Overlays are expressed as Nix functions which accept 2 arguments and return a set of
packages.</para>

<programlisting>
self: super:

{
boost = super.boost.override {
python = self.python3;
};
rr = super.callPackage ./pkgs/rr {
Copy link
Contributor

@aneeshusa aneeshusa Jan 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this super.callPackage? I would think super.callPackage would use packages from super for dependencies, where the documentation says to use packages from self.

Copy link
Contributor

@aneeshusa aneeshusa Jan 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E.g. I would think we need to provide a custom callPackage for each self layer using
lib.callPackagesWith.

Copy link
Member Author

@nbp nbp Jan 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically you could use both, but this would get some extra meaning in the futur (security-update branch), because self.callPackage goes twice through the self attribute evaluation. The first one to resolve the callPackage function, and the second time to resolve the dependencies from self which are captured by the callPackage function.

Note that self is the same provided to all layers. So using self.callPackage is for the moment identical to super.callPackage, except for this extra hop through the fix-point.

I documented it as functions should be taken from super, because they already capture self if they have to.

Also note that this way, we make it syntactically verifiable, as anything which comes around the dependencies uses super, and anything which is use to find dependencies uses self.

Copy link
Contributor

@aneeshusa aneeshusa Jan 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds reasonable, but can you explain how functions already capture self if they need to? E.g. if I have an overlay that adds two new packages:

self: super: {
    # Let's say all deps are already in super,
    # so I can see this working with super.callPackages or self.callPackages
    foo = super.callPackage {};
    # But this depends on foo, which isn't in super
    bar = super.callPackage {}; 
}

How does the second super.callPackage get a reference to foo to provide to bar?

Copy link
Contributor

@aneeshusa aneeshusa Jan 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I tried it out and looked at the implementation and I now understand why this works, albeit the current behavior is slightly surprising to me.

Copy link
Member Author

@nbp nbp Jan 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

self: super:

{
  callWithSelf = f: f self;
}

Any layer defined after could then do:

self: super:

{
  mySelfClone = super.callWithSelf (x: x);
}

stdenv = self.stdenv_32bit;
};
}
</programlisting>

<para>The first argument, usually named <varname>self</varname>, corresponds to the final package
set. You should use this set for the dependencies of all packages specified in your
overlay. For example, all the dependencies of <varname>rr</varname> in the example above come
from <varname>self</varname>, as well as the overriden dependencies used in the
<varname>boost</varname> override.</para>

<para>The second argument, usually named <varname>super</varname>,
corresponds to the result of the evaluation of the previous stages of
Nixpkgs. It does not contain any of the packages added by the current
overlay nor any of the following overlays. This set should be used either
to refer to packages you wish to override, or to access functions defined
in Nixpkgs. For example, the original recipe of <varname>boost</varname>
in the above example, comes from <varname>super</varname>, as well as the
<varname>callPackage</varname> function.</para>

<para>The value returned by this function should be a set similar to
<filename>pkgs/top-level/all-packages.nix</filename>, which contains
overridden and/or new packages.</para>

</section>

</chapter>
30 changes: 29 additions & 1 deletion nixos/doc/manual/release-notes/rl-1703.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ has the following highlights: </para>

<itemizedlist>
<listitem>
<para></para>
<para>Nixpkgs is now extensible through overlays. See the <link
xlink:href="https://nixos.org/nixpkgs/manual/#sec-overlays-install">Nixpkgs
manual</link> for more information.</para>
</listitem>
</itemizedlist>

Expand Down Expand Up @@ -96,6 +98,32 @@ following incompatible changes:</para>
<literal>networking.timeServers</literal>.
</para>
</listitem>

<listitem>

<para><literal>overridePackages</literal> function no longer exists.
It is replaced by <link
xlink:href="https://nixos.org/nixpkgs/manual/#sec-overlays-install">
overlays</link>. For example, the following code:

<programlisting>
let
pkgs = import &lt;nixpkgs&gt; {};
in
pkgs.overridePackages (self: super: ...)
</programlisting>

should be replaced by:

<programlisting>
let
pkgs = import &lt;nixpkgs&gt; {}; in
Copy link
Contributor

@teh teh Jan 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noticed a double in (see next line) - @nbp if you have direct push permissions maybe a quick fix? Otherwise I'll try to mop it up with some other change.

Copy link
Member Author

@nbp nbp Jan 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fixed as part of commit 0214d94. Thanks!

in
import pkgs.path { overlays = [(self: super: ...)] }
</programlisting>

</para>
</listitem>
</itemizedlist>


Expand Down
48 changes: 35 additions & 13 deletions nixos/modules/misc/nixpkgs.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,19 @@ let
};

configType = mkOptionType {
name = "nixpkgs config";
name = "nixpkgs-config";
description = "nixpkgs config";
check = traceValIfNot isConfig;
merge = args: fold (def: mergeConfig def.value) {};
};

overlayType = mkOptionType {
name = "nixpkgs-overlay";
description = "nixpkgs overlay";
check = builtins.isFunction;
merge = lib.mergeOneOption;
};

in

{
Expand All @@ -43,23 +51,37 @@ in
default = {};
example = literalExample
''
{ firefox.enableGeckoMediaPlayer = true;
packageOverrides = pkgs: {
firefox60Pkgs = pkgs.firefox60Pkgs.override {
enableOfficialBranding = true;
};
};
}
{ firefox.enableGeckoMediaPlayer = true; }
'';
type = configType;
description = ''
The configuration of the Nix Packages collection. (For
details, see the Nixpkgs documentation.) It allows you to set
package configuration options, and to override packages
globally through the <varname>packageOverrides</varname>
option. The latter is a function that takes as an argument
the <emphasis>original</emphasis> Nixpkgs, and must evaluate
to a set of new or overridden packages.
package configuration options.
'';
};

nixpkgs.overlays = mkOption {
default = [];
example = literalExample
''
[ (self: super: {
openssh = super.openssh.override {
hpnSupport = true;
withKerberos = true;
kerberos = self.libkrb5;
};
};
) ]
'';
type = types.listOf overlayType;
description = ''
List of overlays to use with the Nix Packages collection.
(For details, see the Nixpkgs documentation.) It allows
you to override packages globally. This is a function that
takes as an argument the <emphasis>original</emphasis> Nixpkgs.
The first argument should be used for finding dependencies, and
the second should be used for overriding recipes.
'';
};

Expand Down
8 changes: 4 additions & 4 deletions pkgs/stdenv/cross/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{ lib
, system, platform, crossSystem, config
, system, platform, crossSystem, config, overlays
}:

let
bootStages = import ../. {
inherit lib system platform;
inherit lib system platform overlays;
crossSystem = null;
# Ignore custom stdenvs when cross compiling for compatability
config = builtins.removeAttrs config [ "replaceStdenv" ];
Expand All @@ -18,7 +18,7 @@ in bootStages ++ [
# should be used to build compilers and other such tools targeting the cross
# platform. Then, `forceNativeDrv` can be removed.
(vanillaPackages: {
inherit system platform crossSystem config;
inherit system platform crossSystem config overlays;
# It's OK to change the built-time dependencies
allowCustomOverrides = true;
stdenv = vanillaPackages.stdenv // {
Expand All @@ -30,7 +30,7 @@ in bootStages ++ [

# Run packages
(buildPackages: {
inherit system platform crossSystem config;
inherit system platform crossSystem config overlays;
stdenv = if crossSystem.useiOSCross or false
then let
inherit (buildPackages.darwin.ios-cross {
Expand Down
6 changes: 3 additions & 3 deletions pkgs/stdenv/custom/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
{ lib
, system, platform, crossSystem, config
, system, platform, crossSystem, config, overlays
}:

assert crossSystem == null;

let
bootStages = import ../. {
inherit lib system platform crossSystem;
inherit lib system platform crossSystem overlays;
# Remove config.replaceStdenv to ensure termination.
config = builtins.removeAttrs config [ "replaceStdenv" ];
};
Expand All @@ -15,7 +15,7 @@ in bootStages ++ [

# Additional stage, built using custom stdenv
(vanillaPackages: {
inherit system platform crossSystem config;
inherit system platform crossSystem config overlays;
stdenv = config.replaceStdenv { pkgs = vanillaPackages; };
})

Expand Down
Loading