Skip to content

Commit fffb54d

Browse files
authored
Merge pull request #12594 from NixOS/mergify/bp/2.27-maintenance/pr-12442
Expand manual on derivation outputs (backport #12442)
2 parents 7e61e96 + 8b9d401 commit fffb54d

File tree

14 files changed

+552
-190
lines changed

14 files changed

+552
-190
lines changed

.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
/tests/functional/lang/*.err
1515
/tests/functional/lang/*.ast
1616

17-
outputs/
17+
/outputs
1818

1919
*~
2020

doc/manual/source/SUMMARY.md.in

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,10 @@
2222
- [Store Object](store/store-object.md)
2323
- [Content-Addressing Store Objects](store/store-object/content-address.md)
2424
- [Store Path](store/store-path.md)
25-
- [Store Derivation and Deriving Path](store/drv.md)
25+
- [Store Derivation and Deriving Path](store/derivation/index.md)
26+
- [Derivation Outputs and Types of Derivations](store/derivation/outputs/index.md)
27+
- [Content-addressing derivation outputs](store/derivation/outputs/content-address.md)
28+
- [Input-addressing derivation outputs](store/derivation/outputs/input-address.md)
2629
- [Building](store/building.md)
2730
- [Store Types](store/types/index.md)
2831
{{#include ./store/types/SUMMARY.md}}

doc/manual/source/advanced-topics/distributed-builds.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ For a local machine to forward a build to a remote machine, the remote machine m
2020

2121
## Testing
2222

23-
To test connecting to a remote Nix instance (in this case `mac`), run:
23+
To test connecting to a remote [Nix instance] (in this case `mac`), run:
2424

2525
```console
2626
nix store info --store ssh://username@mac
@@ -106,3 +106,5 @@ file included in `builders` via the syntax `@/path/to/file`. For example,
106106

107107
causes the list of machines in `/etc/nix/machines` to be included.
108108
(This is the default.)
109+
110+
[Nix instance]: @docroot@/glossary.md#gloss-nix-instance

doc/manual/source/glossary.md

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
# Glossary
22

3+
- [build system]{#gloss-build-system}
4+
5+
Generic term for software that facilitates the building of software by automating the invocation of compilers, linkers, and other tools.
6+
7+
Nix can be used as a generic build system.
8+
It has no knowledge of any particular programming language or toolchain.
9+
These details are specified in [derivation expressions](#gloss-derivation-expression).
10+
311
- [content address]{#gloss-content-address}
412

513
A
@@ -19,18 +27,22 @@
1927

2028
Besides content addressing, the Nix store also uses [input addressing](#gloss-input-addressed-store-object).
2129

30+
- [content-addressed storage]{#gloss-content-addressed-store}
31+
32+
The industry term for storage and retrieval systems using [content addressing](#gloss-content-address). A Nix store also has [input addressing](#gloss-input-addressed-store-object), and metadata.
33+
2234
- [store derivation]{#gloss-store-derivation}
2335

2436
A single build task.
25-
See [Store Derivation](@docroot@/store/drv.md#store-derivation) for details.
37+
See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details.
2638

2739
[store derivation]: #gloss-store-derivation
2840

2941
- [derivation path]{#gloss-derivation-path}
3042

3143
A [store path] which uniquely identifies a [store derivation].
3244

33-
See [Referencing Store Derivations](@docroot@/store/drv.md#derivation-path) for details.
45+
See [Referencing Store Derivations](@docroot@/store/derivation/index.md#derivation-path) for details.
3446

3547
Not to be confused with [deriving path].
3648

@@ -88,6 +100,12 @@
88100

89101
[store]: #gloss-store
90102

103+
- [Nix instance]{#gloss-nix-instance}
104+
<!-- ambiguous -->
105+
1. An installation of Nix, which includes the presence of a [store], and the Nix package manager which operates on that store.
106+
A local Nix installation and a [remote builder](@docroot@/advanced-topics/distributed-builds.md) are two examples of Nix instances.
107+
2. A running Nix process, such as the `nix` command.
108+
91109
- [binary cache]{#gloss-binary-cache}
92110

93111
A *binary cache* is a Nix store which uses a different format: its
@@ -220,7 +238,7 @@
220238
directly or indirectly “reachable” from that store path; that is,
221239
it’s the closure of the path under the *references* relation. For
222240
a package, the closure of its derivation is equivalent to the
223-
build-time dependencies, while the closure of its output path is
241+
build-time dependencies, while the closure of its [output path] is
224242
equivalent to its runtime dependencies. For correct deployment it
225243
is necessary to deploy whole closures, since otherwise at runtime
226244
files could be missing. The command `nix-store --query --requisites ` prints out
@@ -252,7 +270,7 @@
252270

253271
Deriving paths are a way to refer to [store objects][store object] that might not yet be [realised][realise].
254272

255-
See [Deriving Path](./store/drv.md#deriving-path) for details.
273+
See [Deriving Path](./store/derivation/index.md#deriving-path) for details.
256274

257275
Not to be confused with [derivation path].
258276

doc/manual/source/language/advanced-attributes.md

Lines changed: 131 additions & 132 deletions
Original file line numberDiff line numberDiff line change
@@ -99,8 +99,8 @@ Derivations can declare some infrequently used optional attributes.
9999
to make it use the proxy server configuration specified by the user
100100
in the environment variables `http_proxy` and friends.
101101
102-
This attribute is only allowed in *fixed-output derivations* (see
103-
below), where impurities such as these are okay since (the hash
102+
This attribute is only allowed in [fixed-output derivations][fixed-output derivation],
103+
where impurities such as these are okay since (the hash
104104
of) the output is known in advance. It is ignored for all other
105105
derivations.
106106
@@ -119,135 +119,6 @@ Derivations can declare some infrequently used optional attributes.
119119
[`impure-env`](@docroot@/command-ref/conf-file.md#conf-impure-env)
120120
configuration setting.
121121
122-
- [`outputHash`]{#adv-attr-outputHash}; [`outputHashAlgo`]{#adv-attr-outputHashAlgo}; [`outputHashMode`]{#adv-attr-outputHashMode}\
123-
These attributes declare that the derivation is a so-called *fixed-output derivation* (FOD), which means that a cryptographic hash of the output is already known in advance.
124-
125-
As opposed to regular derivations, the [`builder`] executable of a fixed-output derivation has access to the network.
126-
Nix computes a cryptographic hash of its output and compares that to the hash declared with these attributes.
127-
If there is a mismatch, the derivation fails.
128-
129-
The rationale for fixed-output derivations is derivations such as
130-
those produced by the `fetchurl` function. This function downloads a
131-
file from a given URL. To ensure that the downloaded file has not
132-
been modified, the caller must also specify a cryptographic hash of
133-
the file. For example,
134-
135-
```nix
136-
fetchurl {
137-
url = "http://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz";
138-
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
139-
}
140-
```
141-
142-
It sometimes happens that the URL of the file changes, e.g., because
143-
servers are reorganised or no longer available. We then must update
144-
the call to `fetchurl`, e.g.,
145-
146-
```nix
147-
fetchurl {
148-
url = "ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz";
149-
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
150-
}
151-
```
152-
153-
If a `fetchurl` derivation was treated like a normal derivation, the
154-
output paths of the derivation and *all derivations depending on it*
155-
would change. For instance, if we were to change the URL of the
156-
Glibc source distribution in Nixpkgs (a package on which almost all
157-
other packages depend) massive rebuilds would be needed. This is
158-
unfortunate for a change which we know cannot have a real effect as
159-
it propagates upwards through the dependency graph.
160-
161-
For fixed-output derivations, on the other hand, the name of the
162-
output path only depends on the `outputHash*` and `name` attributes,
163-
while all other attributes are ignored for the purpose of computing
164-
the output path. (The `name` attribute is included because it is
165-
part of the path.)
166-
167-
As an example, here is the (simplified) Nix expression for
168-
`fetchurl`:
169-
170-
```nix
171-
{ stdenv, curl }: # The curl program is used for downloading.
172-
173-
{ url, sha256 }:
174-
175-
stdenv.mkDerivation {
176-
name = baseNameOf (toString url);
177-
builder = ./builder.sh;
178-
buildInputs = [ curl ];
179-
180-
# This is a fixed-output derivation; the output must be a regular
181-
# file with SHA256 hash sha256.
182-
outputHashMode = "flat";
183-
outputHashAlgo = "sha256";
184-
outputHash = sha256;
185-
186-
inherit url;
187-
}
188-
```
189-
190-
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
191-
The "nix32" encoding is an adaptation of base-32 encoding.
192-
The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings, and the [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
193-
194-
The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
195-
It can currently be `"blake3", "sha1"`, `"sha256"`, `"sha512"`, or `null`.
196-
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format.
197-
198-
The `outputHashMode` attribute determines how the hash is computed.
199-
It must be one of the following values:
200-
201-
- [`"flat"`](@docroot@/store/store-object/content-address.md#method-flat)
202-
203-
This is the default.
204-
205-
- [`"recursive"` or `"nar"`](@docroot@/store/store-object/content-address.md#method-nix-archive)
206-
207-
> **Compatibility**
208-
>
209-
> `"recursive"` is the traditional way of indicating this,
210-
> and is supported since 2005 (virtually the entire history of Nix).
211-
> `"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
212-
> however support for it is only added in Nix version 2.21.
213-
214-
- [`"text"`](@docroot@/store/store-object/content-address.md#method-text)
215-
216-
> **Warning**
217-
>
218-
> The use of this method for derivation outputs is part of the [`dynamic-derivations`][xp-feature-dynamic-derivations] experimental feature.
219-
220-
- [`"git"`](@docroot@/store/store-object/content-address.md#method-git)
221-
222-
> **Warning**
223-
>
224-
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.
225-
226-
- [`__contentAddressed`]{#adv-attr-__contentAddressed}
227-
228-
> **Warning**
229-
> This attribute is part of an [experimental feature](@docroot@/development/experimental-features.md).
230-
>
231-
> To use this attribute, you must enable the
232-
> [`ca-derivations`][xp-feature-ca-derivations] experimental feature.
233-
> For example, in [nix.conf](../command-ref/conf-file.md) you could add:
234-
>
235-
> ```
236-
> extra-experimental-features = ca-derivations
237-
> ```
238-
239-
If this attribute is set to `true`, then the derivation
240-
outputs will be stored in a content-addressed location rather than the
241-
traditional input-addressed one.
242-
243-
Setting this attribute also requires setting
244-
[`outputHashMode`](#adv-attr-outputHashMode)
245-
and
246-
[`outputHashAlgo`](#adv-attr-outputHashAlgo)
247-
like for *fixed-output derivations* (see above).
248-
249-
It also implicitly requires that the machine to build the derivation must have the `ca-derivations` [system feature](@docroot@/command-ref/conf-file.md#conf-system-features).
250-
251122
- [`passAsFile`]{#adv-attr-passAsFile}\
252123
A list of names of attributes that should be passed via files rather
253124
than environment variables. For example, if you have
@@ -370,6 +241,134 @@ Derivations can declare some infrequently used optional attributes.
370241

371242
ensures that the derivation can only be built on a machine with the `kvm` feature.
372243

373-
[xp-feature-ca-derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
244+
## Setting the derivation type
245+
246+
As discussed in [Derivation Outputs and Types of Derivations](@docroot@/store/derivation/outputs/index.md), there are multiples kinds of derivations / kinds of derivation outputs.
247+
The choice of the following attributes determines which kind of derivation we are making.
248+
249+
- [`__contentAddressed`]
250+
251+
- [`outputHash`]
252+
253+
- [`outputHashAlgo`]
254+
255+
- [`outputHashMode`]
256+
257+
The three types of derivations are chosen based on the following combinations of these attributes.
258+
All other combinations are invalid.
259+
260+
- [Input-addressing derivations](@docroot@/store/derivation/outputs/input-address.md)
261+
262+
This is the default for `builtins.derivation`.
263+
Nix only currently supports one kind of input-addressing, so no other information is needed.
264+
265+
`__contentAddressed = false;` may also be included, but is not needed, and will trigger the experimental feature check.
266+
267+
- [Fixed-output derivations][fixed-output derivation]
268+
269+
All of [`outputHash`], [`outputHashAlgo`], and [`outputHashMode`].
270+
271+
<!--
272+
273+
`__contentAddressed` is ignored, becaused fixed-output derivations always content-address their outputs, by definition.
274+
275+
**TODO CHECK**
276+
277+
-->
278+
279+
- [(Floating) content-addressing derivations](@docroot@/store/derivation/outputs/content-address.md)
280+
281+
Both [`outputHashAlgo`] and [`outputHashMode`], `__contentAddressed = true;`, and *not* `outputHash`.
282+
283+
If an output hash was given, then the derivation output would be "fixed" not "floating".
284+
285+
Here is more information on the `output*` attributes, and what values they may be set to:
286+
287+
- [`outputHashMode`]{#adv-attr-outputHashMode}
288+
289+
This specifies how the files of a content-addressing derivation output are digested to produce a content address.
290+
291+
This works in conjunction with [`outputHashAlgo`](#adv-attr-outputHashAlgo).
292+
Specifying one without the other is an error (unless [`outputHash` is also specified and includes its own hash algorithm as described below).
293+
294+
The `outputHashMode` attribute determines how the hash is computed.
295+
It must be one of the following values:
296+
297+
- [`"flat"`](@docroot@/store/store-object/content-address.md#method-flat)
298+
299+
This is the default.
300+
301+
- [`"recursive"` or `"nar"`](@docroot@/store/store-object/content-address.md#method-nix-archive)
302+
303+
> **Compatibility**
304+
>
305+
> `"recursive"` is the traditional way of indicating this,
306+
> and is supported since 2005 (virtually the entire history of Nix).
307+
> `"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
308+
> however support for it is only added in Nix version 2.21.
309+
310+
- [`"text"`](@docroot@/store/store-object/content-address.md#method-text)
311+
312+
> **Warning**
313+
>
314+
> The use of this method for derivation outputs is part of the [`dynamic-derivations`][xp-feature-dynamic-derivations] experimental feature.
315+
316+
- [`"git"`](@docroot@/store/store-object/content-address.md#method-git)
317+
318+
> **Warning**
319+
>
320+
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.
321+
322+
See [content-addressing store objects](@docroot@/store/store-object/content-address.md) for more information about the process this flag controls.
323+
324+
- [`outputHashAlgo`]{#adv-attr-outputHashAlgo}
325+
326+
This specifies the hash alorithm used to digest the [file system object] data of a content-addressing derivation output.
327+
328+
This works in conjunction with [`outputHashMode`](#adv-attr-outputHashAlgo).
329+
Specifying one without the other is an error (unless [`outputHash` is also specified and includes its own hash algorithm as described below).
330+
331+
The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
332+
It can currently be `"blake3"`, "sha1"`, `"sha256"`, `"sha512"`, or `null`.
333+
334+
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format, because in that case the choice of hash algorithm is determined by `outputHash`.
335+
336+
- [`outputHash`]{#adv-attr-outputHashAlgo}; [`outputHash`]{#adv-attr-outputHashMode}\
337+
338+
This will specify the output hash of the single output of a [fixed-output derivation].
339+
340+
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
341+
The "nix32" encoding is an adaptation of base-32 encoding.
342+
343+
> **Note**
344+
>
345+
> The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings.
346+
> The [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
347+
348+
- [`__contentAddressed`]{#adv-attr-__contentAddressed}
349+
350+
> **Warning**
351+
>
352+
> This attribute is part of an [experimental feature](@docroot@/development/experimental-features.md).
353+
>
354+
> To use this attribute, you must enable the
355+
> [`ca-derivations`][xp-feature-ca-derivations] experimental feature.
356+
> For example, in [nix.conf](../command-ref/conf-file.md) you could add:
357+
>
358+
> ```
359+
> extra-experimental-features = ca-derivations
360+
> ```
361+
362+
This is a boolean with a default of `false`.
363+
It determines whether the derivation is floating content-addressing.
364+
365+
[`__contentAddressed`]: #adv-attr-__contentAddressed
366+
[`outputHash`]: #adv-attr-outputHash
367+
[`outputHashAlgo`]: #adv-attr-outputHashAlgo
368+
[`outputHashMode`]: #adv-attr-outputHashMode
369+
370+
[fixed-output derivation]: @docroot@/glossary.md#gloss-fixed-output-derivation
371+
[file system object]: @docroot@/store/file-system-object.md
372+
[store object]: @docroot@/store/store-object.md
374373
[xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations
375374
[xp-feature-git-hashing]: @docroot@/development/experimental-features.md#xp-feature-git-hashing

0 commit comments

Comments
 (0)