lib: Add splicing utilities

[roberth]

This PR aims to achieve two things:

1. Make the utilities from First class cross nix#13073 available
2. Improve our terminology.

I'm really not sure about (2). "Splice" is a nice name. It has got a bad reputation due to the "tacking on" aspect, but I don't know if we should let that taint the concept of splicing. Alternatively, we could call that part of splicing "carousel" - which as a term also covers the pattern we have for multiple outputs. To define that briefly, it's the pattern where

• Users of the carousel object can treat it as a normal instance of the object
• Variations of the object are made available as attributes attached to the otherwise normal instance of the object
• Each of the variations is also a carousel object
• The variations are memoized

@Ericson2314 wdyt? Should we add a new term to underpin splicing, or add a term for the part people don't like, or both? (cross index + carousel = splicing)

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and others READMEs.

---

Add a 👍 reaction to pull requests you find important.

[philiptaron]

Force pushed to go beyond the formatting errors. I hope to take a look at the performance report.

EDIT: it looks like it is net neutral (essentially) @roberth.

[Ericson2314]

I like "cross index"! :)

[Ericson2314]

@roberth can this be used to simplify Nix's packaging too?

[emilazy]

Force pushed to go beyond the formatting errors. I hope to take a look at the performance report.

EDIT: it looks like it is net neutral (essentially) @roberth.

This would need to be tested on a cross package set to assess eval impact, as all the overhead would be there rather than in a native one.

[roberth]

@emilazy Good news

Numbers

With nix shell nixpkgs#bench

bench -- 'nix-instantiate --argstr crossSystem aarch64-linux --argstr system x86_64-linux -A nix -A nixfmt -A python3Packages.transformers -A pandoc'

Before:

benchmarking nix-instantiate --argstr crossSystem aarch64-linux --argstr system x86_64-linux -A nix -A nixfmt -A python3Packages.transformers -A pandoc
time                 1.931 s    (1.903 s .. 1.948 s)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.923 s    (1.908 s .. 1.929 s)
std dev              10.31 ms   (1.880 ms .. 13.41 ms)
variance introduced by outliers: 19% (moderately inflated)

After:

benchmarking nix-instantiate --argstr crossSystem aarch64-linux --argstr system x86_64-linux -A nix -A nixfmt -A python3Packages.transformers -A pandoc
time                 1.897 s    (1.869 s .. 1.948 s)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.939 s    (1.918 s .. 1.957 s)
std dev              21.37 ms   (12.47 ms .. 26.31 ms)
variance introduced by outliers: 19% (moderately inflated)

So those are too noisy, as usual.

NIX_SHOW_STATS relevant fields:

gc.totalBytes: 709977680 -> 710303424 (+0.045%)
values.number:  8841021 -> 8838549 (-)
envs.bytes: 70168192 -> 70466376 (+)

So that's a movement from value allocations to closure allocations, but the total allocations are up insignificantly.

The insignificant overhead (on the order of 0.05%) could easily be counteracted by future cleanups that remove the conversions and make things work with the single representation throughout.

[roberth]

@Ericson2314

@roberth can this be used to simplify Nix's packaging too?

These functions were adapted from NixOS/nix#13073, so we can switch to these implementations instead. I'd recommend to do so in January, which is when we can forget about 25.05.

[Ericson2314]

@roberth oh does the in-Nixpkgs packaging not need them?