symlinkJoin: ability to strip prefix from tree

[CertainLach]

Currently, symlinkJoin always joins packages at its roots. It is possible to just suffix every package with wanted path to avoid that, but this causes an error due to missing path (If there is one packages option, but multiple things that you want to extract from those packages)

We already have uses for similar functionality in tree, but it it always duplicated, e.g

• udev (also includes some checking/patching, but doesn't matter, patching would be better implemented as a build hook, and checking as symlinkJoin postBuild):

  nixpkgs/nixos/modules/services/hardware/udev.nix

  Lines 52 to 59 in 818089b

  	# Add the udev rules from other packages.
  	for i in $packages; do
  	echo "Adding rules for package $i"
  	for j in $i/{etc,lib}/udev/rules.d/*; do
  	echo "Copying $j to $out/$(basename $j)"
  	cat $j > $out/$(basename $j)
  	done
  	done

• systemd: https://github.com/NixOS/nixpkgs/blob/nixos-24.05/nixos/modules/system/boot/systemd.nix#L507-L521
• tmpfiles (it is suffixing packages, and it is enough in this case. Still, stripPrefix might look better): https://github.com/NixOS/nixpkgs/blob/nixos-24.05/nixos/modules/system/boot/systemd/tmpfiles.nix#L189-L202

The patch is a little bit ugly to avoid causing mass-rebuilds, it would be nice to make it prettier during one.

Required for Qubes: #341215

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[philiptaron]

Thanks for your contribution, Yaroslav (@CertainLach).

I'm requesting changes for three things.

1. Documentation of the new behavior.
2. Tests of the facility.
3. Good errors from the new assert.

Of these, I care the most about the tests. 🙂

I appreciate the research you did to motivate adding this new functionality to symlinkJoin.

[philiptaron]

Now that we're adding in something more than just a simple for loop, I think this needs a test.

See the file pkgs/build-support/trivial-builders/test/default.nix for the set of tests.

Run the tests using nix-build -A tests.trivial-builders.symlinkJoin.

Take a look at pkgs/test/replace-vars/default.nix for an example of how to write those tests. testers.testEqualContents in particular will be useful, and testBuildFailure for demonstrating what happens when the derivation ought to error.

That'll give you the ability to do demonstrate the following cases:

1. Direct, normal use sans any path stripping (maybe with a nested list, per the recursive call).
2. Build failure due to a missing path.
3. Path stripping with missing paths
4. Path stripping with failures due to other reasons that are skipped.

[CertainLach]

Path stripping with failures due to other reasons that are skipped.

Error skipping behavior was unintentional, it was added to make command not error, i.e test -d nonexisting && lndir ... would fail because of test -d exit code, and test -d ... && lndir ... || true would fix that error code.
Changed to if test -d ...; then instead. This way the code is even more ugly, but more correct.

It will skip those paths, whose prefix references some file (This test was added too), but should not ignore other I/O errors.