Make fetch* source derivation names (optionally) more descriptive and homogenize them across fetchers

[oxij]

Description of changes

This patch adds lib.repoRevToName function that generalizes away most of the code used for derivation name generation by fetch* (fetchzip, fetchFromGitHub, etc) functions.

It's first argument controls how the resulting name will look (see below).

Since lib has no equivalent of Nixpkgs' config, this patch adds config.fetchedSourceNameDefault option to Nixpkgs and then re-exposes lib.repoRevToName config.fetchedSourceNameDefault expression as pkgs.repoRevToNameMaybe which is then used in fetch* derivations.

The result is that different values of config.fetchedSourceNameDefault now control how the src derivations produced by fetch* functions are to be named, e.g.:

• fetchedSourceNameDefault = "source" (the default):

  $ nix-instantiate -A fuse.src
  /nix/store/<hash>-source.drv
  

• fetchedSourceNameDefault = "versioned":

  $ nix-instantiate -A fuse.src
  /nix/store/<hash>-libfuse-2.9.9-source.drv
  

• fetchedSourceNameDefault = "full":

  $ nix-instantiate -A fuse.src
  /nix/store/<hash>-libfuse-2.9.9-github-source.drv
  

See documentation of config.fetchedSourceNameDefault for more info.

In effect, this patch also homogenizes derivation names produced by fetch* functions so that switching from e.g. fetchFromGitHub to fetchgit would be a noop (assuming the content hash does not change, which is not always the case for fetchFromGitHub since GitHub uses git archive internally and fetchgit does not).

[edolstra]

No, we expressly switched to source last year to prevent the situation where every fetcher produced different store paths. This meant that (for instance) switching from fetchGit to fetchTarball would trigger a rebuild. See NixOS/nix#904.

[GrahamcOfBorg]

Failure on aarch64-linux (full log)

Attempted: gnuradio-rds

Partial log (click to expand)

shrinking /nix/store/z2nrx0lk13f3hd9fb7dfx2193r8fw15v-python2.7-PyQt-x11-gpl-4.12/lib/python2.7/site-packages/PyQt4/QtCore.so
shrinking /nix/store/z2nrx0lk13f3hd9fb7dfx2193r8fw15v-python2.7-PyQt-x11-gpl-4.12/lib/python2.7/site-packages/dbus/mainloop/qt.so
strip is /nix/store/6dpnd5aniypn8124mmy8f88s4mq2zl07-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/z2nrx0lk13f3hd9fb7dfx2193r8fw15v-python2.7-PyQt-x11-gpl-4.12/lib  /nix/store/z2nrx0lk13f3hd9fb7dfx2193r8fw15v-python2.7-PyQt-x11-gpl-4.12/bin
patching script interpreter paths in /nix/store/z2nrx0lk13f3hd9fb7dfx2193r8fw15v-python2.7-PyQt-x11-gpl-4.12
/nix/store/z2nrx0lk13f3hd9fb7dfx2193r8fw15v-python2.7-PyQt-x11-gpl-4.12/bin/.pyuic4-wrapped: interpreter directive changed from "/bin/sh" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/z2nrx0lk13f3hd9fb7dfx2193r8fw15v-python2.7-PyQt-x11-gpl-4.12...
cannot build derivation '/nix/store/capjs47ha289ppvybp2q7jpwk5z9c102-gnuradio-3.7.13.4.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/71vxi4bllz854dj1p538gbvwagk31qjz-gnuradio-rds-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/71vxi4bllz854dj1p538gbvwagk31qjz-gnuradio-rds-1.0.0.drv' failed

[GrahamcOfBorg]

Timed out, unknown build status on x86_64-linux (full log)

Attempted: gnuradio-rds

Partial log (click to expand)

cannot build derivation '/nix/store/qknjgpiqg4rcm3wqdjr744navbbqs927-python2.7-wxPython-3.0.2.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/0yd2m8hysdf5855j0i6f31ira41gqlmk-qt-4.8.7.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/lkvmzj02jlflzwgzzxnaj99df8ajhila-hook.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/bwjvjw92p0mifbi7384859qjgp06lqc2-libpulseaudio-12.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vxn58fji490a8mkgwqbbf06bf5f6z6q3-python2.7-PyQt-x11-gpl-4.12.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/xxhj6vhzhv82c6filpn02r4ifj60ivyf-SDL-1.2.15.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/ly0y1m1q9ijhpn3mb0aymwh1ryiq2szd-qwt-6.1.3.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/ajsgz3mikjqzfb5iwfyr4bdf5yyq6br1-gnuradio-3.7.13.4.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/7n6jk065khcmgl30dcdbk6hf7kd1l0wk-gnuradio-rds-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/7n6jk065khcmgl30dcdbk6hf7kd1l0wk-gnuradio-rds-1.0.0.drv' failed

[oxij]

In practice switching from `fetchgit` to anything else triggers a rebuild anyway because, e.g. tarballs are usually produced by `git-archive` (which `fetchgit` does not do) followed by `autogen.sh`/`bootstrap.sh`. And similarly for most other fetchers. The only exceptions to this rule are different versions of the same thing, like `fetchgit` and `fetchgitLocal`, and fetchers based of `fetchzip`. This patchset doesn't touch the first and keeping the hashes for the `fetchzip`-based is a trivial addition to this, if anybody actually cares. Also note that on master _most_ fetchers don't use `-source` suffix, only the most common ones do. I'd say closing this is premature.

[antislava]

In defense of the current naming scheme, I would like to add that I often switch between github and a other git mirrors, and then I also switch between fetchTarball (with url = "${arg.url}/archive/${rev}.tar.gz") and fetchGit (with url = <path-to-the-mirror-clone>), respectively. Normally, such a switch doesn't cause rebuilds and the out paths are identical (as long as the hash produced by nix-prefetch-git is the same in both cases, which they always are in my experience - maybe I was lucky...). Such an invariance is obviously a very important property to have (even if I also found this very discussion while looking for a way to append a meaningful name to the out path, rather out of curiosity than necessity really.) Nevertheless, could it be possible to pass something like name ? "source" as an attribute of an argument set to allow explicit overrides, if necessary, and maybe add the corresponding option to the nix-prefetch-git function (now there is an --out option but I didn't quite understand what it did)?

[oxij]

As I said above, to get the equivalent of the current naming scheme in that regard all you need to do is to make `repoToName` of this patchset ignore its first argument. One mass rebuild later things will stabilize again.

as long as the hash produced by `nix-prefetch-git` is the same in both cases, which they always are in my experience - maybe I was lucky...

`fetchGit` and `fetchgit` do the same thing, pretty much, so that's not surprising. But `fetchgit` and `fetchFromGitHub` do different things, their results don't match on a lot of repositories.

Nevertheless, could it be possible to pass something like `name ? "source"` as an attribute of an argument set to allow explicit overrides, if necessary, and maybe add the corresponding option to the `nix-prefetch-git` function (now there is an `--out` option but I didn't quite understand what it did)?

I don't understand, please elaborate. AFAIK you can pass `name` to most `fetch*` already, the point here is that the default naming of those `fetch*` outputs is very inconvenient for when you want to glance at the source real quick.

[antislava]

@oxij I meant exactly this thing you mentioned above:

In practice switching from fetchgit to anything else triggers a rebuild anyway because, e.g. tarballs are usually produced by git-archive

In my (limited) experience switching from a github repo and fetchFromGitHub to its git clone --mirror and builtins.fetchGit did produce the same store paths and didn't cause rebuilds, i.e. the desired behaviour which I didn't take for granted. I guess you have more experience, in particular when this is not the case (my scope was mainly haskell projects so far).

On the second point, I thought that e.g. fetchTarball and nix-prefetch-url commands (which I call from make in my workflow to produce json files to be imported by nix, to avoid having explicit hashes in my nix files/logic) did not have the optional name argument, which would then be added to the store paths to make them more informative. If I am wrong then I (personally, at least) do see this issue as closed. Otherwise, I wonder if such an optional argument could be added (and then it wouldn't have to cause any massive rebuilds, would it?).

[oxij]

For instance, repositories with `.gitattributes` file usually produce different hashes when switching between `fetchgit` and `fetchFromGitHub`. Some random examples: firefox/tor-browser/palemoon, qTox, linux, nixpkgs itself...

`fetchTarball` and `nix-prefetch-url` commands *did not* have the optional `name` argument, which would then be added to the store paths to make them more *informative*

`nix-prefetch-url` has `--name` option, see the man page. `fetchTarball`, judging from a glance at the sources, does not.

If I am wrong then I (personally, at least) do see this issue as closed. Otherwise, I wonder if such an optional argument could be added (and then it wouldn't have to cause any massive rebuilds, would it?).

This patchset causes a single massive rebuild. Prefetch scripts, as noted above, can either be fixed by reimplementing some of `repoToName` in bash or, IMO, preferably, just dropped in favor of building derivations without requiring those hashes via nix itself.

[antislava]

@oxij

For instance, repositories with .gitattributes file usually produce different hashes when switching between fetchgit and fetchFromGitHub. Some random examples: firefox/tor-browser/palemoon, qTox, linux, nixpkgs itself...

Thank you for the information - good to know about these cases.

BTW, I checked that both builtins.fetchurl (<nix/fetchurl.nix>) and builtins.fetchTarball accept the optional name argument (appended to the store path). Given that, I personally prefer the current solution #49862 (comment) (avoiding rebuilds on switching the fetcher by default but providing enough flexibility if needed) as it is.

[oxij]

Reopening...

[oxij]

Well, I updated this, but #316668 still needs to be merged first.

[winterqt]

I’d like to try to get this (and #316668) into 25.05, given this won’t break anything due to being opt-in. @oxij Are you up to rebasing, hopefully for the last time? If not, I’d be happy to take care of it for both of them.

[arianvp]

Given that this doesn't change any default can we retarget this to master?

Edit: oh no the last commit does produce a mass rebuild I think. Sorry I misread.

[oxij]

@winterqt Done.

[arianvp]

This will need another rebase now that the dependent PR has been merged. Then we can merge this into staging and we should be all good to go!

[oxij]

Done.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fix-your-fods-fods-and-security/71531/2

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fix-your-fods-fods-and-security/71531/15