treewide: migrate usages of nix hash to nix-hash

[Frontear]

NOTE: This PR is incomplete, pending discussion of the changes. Once a consensus is reached I will add in the rest of the changes.

This PR is inspired by the discussion at #436230, and seeks to resolve the issues mentioned in #421823, but from a different direction.

The nix hash subcommand requires experimental-features = nix-command ... in /etc/nix/nix.conf (or passed via CLI args) in order to be used, otherwise it will fail with an error. Furthermore, other distributions of Nix (namely Lix) seem to lack the nix hash convert subcommand that the official Nix has, despite it having been added 2 years ago in Nix 2.20 (NixOS/nix#9452). This means Lix users have to use nix hash to-(base16|base32|base64|sri) ..., which is considered deprecated by Nix (but still supported as of 2.28.4, though could be removed anytime).

Both of these issues will cause problems for other users looking to execute any code that uses them in Nixpkgs. One obvious solution is to tack on --extra-experimental-features 'nix-command' to the Nix command, but this flag is really long and easy to forget by most users (a few scripts were missed or added after #421823 that have been fixed here), and the issue of nix hash convert being non-existent on Lix is an issue that can cause a lot of confusion (as per #436230).

My solution involves transforming all calls to nix hash (and variations with --(extra-)?experimental-features) into the equivalent nix-hash invocations. This will avoid the issues of forgetting the nix-command experimental-feature and be stable across all other Nix implementations since this is part of the legacy CLI interface.

cc @philiptaron @lonerOrz as relevant parties to this discussion.

Problems

The most obvious one to me is that nix-hash by default uses md5 as the hashing algorithm (nix hash uses sha256), and uses base16 representation (nix hash uses sri). These can be fixed by adding --sri --type sha256 to the CLI arguments, but this creates an issue for forward compatibility with nix hash, and once again brings the same issue of "forgetting to add flags".

Alternative

We can instead choose to not concern ourselves with Lix being incompatible with upstream Nix (this isn't the only one, the experimental feature pipe-operator vs pipe-operators is another). This does mean that any contributors who use Lix will have to resolve their issues as they are encountered.

In this situation, the only thing this PR would seek to do would be to add the missing --extra-experimental-features to all nix hash calls that have it missing and remove all of the nix hash -> nix-hash changes.

Sidenote

I originally considered PRing to replace all of the new nix3 subcommands with their nix-legacy equivalents, but I didn't do so because I figured that warranted a larger discussion. One benefit of doing so would be faster evaluation times (going from nix eval to nix-instantiate had pretty significant improvements due to not needing to copy the flake directory so much), but similar issues about forward compatibility arise, hence the need for a discussion.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[RaitoBezarius]

cc @wolfgangwalther -- seems relevant to the minfeatures discussion?

[wolfgangwalther]

Here's my take on this:

• First of all, we should absolutely make sure to support both Lix and Nix.
• Then, we should 100% put in sri hashes. That's consistent with what you get on stderr when you put in a fake hash, and we need that consistency for our users to not be confused about whether they have the right hash or not.
• minfeatures is only related, but this is not a case to be covered by it. minfeatures is concerned about the ability to evaluate Nixpkgs, while the issue here is more about which Lix/Nix you need to be able to develop for Nixpkgs. Those can and probably should be two distinct things.
• We don't need to concern ourselves with "upgrade paths" like in the minfeatures discussion. It would be enough to support recent Lix/Nix versions. Imho, that roughly means "everything that is in nixpkgs-unstable"? We don't need to look at the stable branch, imho.
• I'm not too concerned about things being deprecated. If they still work, we can use them. We can deal with things being removed when they are - these update scripts are not going to break Nixpkgs as a whole, so there is little risk involved.
• I'm not too concerned about "forgetting to add flags". This is just a case of "we gotta do it everywhere consistently, and then people will start copy&pasting the right stuff". Alternatively, we could write a wrapper script around these parts to be used everywhere. I'm sure.. there are already some update script related helpers that deal with hashes (?). GIven that we are only looking at ~10 changes here, this probably has been solved differently already.

[Frontear]

GIven that we are only looking at ~10 changes here, this probably has been solved differently already.

I've intentionally skipped out on a hundred or so files that still use nix hash because I wanted to see what the consensus was before going down the path of implementing them. Quite a lot of them use a mixture of nix hash convert (no work on Lix), and nix hash to-(...) (works on both).

[philiptaron]

I agree with Wolfgang's thoughts above.

For update scripts, it seems worthwhile to insist on them being self-contained: that is, they're not reliant on the user's ambient environment and $PATH to do their job. The use of the double shebang and the --pure flag in the qq update script seems right to me.

nixpkgs/pkgs/by-name/qq/qq/update.sh

Lines 1 to 2 in 45bbb3c

	#! /usr/bin/env nix-shell
	#! nix-shell -i bash --pure --keep GITHUB_TOKEN -p nix git curl cacert nix-prefetch-git jq

So the only thing they rely on, then, is that nix-shell resolves things on the machine. Both Nix and Lix do this and will not break it.

If they are self-contained, in their update step they have their choice of using Nix or Lix or nix-generate-sri (supposing someone made such a tool) or nurl or what-have-you in order to resolve the hashes. All fine choices. There's no need for consistency here from a practical standpoint, only from a easy-for-others-to-review standpoint. And if we need to do a treewide change to make them all consistent, it's pretty easy to do so.

Now, previously I called the Nix --experimental-features nix-command stuff "ugly as sin" and "an indictment of the Nix feature process". I still think that. I'm very open to PRs like this which make update scripts read better. But they do have to work.

The best option, in my view, is to have a tool which comes from neither Nix nor Lix, is fit for the Nixpkgs update process, and is used across all the update scripts. nurl -H, for instance, is super short and quite effective.

[wolfgangwalther]

For update scripts, it seems worthwhile to insist on them being self-contained: that is, they're not reliant on the user's ambient environment and $PATH to do their job. The use of the double shebang and the --pure flag in the qq update script seems right to me.

I disagree wrt to treefmt, also see the discussion in #433588 and previously in #425068.

If they are self-contained, in their update step they have their choice of using Nix or Lix

I disagree here, too: These scripts should be mostly self-contained, but not regarding treefmt and not regarding Nix/Lix either. The user shouldn't have to pull 4 different versions of Nix or Lix to run various update scripts, if they already have a working implementation on their system.

The best option, in my view, is to have a tool which comes from neither Nix nor Lix, is fit for the Nixpkgs update process, and is used across all the update scripts. nurl -H, for instance, is super short and quite effective.

I don't know nurl, but I fully agree with this part: Whether that's a separate tool or a wrapper script that we build around Lix/Nix that just works doesn't matter to me.

[philiptaron]

I disagree wrt to treefmt, [because it ought to be provided by the nixpkgs-ci shell]

OK, I see that point, though I don't think it's really in opposition -- it's just raising the containedness from "this single script" to "the ambient environment provided by the CI default.nix". Sounds good to me!

The user shouldn't have to pull 4 different versions of Nix or Lix to run various update scripts, if they already have a working implementation on their system.

Here we do have a real opposition. Claiming a "working implementation" in the face of the real lack of coordination w.r.t. command line interface, default configuration, and more between the projects is just false to facts. And the divergences will continue, and they should!. The cited differences between them above are enough for me to want to make either the update script itself or the CI environment remove the non-determinism through providing a consistent copy. It's a false economy to avoid the download.

[wolfgangwalther]

We can't make an update script independent of the "host nix", though: It determines how the nix-shell shebang behaves etc. (also see #425551). So we can't dodge the issue of which implementations in which versions we support or not.

[wolfgangwalther]

Also related: #421261 and #400784. I think there were some other issues somewhere around incompatibilities between running certain Nix versions with certain Lix daemons or the other way around (maybe even newer Nix vs older Nix). So even with a pinned Lix/Nix in the shell, it would still have to talk to the daemon if any builds were made. Not sure whether that's the case for any update script, but certainly for some of the other scripts we have in-tree.

[philiptaron]

Thanks, #425551 is quite relevant, and the other issues you linked sketch out the problem also. I've subscribed to that issue. I think I now agree with you that using the ambient Nix is the only way to go in light of the Nix-in-Nix (or Lix-in-Lix or various version combinations outlined in those issues) problems.

I wish it were not so.

[Frontear]

From what I'm reading, the consensus seems to be that a) yes, this is an issue and support should be provided for both Nix and Lix, and b) a wrapper/external tool would be a good way to provide this functionality (I'm thinking of the name pkgs.nixpkgs-hash).

My ideas for the wrapper would just be to leverage nix-hash internally and provide some extremely simple arguments such as --path and --file from the script. These will internally decide what flags to use on nix-hash, and in the case where nix-hash isn't sufficient, it'll try to make use of nix hash with the appropriate experimental features enabled and compatibility checks for Lix (should the subcommands diverge further than upstream going forward).

I think I now agree with you that using the ambient Nix is the only way to go

Could you clarify what you mean by ambient here? Does ambient = using the locally installed Nix?

[philiptaron]

Could you clarify what you mean by ambient here? Does ambient = using the locally installed Nix?

Yes.

[wolfgangwalther]

I assume somewhere in pkgs/common-updater would be the right place to put such a script.