Overhaul nix flake update and nix flake lock UX

[iFreilicht]

Motivation

The current interface of nix flake lock and nix flake update is surprising and verbose. This lead to #5110, which has been open for over two years and is currently the fourth-most upvoted issue in this repository.

After my proposal, some comments from users and a dedicated discussion from the Nix Team itself, this is the PR to fix this. The changes are as follows (copied directly from the release notes):

• nix flake lock only creates lock files and adds missing inputs now. It will never update existing inputs.

• nix flake update does the same, but will update inputs.

  • Passing no arguments will update all inputs of the current flake, just like it already did.
  • Passing input names as arguments will ensure only those are updated. This replaces the functionality of nix flake lock --update-input
  • To operate on a flake outside the current directory, you must now pass --flake path/to/flake.

• The flake-specific flags --recreate-lock-file and --update-input have been removed from all commands operating on installables. They are superceded by nix flake update.

Testing

If you want to try out this PR, you can temporarily install it by running these two commands:

nix build github:iFreilicht/nix/flake-update-lock-overhaul
alias nix="$PWD/result/bin/nix --extra-experimental-features 'nix-command flakes'"

After that, running nix flake lock and nix flake update will use the most current version of this PR.

Context

See the links above for context.

My changes are pretty non-invasive. The only thing changed in the underlying lockFlake function are comments referring to the removed --update-input flag and a tiny bugfix that caused nix flake lock to not actually output what it added to the lockfile when initially creating it. The code examples for both nix flake lock and nix flake update were also incorrect, though those fixes are superseded by the new docs anyway.

You'll also see a note in both docs that refers to a potential tripping hazard in the way nix flake lock and nix flake update handle the flake-url argument:

Note: When trying to refer to a flake in a direct descendant directory, write ./another
instead of another, else Nix will try look up the flake in the registry.

To me this is clearly a bug, but I understand that the generality of flakes might make a seemingly obvious fix controversial, and I don't want this PR to be blocked because of that, so I'd be happy to create a separate PR for this issue.

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

• agreed on idea
• agreed on implementation strategy
• tests, as appropriate
  • functional tests - tests/**.sh
  • unit tests - src/*/tests
  • integration tests - tests/nixos/*
• documentation in the manual
• documentation in the internal API docs
• code and comments are self-explanatory
• commit message explains why the change was made
• new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-01-nix-team-meeting-minutes-84/32466/1

[iFreilicht]

Rebased on current master and fixed all tests. It's ready to merge, just needs a review.

EDIT: Huh, seems the completion test passes on macOS and fails on Linux. Will investigate later.

[iFreilicht]

Alright, tests are fixed! This was a bit of a tough one, see the commit message for more info.

[thufschmitt]

Thanks for the nudge @iFreilicht. I actually had a WIP review but never finished it 😬

Apart from a few minor things, this is good to go from my side!

@edolstra you were assigned to this PR. Do you want to have another look? Otherwise I'll just merge it next week.

[fricklerhandwerk]

Docs LGTM. The non-blocking nitpick applies to two instances. Update if you have time and agree it's an improvement.

[thufschmitt]

Just rebased on top of master to fix a small conflict and set auto-merge. Should be good to go in 1h max once the CI is green.

Thanks a lot for that great change @iFreilicht !

[9999years]

@iFreilicht Thanks for your work getting this through!

[chreekat]

This is awesome!

[Ericson2314]

This probably should have kept around the previous --update-input but deprecated. (While we are well within our rights to make sudden breaking changes to experimental features, when it is easy to leave up some back compat affordance we generally do so at least for a few releases. I think it would be easy to do so in this case?)

@iFreilicht Would you be willing to tackle this?

[bryango]

• The flake-specific flags --recreate-lock-file and --update-input have been removed from all commands operating on installables. They are superceded by nix flake update.

Hey this is a nice refresh for nix flake, but it eliminates the possibility of doing the super convenient, single command

nix build --update-input nixpkgs

So should we now do

nix flake update nixpkgs && nix build

instead? 👀 If this is the final decision, I can learn to live with it.

[iFreilicht]

@bryango Yes, that was the decision, and nobody objected at the time. Of course, this was not a data-driven decision, so I'd be curious to hear your use-case for --update-input. Maybe it's genuinely useful over the longer one-liner, though we are only talking about five extra keystrokes here.

[bryango]

Ah, there is not much that is "genuinely more useful" about --update-input. I do think that the old nix build --update-input nixpkgs fits better with my cognitive model: the main objective is nix build, and occasionally I will need to decorate that with some options, such as --update-input nixpkgs (this often comes as an afterthought to nix build).

Therefore, I would hope that the --update-input flag could be un-deprecated for commands such as nix build and nix develop. However, do note that this could be a very personal workflow (obligatory xkcd), and as mentioned before, I can live with the two-step commands as well, so this is not a protest 😆 just to share a possible common workflow. And, for nix flake I think the new interface is much nicer and I fully support the changes.

Update: oh, I just thought of something that could be "genuinely useful", but I am not sure if I have done it before, so this may or may not be realistic: suppose I want to try out some remote flake, but with some updated input; before I can do:

nix run github:some-remote-flake --update-input nixpkgs

... without polluting the working directory at all, but now I will have to clone the remote flake first, nix flake update nixpkgs, and finally nix run. Do note that this is an imaginary workflow; I am not sure if I have done it before.

[bryango]

suppose I want to try out some remote flake, but with some updated input; before I can do:

nix run github:some-remote-flake --update-input nixpkgs

... without polluting the working directory at all, but now I will have to clone the remote flake first, nix flake update nixpkgs, and finally nix run.

In light of this, I think it would be better to un-deprecate the flags (while keeping the improved nix flake behavior). Building on https://github.com/NixOS/nix/pull/9449/files one can simply remove the warn s for these flags. What do you think? @fricklerhandwerk @iFreilicht

[iFreilicht]

Got it, thank you for your thoughts! That imaginary workflow is indeed very useful, and would currently be served by --override-input, though you'd have to write nix run --override-input nixpkgs nixpkgs, meaning you override the input nixpkgs with whatever flake nixpkgs resolves to locally, which could be a pinned version of nixpkgs, too.

What you show with --update-input doesn't actually work as far as I could tell, which is partly why I wanted to get rid of it; it only works for local flakes you have write access to, even though you would expect otherwise.

The other reason is that nix has an issue with too few maintainers for too many features. While the flakes interface is still experimental, it makes sense to remove some features in favor of others to keep it as small as possible. Having multiple ways of achieving the same task makes keeping consistency, testing and documenting more difficult. Your demonstrated usecase is a potential feature for the future, but when it is implemented, we have to ensure it works properly in all situations it is available in.

Right now, the priority regarding flakes is to stabilize them, so putting in the required amount of work to make this flag work as expected is not feasible, unless a contributor wants to take that over, and even then, finding time in a maintainers schedule can take months. So I would say deprecating and removing these flags is the best option right now. It says clearly:
"This might not work as expected, and we will remove it, not fix it."

[pwaller]

https://nixos.wiki/wiki/Automatic_system_upgrades currently documents that you can supply --update-input as the flags to auto-update inputs for a nightly update on flake-based systems. This gets passed through to nix build. Presumably automatic upgrades of nixos systems need to be fixed before this can be removed, otherwise we could end up in a deadlocked situation where an machine taking auto-upgrades breaks and won't take a fix through an automatic update, because automatic updates are broken.

Someone suggests that this is broken currently, but the documented configuration still appears to work for me. Is there a plan to fix it?

[iFreilicht]

@pwaller nixpkgs 24.05 still ships nix 2.18.8, as the more recent releases weren't stable enough when the freeze for NixOS 24.05 happened. This PR was released as part of version 2.19, so it isn't in 24.05 and will not be backported there either.

If system.autoUpgrade is broken, please report an issue in nixpkgs. I found this issue which aligns with the NixOS Wiki article and indicates to me that nixos-rebuild sits at the root of this issue.

As flakes and the CLI around them are still considered unstable, I believe it's unlikely that a change like this will get reverted, though I am not a maintainer and don't speak for the project.

[pwaller]

FWIW, I'm not asking for it to be reverted but pointing out that this is still the documented way of doing something.

If it is deprecated, I would like to know what I should move my system to, lest someday automatic updates stop functioning such that they won't automatically recover.

[iFreilicht]

FWIW, I'm not asking for it to be reverted but pointing out that this is still the documented way of doing something.

The NixOS wiki is not official documentation. Nothing in there is guaranteed to work, it's just community members trying to share knowledge.

I would like to know what I should move my system to, lest someday automatic updates stop functioning such that they won't automatically recover.

The simplest solution is to set nix.package = pkgs.nixVersions.nix_2_18;. This will ensure the nix version isn't updated while the rest of your system is.

[pwaller]

I'm not sure that simple solution works for me. I've encountered NAR hash mismatches and behaviour divergences which mean that I'm now stuck on a ratchet of increasing my nix version to keep things working, see for example: #11428 (this isn't the only issue I've encountered that has forced an update unfortunately, but is one I've publicly documented). #11195 I think is another example where one version of nix will treat an input differently than another.

To clarify what happened there, because the software started producing new NAR hashes, I was incentivised to switch over to the new version for forward compatibility with Nix. I had to accept instances of past flakes being broken so that there is a point in time onwards where things work with future versions of nix.

[pwaller]

I'm speaking up because I'm not the only person who has encountered this. I don't know that many people personally using Nix so N=<a few of us> noticing suggests to me this could be a wider problem. I've filed nixpkgs NixOS/nixpkgs#349734