The nix flake is broken after pull request #1758

[oati]

Reproduction steps

Attempting nix flake update with any commit after pull request #1758 results in the error

error: input 'helix/nixCargoIntegration/devshell/nixpkgs' follows a non-existent input 'nixCargoIntegration/nixpkgs'

Environment

• Platform: Linux

  • system: "x86_64-linux"
  • host os: Linux 5.15.25, NixOS, 22.05 (Quokka)
  • multi-user?: yes
  • sandbox: yes
  • version: nix-env (Nix) 2.6.1

• Helix version: f31e85a

[the-mikedavis]

I think the problem is described here: NixOS/nix#6013, which was provoked by yusdacra/nix-cargo-integration#57, was fixed in NixOS/nix#6036, and it looks like it was released under https://github.com/NixOS/nix/releases/tag/2.7.0 yesterday. But it doesn't look like nix 2.7.0 has made it to the unstable channel yet.

Does this affect nix run/nix build? I have been able to build on nix 2.6.0 (although nix flake update seems to hang actually, it was just downloading submodules. nix flake update works for me on master)

[the-mikedavis]

It's also worth giving this a try on the latest master since #1659 is now merged

[utherpally]

I had the same problem (tried with latest master)
And my current workaround is:

# Helix editor
helix.url = "github:helix-editor/helix";
helix.inputs.nixpkgs.follows = "nixpkgs";
nci.url = "github:yusdacra/nix-cargo-integration";
nci.inputs.nixpkgs.follows = "nixpkgs";
helix.inputs.nixCargoIntegration.follows = "nci";

[oati]

I'm still failing to build on master.

output of sudo nixos-rebuild switch:

error: builder for '/nix/store/zjahv9zvhqimfb2cb2g62h8vcx4kywq6-helix-deps-0.6.0.drv' failed with exit code 1;
       last 6 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/8rps34c3ww5rz7ymkh5m76xsk5183bkl-dummy-src
       > source root is dummy-src
       > patching sources
       > configuring
       > /nix/store/1sa0afk52ww5by5ibrv3v54l90wzk480-stdenv-linux/setup: line 1364: cargo: command not found
       For full logs, run 'nix log /nix/store/zjahv9zvhqimfb2cb2g62h8vcx4kywq6-helix-deps-0.6.0.drv'.
error: 1 dependencies of derivation '/nix/store/2ka9q4nhmbbmx10ybp7205v0bq8bj5nn-helix-0.6.0.drv' failed to build

[rest omitted]

[oati]

I had the same problem (tried with latest master) And my current workaround is:

# Helix editor
helix.url = "github:helix-editor/helix";
helix.inputs.nixpkgs.follows = "nixpkgs";
nci.url = "github:yusdacra/nix-cargo-integration";
nci.inputs.nixpkgs.follows = "nixpkgs";
helix.inputs.nixCargoIntegration.follows = "nci";

Ok, this worked with Cachix.

[the-mikedavis]

I just tried upgrading to nix 2.7.0 in my dotfiles flake and it correctly handles the transitive dependency when adding helix (see here). So when 2.7.0 is released there will be no need for the workaround above with the extra input on nix-cargo-integration. Earlier I tried updating the flake and got the same error about cargo not being available in the stdenv setup script, but I can't seem to reproduce it now.

Locally I see nix flake update succeeding from within the helix repo, I can depend on helix through the flake, and I can nix flake update my dotfiles flake. Shall we call this closed @erinkim4?

(albeit nix flake check fails, but I think that's an issue caused by this IFD, see also NixOS/nix#4265)

[oati]

I'll reopen if there are problems after 2.7.0

[oati]

Is this a bug in Helix?

trace:

The dream-lock.json for input 'helix-term' doesn't exist or is outdated.
...Falling back to on-the-fly evaluation (possibly slow).
To speed up future evalutations run once:
  nix run .#resolve

(nix proceeds to download every tree-sitter submodule)

[the-mikedavis]

Ah that's from nix-cargo-integration's dependency on dream2nix for fetching and building cargo dependencies. It should be harmless (and I think dream2nix plans to remove the trace but I'm not sure).

The grammars are fetched now with fetchTree (see here), so it should be much faster and cacheable in the nix store. Is nix downloading grammars on every build?

[oati]

Ok, I see. Nix is caching properly. I'll close this issue.

[oati]

Updating helix takes an awfully long time now... maybe Cachix would work better if the call to grammars.nix happened outside of flake.nix? I have no idea if that would actually fix it @the-mikedavis

[the-mikedavis]

I would expect the grammars to cache just as well as any of the cargo dependencies or the resulting helix artifact but I just tried a fresh install and it does seem to fetch every git repository from scratch. I also see it downloading each helix-tree-sitter-.. derivation from the cache, though, after cloning all of the repositories. I wonder if it's because builtins.fetchTree is not a fixed-output-derivation, so nix needs to download the repository to verify that the contents are the same as what the cached artifacts say? I'm not sure.

In any case, I think we could probably add another package to the flake called unwrapped that does not attempt to build runtime/ and wrapProgram hx to use it. Since #1659 was merged, you would then be able to fetch and build grammars after installation with hx -g fetch and hx -g build which are far faster than the nix build. This approach would mean that you'd need to copy runtime/ from this repo to ~/.config/helix/runtime in order to have queries, though, but I suppose that could be automated by nix. I'm not exactly sure how this will work with nix-cargo-integration yet but I bet we can do it with an override.

[anna328p]

This adds lots of state that needs to be manually upkept. Copying to the home directory isn't possible with nix either, without things like home-manager.

I think the best option is to try to fix the repository fetching instead.

[the-mikedavis]

I think we can't do the fetching any better as long as we use git repositories for grammars. fetchGit is very slow and not a fixed-output-derivation. fetchTree is pretty good: you can shallow-clone which is much faster but it's also not a fixed-output-derivation. fetchFromGitHub would be ideal since it's even faster than fetchTree (it downloads a tarball instead of using git) and is a fixed-output-derivation, but it's a no-go because we would need the sha256. And there isn't much appetite for adding sha256 to languages.toml since it would be nix-specific and isn't very easy to calculate if you're not using nix.

I have some ideas for a centralized package registry for grammars that could solve some of these problems but it's a long way off.

[oati]

Maybe a solution could be to use git submodules?

[the-mikedavis]

It used to be that each grammar repository was a submodule on this repo, but it was awkward to use them in regular development (they were very slow to clone and git doesn't update their revisions when switching branches) and nix doesn't consider submodules of a flake's repository to be part of the input (it did briefly but the behavior was reverted).

Submodules did solve the fixed-output-derivation problem, though, because you could either have the helix repo as the input of the flake (here) or what we had earlier was a fetchFromGitHub of the source at a particular revision. Those were horribly slow to build albeit easier to cache, but they also suffered from the submodules revisions getting out of date with the queries which gives a panic (#1746).

This: https://github.com/helix-editor/tspm might be a little less "long way off" than I thought. I bet it wouldn't be too hard to integrate and it would solve both fast downloads (the packages are very small) and the fixed-output-derivation problem because the URLs for artifacts contain the sha256s.

[oati]

My cache is expiring, and I'm having to redownload grammars every hour for simple rebuilds. Rebuilds are now taking 10 to 15 minutes, and this is really a problem.

[oati]

I'm going to be pinning helix to some commit before #1659 for now to keep my rebuilds.

[the-mikedavis]

#1872 was merged recently which improves the time it takes to fetch grammars. I suspect that the grammars will still have to be re-downloaded (I think this is related to the tarball-ttl) but re-downloading them will now be much faster.

I'm still investigating fixing this properly (I think with fixed-output-derivations).

[leeola]

I've not used it long enough to experience the grammars issue yet, but just wanted to note that i was experiencing similar problems trying to use this repo's flake on NixOS and NixDarwin. Upgrading from nixpkgs-21-05 to nixpkgs-21-11 solved it on NixOS, and for NixDarwin changing to nix.package = nixUnstable seemed to work.

Quite the adventure since a lot of errors were coming from unknown (to me) origins.