Use nix-portable to allow using the devshell without installing DD-enabled nix globally

[jaen]

I wanted to try nix-ninja, but I also didn't want to switch nix on my machine to some alpha release globally, so I have tried to figure out a way to have it work with a chroot store and what finally worked was nix-portable (with a few fixes to have it work with new nixpkgs and additional features) — now I can run nix build .#examples-nix from the devshell and have it work out of the box. I tried to something more minimal by directly using bubblewrap, but I must've been missing something, as it was not enough.

I'll leave it as a draft for now, because it probably should not be merged as-is, but I figured I can at least start a discussion on improving developer experience on this, until DD is widely available.

Things to consider:

• I have changed the default shell to use this wrapped nix, but maybe that's not desirable for some reason — I imagine there could be two shells and some way to select which one you want direnv to use,
• I have branched off the lix branch, because it has at least started splitting the flake.nix into smaller chunks, so it was a bit easier to get a handle on it. I can rebase it on top of the main branch if you want, but I think it would be useful to split it even more, so it's more manageable (YMMV, but I usually use flake.nix for inputs only, import some outputs.nix for output definition and all devshells, packages and so on go into nix subdirectory, like the one I've added for nix-portable),
• maybe the wrapper could be improved somewhat — for example it could sync the chroot store to your global one (after it finishes, or maybe somehow track what gets added?), so you can run whatever it built easily without any need for chroot/bubblewrap/whatnot on the resulting binaries.

[elpdt852]

Thanks! I wanted to have a way to develop nix-ninja without switching nix so this is an interesting approach. I'll give this a test drive myself and I'm curious if it has any caveats. You should also look at https://github.com/obsidiansystems/sandstone where they have a --store /tmp/store approach using their pinned nix.

I've starting pulling out the flake.nix by splitting some irrelevant changes from the lix branch to #29. I had a good experience using flake-parts in nix-snapshotter, so that might be my future direction, let me know what you think about that. If we decide to stay with nix-portable, we can even introduce an easy boolean NixOS module option toggle to enable this development workflow. Internally at work, we have a few differently named nix binaries each with a different NIX_STORE so it may not be desirable when developing locally unless it accounted for that as well.

[jaen]

Yeah,I've seen what sandstone does, but only after I've gotten this working. I can also see how it compares, as it's certainly simpler and doesn't have sandboxing overhead — when starting this I actually didn't realise that local stores pretend to be /nix/store anyway, which was my main motivation for using something like nix-portable. If it's not necessary, then all the better.

I appreciate flake-parts conceptually, but the only time I've tried to use them in anger I ended up not being able to figure out how to wrangle them xD I'm reasonably sure I was mishandling them though, so if you have had good experience with them then by all means do feel free to use them — any way to not put everything in flake.nix should be helpful for maintainability.

Internally at work, we have a few differently named nix binaries each with a different NIX_STORE so it may not be desirable when developing locally unless it accounted for that as well.

Right, which is why I suggested it could be an alternate shell instead — what I have done in some of my projects is something like this:

source_env_if_exists ".envrc.user"

# Not using flake by default because we might have YUUUGE files
export FLAKE_SHELL=${FLAKE_SHELL:-".#default"}
export USE_FLAKES=${USE_FLAKES:-"false"}

if has nix; then
  # Are flakes supported and enabled?
  if nix show-config | grep experimental-features | grep -q flakes && [ "${USE_FLAKES}" == "true" ]; then
    # Do we have nix-direnv
    if ! has nix_direnv_version || ! nix_direnv_version 3.0.6; then
      source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/3.0.6/direnvrc" "sha256-RYcUJaRMf8oF5LznDrlCXbkOQrywm0HDv1VjYGaJGdM="
    fi

    use flake "${FLAKE_SHELL}"
  else
    use nix --argstr shellName "${FLAKE_SHELL##.#}"
  fi
fi

where you can use the .envrc.user file to select an alternative default shell for direnv. Then the default shell could continue relying on system-wide nix and you could specify the one with pinnned and chrooted nix if you need that (or other way round).

[jaen]

@elpdt852 okay, so I've played around with how sandstone was doing it and it seems to work. So I have written a basic bash wrapper for that, but then I thought about how to sync paths back to the global store and wasn't sure about it — syncing the whole store seemed slow when I tried it, looking for result symlinks after build could work but it wouldn't be ideal (you could only sync back after build is finished and also handling -o properly would be somewhat annoying in Bash) and if I watched the chroot store with inotify, I don't think that would be portable for people who use other OSes than Linux.

So in the end I ended up writing as simple…ish wrapper around nix in Rust — it mostly ensures nix gets scoped to the chroot store and then watches for file changes while nix is running and syncs them back to the global store. It's not done yet and needs more polishing (around config, but also it seems that watching only the .lock files being removed doesn't work for CA derivations), but I felt like reminding myself how writing rust goes.

Anyway, I have made an alternate branch with the simple bash script and with the Rust tool I've written — 6ebf6d6. It defaults to the tool, but you can change the devhsell package to nixDdWrapped to see how the bash version compares.

The question here is — do you think any of those two approaches would work. If so, which one would you prefer? Cleaning up the Rust tool, or the Bash script (maybe with some added code to sync the ./result symlinks?

[jaen]

Hopefully it's unnecessary after NixOS/nix#13181 — I've had some issues trying my solution above again recently, so maybe it's better not to rely on that if upstream finally re-enabled dynamic derivations.

[Ericson2314]

It is good that it is working again upstream, but one may still want to not enable experimental features in their main store, so I think thinking about the chroot store for demos is still prudent.