nix flakes

[shish]

Nix flakes appear to have a bunch of benefits over the current shell.nix approach, but they also seem more complicated to set up and I haven't got my head around them yet...

I think in my ideal world:

• nix develop runs a shell with all the development tools necessary to build things
  • if run in the root, it installs all deps for all langauges; if run inside a language folder, it only installs what's needed for that langauge
  • From within the shell launched from the root folder, ./utils/bench.py --default --frames 10 --threads 4 should report everything passing
• nix build builds binaries for the current language / all languages
  • to avoid naming conflicts, maybe the output binaries should be named bin/rosettaboy-<language>-<variant> instead of <language>/rosettaboy-<variant>?
• it's possible to install by URL?
  • Something like nix shell github:shish/rosettaboy will build and install all the versions?
  • Does nix have a syntax for subfolders, so people can nix shell a specific language version without needing to pull in hundreds of megs of dev tools for all languages?

[jbboehr]

The flake way appears to be to expose them all as attributes in the root flake, so:

# from flake.nix devShells."<system>".${attr}
nix develop .
nix develop .#cpp
# could also probably be nix develop .#cpp.lto
nix develop .#cpp-lto
nix develop .#php
# etc

# from flake.nix packages."<system>".${attr}
nix build . 
nix build .#cpp
nix build .#cpp-lto
nix build .#php
# etc

# from flake.nix apps."<system>".${attr}
nix run . 
nix run .#cpp
nix run .#cpp-lto
nix run .#php
# etc

# Or from a URL:
nix run github.com:shish/rosettaboy#php

Although there isn't currently an implementation for build or run, so that would have to be implemented if you wanted those to work.

Although there's probably better examples, I flake-ified this recently as an example.

See: https://nixos.wiki/wiki/Flakes#Output_schema

[jbboehr]

In general, you can actually re-use most already-written nix scripts in either direction, as long as they are pure.

From flake-style to old-style: just call e.g. pkgs.callPackage ./shell.nix { inherit pkgs; } inside the flake [0]

From old-style to flake-style: https://github.com/edolstra/flake-compat

[jbboehr]

I think I would recommend starting by adding old-style build derivations for each language in e.g. cpp/default.nix and/or cpp/derivation.nix and making sure they are working, and then later importing them into a flake.

You also get most of a devShell for free when you implement the build derivation.

(edit) or maybe not, will end up with a bunch of unnecessary pinned dependencies in the old-style scripts (/edit)

[rrbutani]

I mentioned in #57 that I started putting together a flake for rosettaboy way back when. It's definitely fallen out of date by now but hopefully it's still a useful starting point.

In addition to what you described in the OP:

• devShells.<language> to a dev-shell for a sub-folder with nix develop .#<language>, devShells.default/nix develop for a shell with all the deps
• packages.<language> for nix build
• apps.<language> for nix run

I was also aiming to:

• expose the tests and lint checks under checks. (though I only got around to doing opus5)
• and to expose apps.<language>-bench targets (i.e. nix run .#cpp-bench)

---

• nix develop runs a shell with all the development tools necessary to build things
  • if run in the root, it installs all deps for all langauges; if run inside a language folder, it only installs what's needed for that langauge

• Does nix have a syntax for subfolders, so people can nix shell a specific language version without needing to pull in hundreds of megs of dev tools for all languages?

@jbboehr is correct; the idiomatic way to expose separate devShells/packages/etc. is with attributes; i.e. devShell.<language> outputs in the flake. nix commands within a flake aren't really sensitive to the subdirectory within a flake that they're run in so you'd need to, for example, run nix develop .#cpp instead of just running nix develop in the cpp subdir.

I believe it's possible to get nix develop in language subdirs to do with you're describing using sub-flakes but I think that going that route is more trouble than it's worth (requires a fair bit of duplication; a separate flake + lock file per language in addition to a top-level one).

Using direnv with use flake .#<language> (or use flake .#$(basename "$PWD") if you want since it ultimately is just bash) in an .envrc in each language directory gets you pretty close to what you're describing though (on cd into each language directory your shell is updated with the state from nix develop .#<language>).

• to avoid naming conflicts, maybe the output binaries should be named bin/rosettaboy-<language>-<variant> instead of <language>/rosettaboy-<variant>?

• it's possible to install by URL?
  • Something like nix shell github:shish/rosettaboy will build and install all the versions?

nix run .#<language> would let you run without "installing" the rosettaboy binaries; I think nix profile install would be more akin to installing.

For the former (nix run) there's no issue with the binaries all being named rosettaboy but in the latter case you're totally right: the binary names would clash (and nix profile install would complain about this assuming all the packages have the same priority).

---

I think the bulk of the work here will be getting each language's impl to build in the sandbox (and then to run in the sandbox if you want to have checks.* in the flake outputs).

A good first step is probably to add a flake with just devShells for each language (where each output just imports the corresponding shell.nix); something like this:

rosettaboy/flake.nix

Lines 8 to 39 in 453043b

	outputs = { self, nixpkgs, flake-utils }: flake-utils.lib.eachDefaultSystem (system: let
	pkgs = nixpkgs.legacyPackages.${system};
	lib = pkgs.lib;
	# Get each directory with a `shell.nix`:
	languages = with builtins; lib.pipe ./. [
	readDir
	(lib.filterAttrs (_: value: value == "directory"))
	attrNames
	(filter (dir: pathExists (./${dir}/shell.nix)))
	# Exclude `utils`:
	(filter (dir: dir != "utils"))
	];
	# For each language, expose `shell.nix` as a devShell:
	#
	# Also include the deps of `utils` in the shell.
	utilsShell = import utils/shell.nix { inherit pkgs; };
	langDevShells = lib.genAttrs languages (lang: pkgs.mkShell {
	name = "rosettaboy-${lang}";
	inputsFrom = [
	(import ./${lang}/shell.nix { inherit pkgs; })
	utilsShell
	];
	});
	in {
	devShells = langDevShells // {
	default = pkgs.mkShell { inputsFrom = builtins.attrValues langDevShells; };
	utils = utilsShell;
	};
	});

(branch here; if the above looks okay, let me know and I can open a PR)

[shish]

run nix develop .#cpp instead of just running nix develop in the cpp subdir.

This seems fine to me - I'm happy to adjust myself to do things the idiomatic way, rather than trying to force nix to conform to my idea of how I'd do it if I were building from scratch :)

[jbboehr]

I've been looking into writing derivations for some of the other languages and a lot of the nix language tooling is moving to flakes, so I think it would be good to get a basic flake up and running. I can make it work without flakes, but it's ugly.

@rrbutani's nix-flake-devShells branch looks like a solid base to me, although, and I know the existing code does this, I would avoid using iteration for the language packages as, assuming you want to put language-specific nix files in the language folders, it'll probably be desirable to pass a subset of inputs from the flake into each one, e.g.

rec {
  packages = {
    go = pkgs.callPackage ./go/derivation.nix { inherit gomod2nix; };
    rust = pkgs.callPackage ./rust/derivation.nix { inherit naersk; };
    rust-debug = pkgs.callPackage ./rust/derivation.nix { inherit naersk; debugSupport = true; };
    rust-lto = pkgs.callPackage ./rust/derivation.nix { inherit naersk; ltoSupport = true; };
  };
}

Some of the ones I looked into briefly so far:

Rust via https://github.com/nix-community/naersk
Go via https://github.com/nix-community/gomod2nix
Python via https://github.com/nix-community/poetry2nix