add maintainer scripts for haskell package generation

[maralorn]

This is a slightly reworked version of #86699. Thx again @hyperfekt.

Introduces a script that can be used to update the Nix expressions for
the Haskell package set. In service of that, also

• introduces cabal2nix-latest, which pins the hackage2nix version used
• changes all-cabal-hashes to use fetchFromGitHub
• adds update-hackage.sh & update-cabal2nix-latest.sh maintainer scripts

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[maralorn]

Okay, as I feared this PR breaks the callHackage function, because it expects all-cabal-hashes to be a tarball.

[maralorn]

There are two ways out. Either change callHackage to not expect a tarball or change this PR to not unpack.

I think the second is the way to go, because we are talking about a lot of files and directories here and it seems quite inefficient. But I may be unaware of other arguments or it might be that we pay the performance cost anyways …

[ghost]

I've packaged ratarmount and am currently reworking my PR to use the compressed version of all-cabal-hashes instead. Ideally cabal2nix would adopt the same technique cabal does to work directly with the compressed archive (as suggested by peti), but in lieu of that it seems like a workable solution.

[maralorn]

@hyperfekt Oh, I have already pushed a fixed version of this PR. What do you think?

[ghost]

I think that the last time I tried to unpack that archive on ext4 it took dozens of minutes if I recall correctly. ^^
Not to mention that it's 700MB big, which even considering GCing is a lot.

[maralorn]

Yes, put since hackage2nix expects it to be unpacked we don‘t have a choice, do we?

At least this way people using callHackage don‘t need to unpack it. Only people who run the regenerate script.

[maralorn]

But, yeah, if the ratarmount solution is cooler. Let’s go for it.

[ghost]

That is why I packaged ratarmount. It allows us to mount the archive without actually unpacking it onto the disk. I'll see in a bit how fast that ends up being.

[maralorn]

Yeah, go for it!

[maralorn]

Sadly there might be a trade-off. Unpacking with nix once is probably slow on the first run, but maybe it‘ll be faster when rerunning?

[ghost]

Ideally it'll be almost as fast, at least if I can get Hydra to build the index file for the archive once and have that be used by everyone for every run.

[sternenseemann]

I would rather rework callHackage and have all-cabal-hashes be the unpacked version. Having a compressed store path has no real advantages in that case because we compress them anyways while fetching from a binary cache. However mounting a tar archive will probably have significant impact on either RAM usage or access performance over having the archive already unpacked in store.

[maralorn]

I don‘t have a strong opinion about this. (And the old solution still in the reflog.)

But it’s a 700MB file that everyone would need to write to disk and keep around. I am not exactly sure what callHackage does with the tarball but it seems pretty fast. So when we can get hackage2nix to also use that tarball with reasonable speed I‘d prefer that.

I personally can spare the RAM more than the disk space … For a real principled decision we would need measurements. But that’s likely not worth the effort.

[ghost]

The only reason I got into this in the first place was that the number of files was so high that some file systems had serious trouble with it.
But after measuring it the ratarmount version is almost 5 times slower at regenerating hackage-packages.nix.
I guess the slowness for some is just something we'll have to live with, rather than making it this much slower for everyone.

Leaving all-cabal-hashes uncompressed would mean that anyone using callHackage gets 700MB dumped into their store (filesystem compression is presumably without effect because these are tiny files), versus only the people regenerating the Haskell package set.

[sternenseemann]

callHackage extracts only the necessary subset of the tar:

nixpkgs/pkgs/development/haskell-modules/make-package-set.nix

Lines 144 to 148 in d05202e

	all-cabal-hashes-component = name: version: buildPackages.runCommand "all-cabal-hashes-component-${name}-${version}" {} ''
	tar --wildcards -xzvf ${all-cabal-hashes} \*/${name}/${version}/${name}.{json,cabal}
	mkdir -p $out
	mv */${name}/${version}/${name}.{json,cabal} $out
	'';

I guess ratarmount is fine for now, we can always change things if we run into problems.

[ghost]

This should unpack directly to $out instead of using the unpackPhase, since it takes almost twice as long otherwise due to having to move all files into the store.

[maralorn]

Good catch.

[maralorn]

@hyperfekt Does the change I pushed look better?

[ghost]

Yes, that should work well.

[maralorn]

@sternenseemann Well, this PR goes the middle ground. callHackage stays efficient but for actually doing the update it will extract into the nix-store. (Which apparently is faster than using ratarmount.)

[ghost]

Ideally a comment would either indicate that an alternative hackage2nix can be passed to this script (which may be useful for developing fixes to hackage2nix), or the functionality would be removed; since the syntax for it isn't very obvious, well known, or googleable and will probably just become arcana without it.

[maralorn]

Well then let’s remove it until someone needs it. It’s easy enough to add again.

[sternenseemann]

Can we add the update script via passthru here?

[maralorn]

Can you enlighten me as to who or what does actually use the passthru update script?

I am a bit sceptical because at least fetchgit didn‘t accept a passthru attribute.

[sternenseemann]

fetchurl accepts passthru. IIRC some automation uses it.

[maralorn]

Okay, done.

nix-shell maintainers/scripts/update.nix --argstr package all-cabal-hashes is an alias for our script now.^^

[sternenseemann]

Adding the update script here as well would be nice, but should probably be added via an override in non-hackage-packages.nix to avoid a terrible sed hack.

[maralorn]

Okay, well. Now that I learned that I can use nix-shell maintainers/scripts/update.nix --argstr package haskellPackages.cabal2nix-latest I wonder if it’s a smart id to have the update script at such a prominent position as maintainers/scripts/haskell.

[teto]

The faith is reborn #62105 (comment)

[cdepillabout]

Now that we're on Stackage Nightly, I imagine we can drop quite a few of these lines here, since these packages will generally also be the latest version on Hackage. That is to say, the version of the package in Stackage Nightly should also be the latest version of the package on Hackage.

If the packages in Stackage Nightly are not the latest version on Hackage, then the Stackage maintainers have probably done that for a good reason. For instance, the packages aren't compatible and don't compile together.

Although this is not something that should block this PR. We can do this at some point in the future.

[sternenseemann]

We should drop every one of these except the ones that are nixpkgs-related jailbreak-cabal, cabal2nix, ...

[maralorn]

Yeah, let’s clean that up. But that requires a bit of careful deliberation to see if we break anything right now, so I would like this to happen in an independent PR.

[cdepillabout]

I tried out all of the maintainer scripts in this PR and they all seem to work well.

This LGTM.

[maralorn]

Thanks everyone! Especially to @teto and @hyperfekt for your tries of doing this.

[ghost]

Thanks a lot to you as well! I really appreciate you taking this over the finish line; and I'm excited for the future of the nixpkgs Haskell package set.