haskellPackages.mkDerivation: New intermediates output

[Gabriella439]

This adds a new intermediates output that can be used to accelerate a Haskell build by importing build products exported from a prior similar build.

The motivation for this is explained by the following post:

https://www.haskellforall.com/2022/12/nixpkgs-support-for-incremental-haskell.html

This is a subset of the changes from #204020 that I believe are safe to merge while waiting for NixOS/nix#7362 to be reviewed. I've incorporated outstanding feedback from the last PR into this one.

I structured this to be a hash-preserving changes for Haskell packages that don't use the feature, so this won't trigger unnecessary rebuilds or breakage.

This is a redo of #213817 since that inadvertently subscribed too many people to the pull request discussion.

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[cdepillabout]

LGTM, thanks!

[sternenseemann]

Can we do btter than this? This test would succeed as well if the intermediates were just ignored.

[9999years]

Hm. We could add a check to the buildPhase to make sure that find dist/build | wc -l is positive, or something? I'm open to suggestions here.

The nice thing about incremental builds in general is that they degrade gracefully — the build works just as well without the intermediates, it's just slower.

In any case, this test does confirm that the code to copy the intermediate artifacts doesn't break the build.

[cdepillabout]

The nice thing about incremental builds in general is that they degrade gracefully

Yeah, I was thinking about this as well, and this is the same conclusion I came to. I couldn't think of a good way to test that the intermediates were actually being used (without doing something hacky/brittle), but this does test at least that the intermediate artifacts don't break the build.

[sternenseemann]

New arguments should also be documented in the nixpkgs manual.

[9999years]

Done.

[sternenseemann]

I wonder if we can find a better dir for this (under share?) that includes pname and version so there are no collisions if enableSeparateIntermediatesOutput is false and it is installed by accident.

[9999years]

Good idea. I updated this to install intermediates to share/haskell/${ghc.version}/${pname}-${version}/dist. I think it might be better to key them under ${pname} rather than ${pname}-${version}, though — we wouldn't want to have an incremental build cache miss just because a version number incremented. Let me know what you think.

[cdepillabout]

This LGTM.

Sorry to take so long with this. Could you fix the merge conflicts? Then lets merge in.

[9999years]

Fixed merge conflicts.

[cdepillabout]

Thanks! Looking forward to see if anyone uses this in any interesting ways.

[sternenseemann]

Please remember to make to rebase the branch to have conforming commit messages next time!