nix/haskell-packages: replace IFD with importing generated files by basvandijk · Pull Request #1220 · caffeinelabs/motoko

basvandijk · 2020-02-19T13:41:28Z

https://dfinity.atlassian.net/browse/INF-1002

Replace IFD calls to callCabal2nix with importing generated default.nix files.

The generated default.nix files are stored in the repository and contain instructions on how to regenerate them.

The generation of the default.nix files is done using the haskellSrc2nixWithDoc function which generates a directory containing a default.nix which is the result of running cabal2nix
with the extraCabal2nixOptions on the provided src. A header is then added to the default.nix which contains instructions on how to regenerate that file.

Finally the new jobs are added which check that the stored default.nix matches the expected default.nix. So whenever the .cabal files are updated and don't match the stored default.nix anymore CI will complain.

nmattia

Damn I hate the Nix haskell infrastructure

nomeata

Nice solution!

I wonder if it would be nicer to keep the generated files all in

./nix/generated/

(e.g. ./nix/generated/winter.nix), have one command to update all generated files, and only one check, and not require an essentially empty winter/ directory. But probably overkill.

Maybe add a section in Building.md about Updating Haskell dependencies?

basvandijk · 2020-02-19T18:32:27Z

I wonder if it would be nicer to keep the generated files all in

I opted for keeping the .cabal, default.nix and generate.nix in the same directory so folks don't have to search for these files.

Maybe add a section in Building.md about Updating Haskell dependencies?

Good idea. Done.

Replace IFD calls to callCabal2nix with importing generated `default.nix` files. The generated `default.nix` files are stored in the repository and contain instructions on how to regenerate them. The generation of the `default.nix` files is done using the `haskellSrc2nixWithDoc` function which generates a directory containing a `default.nix` which is the result of running `cabal2nix` with the `extraCabal2nixOptions` on the provided `src`. A header is then added to the `default.nix` which contains instructions on how to regenerate that file. Finally the new jobs are added which check that the stored `default.nix` matches the expected `default.nix`. So whenever the `.cabal` files are updated and don't match the stored `default.nix` anymore CI will complain.

nomeata · 2020-02-19T23:27:19Z

Ugh, now I get this. Any idea?

$ nix-build -A ic-stub  --option extra-substituters https://nix.dfinity.systems --option trusted-public-keys cache.dfinity.systems-1:IcOn/2SVyPGOi8i3hKhQOlyiSQotiOBKwTFmyPX5YNw=
these derivations will be built:
  /nix/store/fc8szv58sak7f8rnsg39slhy1kn8d9ph-ic-stub-0.1.0.0.drv
building '/nix/store/fc8szv58sak7f8rnsg39slhy1kn8d9ph-ic-stub-0.1.0.0.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/5rk546bp91dqb700kxikw9gnm5xkxh5n-ghc-8.6.5.
unpacking sources
unpacking source archive /nix/store/hsnww9bcwgnzggm4b4l0d78q7pbiycfq-ic-stub
source root is ic-stub
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/tmp/nix-build-ic-stub-0.1.0.0.drv-0/setup-package.conf.d -j4 -threaded
Loaded package environment from /tmp/nix-build-ic-stub-0.1.0.0.drv-0/ic-stub/.ghc.environment.x86_64-linux-8.6.5
[1 of 1] Compiling Main             ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /tmp/nix-build-ic-stub-0.1.0.0.drv-0/Main.o )

/nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs:1:1: error:
    Could not load module ‘Distribution.Simple’
    It is a member of the hidden package ‘Cabal-2.4.0.1’.
    You can run ‘:set -package Cabal’ to expose it.
    (Note: this unloads all the modules in the current scope.)
    Use -v to see a list of the files searched for.
  |
1 | import Distribution.Simple
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^

builder for '/nix/store/fc8szv58sak7f8rnsg39slhy1kn8d9ph-ic-stub-0.1.0.0.drv' failed with exit code 1
error: build of '/nix/store/fc8szv58sak7f8rnsg39slhy1kn8d9ph-ic-stub-0.1.0.0.drv' failed

nomeata · 2020-02-19T23:33:14Z

Ah, it somehow broke the proper use of gitSource…

nomeata · 2020-02-19T23:36:53Z

              src_subst = "./.";

is of course wrong. Will refactor…

the reiterates on #1220. The motivation for the refactoring was that the generated files were not using `gitSource`, so I would get cache misses and dirty files in the repo. Fixing this was easier if all generated nix files are in one place, so I did the following changes: * All generated nix files are in `nix/generated`. This de-pollutes the normal working area with generated files, and keeps them out from sources (more precise derivations). * The local generated files now use `gitSource`. * The generator is now `nix/generate.nix`. * I moved complex logic from `nix/default.nix` to `nix/generate.nix`, this means everything is in one place, instead of spread over several nix files. * There is now only a single check to check if the generated files are upto date, and a single command to update the files. * This check is no longer marked `allowSubstitutes = false`. The check isn’t cheap (it requires `cabal2nix`) and we want the check to happen on builders, not hydra itself, so that the cache has `cabal2nix`. * The check is now included in `all-systems-go`.

basvandijk force-pushed the basvandijk/get-rid-of-IFD branch 3 times, most recently from 3e5a156 to 6c5b5ea Compare February 19, 2020 14:18

basvandijk marked this pull request as ready for review February 19, 2020 14:19

basvandijk requested review from nmattia and nomeata February 19, 2020 15:14

nmattia approved these changes Feb 19, 2020

View reviewed changes

nomeata approved these changes Feb 19, 2020

View reviewed changes

basvandijk added the automerge-squash When ready, merge (using squash) label Feb 19, 2020

basvandijk added 2 commits February 19, 2020 20:31

Building.md: add a note about Updating Haskell Packages

ff94efb

basvandijk force-pushed the basvandijk/get-rid-of-IFD branch from 6b482cc to ff94efb Compare February 19, 2020 19:31

basvandijk merged commit b90fbc5 into master Feb 19, 2020

mergify bot deleted the basvandijk/get-rid-of-IFD branch February 19, 2020 20:20

mergify bot removed the automerge-squash When ready, merge (using squash) label Feb 19, 2020

nomeata mentioned this pull request Feb 19, 2020

Make sure cabal2nix is used also at build time #1200

Closed

nomeata mentioned this pull request Feb 20, 2020

Refactor setup for generated nix files #1223

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nix/haskell-packages: replace IFD with importing generated files#1220

nix/haskell-packages: replace IFD with importing generated files#1220
basvandijk merged 2 commits intomasterfrom
basvandijk/get-rid-of-IFD

basvandijk commented Feb 19, 2020 •

edited

Loading

Uh oh!

nmattia left a comment

Uh oh!

nomeata left a comment

Uh oh!

basvandijk commented Feb 19, 2020

Uh oh!

nomeata commented Feb 19, 2020

Uh oh!

nomeata commented Feb 19, 2020 •

edited

Loading

Uh oh!

nomeata commented Feb 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

basvandijk commented Feb 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nmattia left a comment

Choose a reason for hiding this comment

Uh oh!

nomeata left a comment

Choose a reason for hiding this comment

Uh oh!

basvandijk commented Feb 19, 2020

Uh oh!

nomeata commented Feb 19, 2020

Uh oh!

nomeata commented Feb 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nomeata commented Feb 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

basvandijk commented Feb 19, 2020 •

edited

Loading

nomeata commented Feb 19, 2020 •

edited

Loading