Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make nix-shell evaluation faster #2300

Merged
merged 8 commits into from
Nov 6, 2020
Merged

Make nix-shell evaluation faster #2300

merged 8 commits into from
Nov 6, 2020

Conversation

rvl
Copy link
Contributor

@rvl rvl commented Nov 5, 2020

Issue Number

ADP-382 / #1888

Overview

  • Updates Haskell.nix rev to latest.
  • Use the "materialization" feature so that evaluation of the nix-shell (done locally and on hydra) is cached as much as possible.

Comments

$ GC_PRINT_STATS=1 command time nix-shell --option substitute false --run "cabal --version"

master branch:
  ...
  Grow heap to 2786488 KiB after 806218912 bytes allocated
  cabal-install version 3.2.0.0
  compiled using version 3.2.0.0 of the Cabal library 
  30.51user 2.34system 0:29.05elapsed 113%CPU (0avgtext+0avgdata 3300584maxresident)k
  0inputs+0outputs (0major+915357minor)pagefaults 0swaps

PR branch:
  ...
  Grow heap to 2393272 KiB after 442027296 bytes allocated
  cabal-install version 3.2.0.0
  compiled using version 3.2.0.0 of the Cabal library 
  26.63user 1.84system 0:24.78elapsed 114%CPU (0avgtext+0avgdata 2889004maxresident)k
  0inputs+0outputs (0major+812718minor)pagefaults 0swaps

memory usage: 2.3GiB  ( -14%)
startup time: 0:24.78 ( -15%)

@rvl rvl added the RESOLVING ISSUE Mark a PR as resolving issues, for auto-generated CHANGELOG label Nov 5, 2020
@rvl rvl self-assigned this Nov 5, 2020
@rvl rvl requested a review from hamishmack November 5, 2020 13:11
shell.nix Outdated
cabal = {
version = "3.2.0.0";
plan-sha256 = "0ik4ws852dk7wchbhc84w5ac149myvc1az7rxapfy0hdmiiwjh2j
";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooops

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, fixed.

@hamishmack
Copy link
Contributor

I am not sure if we still need plan-sha256 values. You might want to try without them as it would probably make updating the materialized nix easier.

@rvl
Copy link
Contributor Author

rvl commented Nov 6, 2020

@hamishmack I tried removing the plan-sha256 lines but was then getting these kind of messages from the nix-shell:

building '/nix/store/hq5bl3bc3f6cps36yaw28dy89czgz68l-hoogle-plan-to-nix-pkgs.drv'...

So perhaps they do help with caching the cabal plan.

@rvl rvl force-pushed the rvl/adp-382/nix-faster branch from bd22df4 to cc6fd10 Compare November 6, 2020 06:18
@hamishmack
Copy link
Contributor

Is that derivation actually running cabal configure though or just copying the files from the materialized folder. Perhaps check nix show-derivation /nix/store/hq5bl3bc3f6cps36yaw28dy89czgz68l-hoogle-plan-to-nix-pkgs.drv. I think it should only be one of these https://github.com/input-output-hk/haskell.nix/blob/b06fdfa06595668b367a8b9298792076c87e4618/lib/materialize.nix#L116-L120.

@rvl
Copy link
Contributor Author

rvl commented Nov 6, 2020

I see, yes you are right, it was just copying files.
I have removed plan-sha256 and cleaned up the update instructions.

@rvl
Copy link
Contributor Author

rvl commented Nov 6, 2020

bors r+

@iohk-bors
Copy link
Contributor

iohk-bors bot commented Nov 6, 2020

Build succeeded:

@iohk-bors iohk-bors bot merged commit 78c2382 into master Nov 6, 2020
@iohk-bors iohk-bors bot deleted the rvl/adp-382/nix-faster branch November 6, 2020 09:50
iohk-bors bot added a commit that referenced this pull request Nov 12, 2020
2301: Reduce nix-shell eval memory usage r=jbgi a=rvl

### Issue Number

ADP-382 / #1888

### Overview

1. Similar memory and speed gains as in #2300. cardano-node was previously imported as a niv source. It is also part of the haskell.nix package set, because it's a stack.yaml dependency. So there were haskell package sets duplicated basically.

2. Add a `platform` argument to the `release.nix` jobset. This will split the jobset into three. With support from Hydra (input-output-hk/ci-ops#26), this will make evaluation use about 1/3 of the memory, and allow jobs to start building quicker.

### Comments

Removing cardano-wallet-jormungandr and nix build of jormungandr is done in #2313. This also reduces the weight to the nix-shell.



Co-authored-by: Rodney Lorrimar <[email protected]>
iohk-bors bot added a commit that referenced this pull request Nov 30, 2020
2301: Reduce nix-shell eval memory usage r=rvl a=rvl

### Issue Number

ADP-382 / #1888

### Overview

1. Similar memory and speed gains as in #2300. cardano-node was previously imported as a niv source. It is also part of the haskell.nix package set, because it's a stack.yaml dependency. So there were haskell package sets duplicated basically.

2. Add a `platform` argument to the `release.nix` jobset. This will split the jobset into three. With support from Hydra (input-output-hk/ci-ops#26), this will make evaluation use about 1/3 of the memory, and allow jobs to start building quicker.

### Comments

- Removing cardano-wallet-jormungandr and nix build of jormungandr is done in #2313. This also reduces the weight to the nix-shell.
- It is best to merge this after #2245, because the iohk-monitoring build will be simpler.


Co-authored-by: Rodney Lorrimar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
RESOLVING ISSUE Mark a PR as resolving issues, for auto-generated CHANGELOG
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants