haskell: tune generic-builder for more parallel build performance#86948
Merged
peti merged 1 commit intoNixOS:haskell-updatesfrom May 7, 2020
Merged
haskell: tune generic-builder for more parallel build performance#86948peti merged 1 commit intoNixOS:haskell-updatesfrom
peti merged 1 commit intoNixOS:haskell-updatesfrom
Conversation
peti
added
the
6.topic: haskell
nh2
reviewed
May 5, 2020
Contributor
nh2
left a comment
nh2
left a comment
There was a problem hiding this comment.
This looks sensible to me, but one question:
If it is done at the ghcWithPackages level, do/can/will we also have the ability to override it per Haskell package? I am suspecting that there might be certain large packages for which we may need to tune down -j or -A to not consume too much memory on builders that have less than the casual 3 TB RAM.
3 tasks
cdepillabout
reviewed
May 6, 2020
Member
Author
|
It just occurred to me that the change to the ghcWithPackages wrapper is unnecessary, because you can do the same thing in there has the same effect -- at least if you're building with cabal-install. I'm pretty sure an identical mechanism exists for |
peti
force-pushed
the
t/parallel-ghc-builds
branch
from
9c4ac9c to
7255d30
Compare
peti
changed the title
ofborg
bot
added
10.rebuild-darwin: 501+
Raise the maximum number of cores used for parallel building from 4 to 16. Increase the size of the allocation area for GHC's garbage collector from 1 MiB to 64 MiB. See https://www.twitch.tv/videos/611899011 for the motivation for this change.
peti
force-pushed
the
t/parallel-ghc-builds
branch
from
7255d30 to
f6df169
Compare
cdepillabout
approved these changes
May 7, 2020
Member
Author
|
I'll merge this change to |
Contributor
|
This change causes an uncomfortably high load on my poor machine.
I've also noticed loads of 120+ on my 8(16) core desktop, surely this can't be faster. (sorry if I posted this before somewhere, I thought I did for another machine but couldn't find anything) |
2 tasks
wolfgangwalther
added a commit
to wolfgangwalther/nixpkgs
that referenced
this pull request
Jun 10, 2024
Those flags were not actually passed to GHC before, but to Setup.hs. They were introduced in NixOS#86948. The related twitch live stream uses the build of git-annex as a measurement. I get the following numbers when building git-annex with doCheck = false: - for current master: 1:40 wall clock / 340s user - without any -A64M argument: 1:40 wall clock / 340s user - with this fix: 1:13 wall clock / 280s user The idea was good, but the settings were never active. More testing revealed that this seems to work on darwin just as well, so we're removing the isLinux condition, too.
sternenseemann
pushed a commit
that referenced
this pull request
Jun 10, 2024
Those flags were not actually passed to GHC before, but to Setup.hs. They were introduced in #86948. The related twitch live stream uses the build of git-annex as a measurement. I get the following numbers when building git-annex with doCheck = false: - for current master: 1:40 wall clock / 340s user - without any -A64M argument: 1:40 wall clock / 340s user - with this fix: 1:13 wall clock / 280s user The idea was good, but the settings were never active. More testing revealed that this seems to work on darwin just as well, so we're removing the isLinux condition, too.
github-actions bot
pushed a commit
that referenced
this pull request
Jun 20, 2024
Those flags were not actually passed to GHC before, but to Setup.hs. They were introduced in #86948. The related twitch live stream uses the build of git-annex as a measurement. I get the following numbers when building git-annex with doCheck = false: - for current master: 1:40 wall clock / 340s user - without any -A64M argument: 1:40 wall clock / 340s user - with this fix: 1:13 wall clock / 280s user The idea was good, but the settings were never active. More testing revealed that this seems to work on darwin just as well, so we're removing the isLinux condition, too. (cherry picked from commit 3286300)
sternenseemann
pushed a commit
that referenced
this pull request
Jun 21, 2024
Those flags were not actually passed to GHC before, but to Setup.hs. They were introduced in #86948. The related twitch live stream uses the build of git-annex as a measurement. I get the following numbers when building git-annex with doCheck = false: - for current master: 1:40 wall clock / 340s user - without any -A64M argument: 1:40 wall clock / 340s user - with this fix: 1:13 wall clock / 280s user The idea was good, but the settings were never active. More testing revealed that this seems to work on darwin just as well, so we're removing the isLinux condition, too. (cherry picked from commit 3286300)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Raise the maximum number of cores used for parallel building from 4 to 16. Increase the size of the allocation area for GHC's garbage collector from 1 MiB to 64 MiB.
See https://www.twitch.tv/videos/611899011 for the motivation for this change.