Parallel build fixes #984
Draft
Parallel build fixes #984
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Mar 23, 2021
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Aug 19, 2021
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Aug 19, 2021
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit)
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) Output of hashes is now not mandatory
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Change CPUS=16 to -j16 in CircleCI config Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 25, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional
|
@osresearch this would need love. |
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 29, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional 29/11/2021 CircleCI public information changes large->xlarge resource class availibility for free tier, with parallelization up to 30 jobs at a time. Let's play
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 29, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional 29/11/2021 CircleCI public information available states parallelization of up to 30 jobs at a time. Let's play - We first build heads musl-cross-make and persist (passing musl-cross-make into next job) - We then build per coreboot version board with coreboot make statement only and persist (passing musl-cross-make + coreboot's musl-cross buildstack) - We then build per coreboot version board (reusing past build musl-cross-make and coreboot's version musl-cross buildstack)
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Nov 29, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional 29/11/2021 CircleCI public information available states parallelization of up to 30 jobs at a time. Let's play - We first build heads musl-cross-make and persist (passing musl-cross-make into next job) - We then build per coreboot version board with coreboot make statement only and persist (passing musl-cross-make + coreboot's musl-cross buildstack) - We then build per coreboot version board (reusing past build musl-cross-make and coreboot's version musl-cross buildstack) Remove 4.11 boards for the moment to test only build time and parallelization
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Dec 1, 2021
CircleCI: We currently drop coreboot 4.11 builds. - There is a file missing in the builds. Not sure why/how this is happening src/soc/intel/fsp_broadwell_de/romstage/romstage.c:41:10: fatal error: build.h: No such file or directory Example:https://app.circleci.com/pipelines/github/tlaurion/heads/877/workflows/7d0248d2-459c-42ad-b741-8fd56a75d527/jobs/2487 - kgpe-d16_workstation building for all GPUs is unfortunately taking too much time to build (40 minutes). - Not sure why, but it seems that the kernel build paralellization is not working for 4.11 while it works for 4.13 Makefile: Uncomment MAKE_JOBS which passes the number of jobs to numbers cores by default and --max-load of 16 CircleCI: Remove CPUS statement to use Makefile default modules/newt: force build with one make job, otherwise there is a race condition in module which fails randomly expecting build modules. (TODO: FIX) Interestingly, building all coreboot 4.13 boards is happening on a clean commit just above 1h limit. More details: - CircleCI changed job build time to a maximum of 1h each. - CircleCI now permits parallelization of 30 jobs - 6000 build minutes a month. - Still waiting for osresearch/heads CircleCI project to be unlocked (currently not recognized as open source project?!)
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Dec 1, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd linuxboot#984 without cache Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional 29/11/2021 CircleCI public information available states parallelization of up to 30 jobs at a time. Let's play - We first build heads musl-cross-make and persist (passing musl-cross-make into next job) - We then build per coreboot version board with coreboot make statement only and persist (passing musl-cross-make + coreboot's musl-cross buildstack) - We then build per coreboot version board (reusing past build musl-cross-make and coreboot's version musl-cross buildstack) Remove 4.11 boards for the moment to test only build time and parallelization
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Dec 1, 2021
CircleCI: We currently drop coreboot 4.11 builds. - There is a file missing in the builds. Not sure why/how this is happening src/soc/intel/fsp_broadwell_de/romstage/romstage.c:41:10: fatal error: build.h: No such file or directory Example:https://app.circleci.com/pipelines/github/tlaurion/heads/877/workflows/7d0248d2-459c-42ad-b741-8fd56a75d527/jobs/2487 - kgpe-d16_workstation building for all GPUs is unfortunately taking too much time to build (40 minutes). - Not sure why, but it seems that the kernel build paralellization is not working for 4.11 while it works for 4.13 Makefile: Uncomment MAKE_JOBS which passes the number of jobs to numbers cores by default and --max-load of 16 CircleCI: Remove CPUS statement to use Makefile default modules/newt: force build with one make job, otherwise there is a race condition in module which fails randomly expecting build modules. (TODO: FIX) Interestingly, building all coreboot 4.13 boards is happening on a clean commit just above 1h limit. More details: - CircleCI changed job build time to a maximum of 1h each. - CircleCI now permits parallelization of 30 jobs - 6000 build minutes a month. - Still waiting for osresearch/heads CircleCI project to be unlocked (currently not recognized as open source project?!)
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
Dec 1, 2021
…impact on the now working build system prior of mergin linuxboot#1035 for tracibility of linuxboot#935 and linuxboot#984
tlaurion
added a commit
that referenced
this pull request
Dec 4, 2021
…tps://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only- Readd #984 without cache Add kgpe-d16 musl-cross target prior of having kgpe-d16 depend on musl-cross target (To try to have musl-cross step successfull under 1h CircleCI new limit) CircleCI: add a subcommand that can follow a target (to build musl-cross-make now and coreboot version specific musl-cross later) Output of hashes is now optional 29/11/2021 CircleCI public information available states parallelization of up to 30 jobs at a time. Let's play - We first build heads musl-cross-make and persist (passing musl-cross-make into next job) - We then build per coreboot version board with coreboot make statement only and persist (passing musl-cross-make + coreboot's musl-cross buildstack) - We then build per coreboot version board (reusing past build musl-cross-make and coreboot's version musl-cross buildstack) Remove 4.11 boards for the moment to test only build time and parallelization
tlaurion
added a commit
that referenced
this pull request
Dec 4, 2021
CircleCI: We currently drop coreboot 4.11 builds. - There is a file missing in the builds. Not sure why/how this is happening src/soc/intel/fsp_broadwell_de/romstage/romstage.c:41:10: fatal error: build.h: No such file or directory Example:https://app.circleci.com/pipelines/github/tlaurion/heads/877/workflows/7d0248d2-459c-42ad-b741-8fd56a75d527/jobs/2487 - kgpe-d16_workstation building for all GPUs is unfortunately taking too much time to build (40 minutes). - Not sure why, but it seems that the kernel build paralellization is not working for 4.11 while it works for 4.13 Makefile: Uncomment MAKE_JOBS which passes the number of jobs to numbers cores by default and --max-load of 16 CircleCI: Remove CPUS statement to use Makefile default modules/newt: force build with one make job, otherwise there is a race condition in module which fails randomly expecting build modules. (TODO: FIX) Interestingly, building all coreboot 4.13 boards is happening on a clean commit just above 1h limit. More details: - CircleCI changed job build time to a maximum of 1h each. - CircleCI now permits parallelization of 30 jobs - 6000 build minutes a month. - Still waiting for osresearch/heads CircleCI project to be unlocked (currently not recognized as open source project?!)
|
Most of the fixes here but cache were merged under #1015 |
|
As tlaurion@8f9ccae documents, parallelization needed some hacks to work properly, while still imperfect. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This WIP patch addresses #935 : fixes a dependency for the coreboot utilities, removes an unused i386 musl-cross build, and adds a cache of the musl-cross
configurefiles to save time during the cross compiler build.