Skip to content

Disable parallel sphinx builds by default#6539

Merged
mergify[bot] merged 3 commits intoQiskit:mainfrom
mtreinish:disable-parallel-sphinx-ci
Jul 22, 2021
Merged

Disable parallel sphinx builds by default#6539
mergify[bot] merged 3 commits intoQiskit:mainfrom
mtreinish:disable-parallel-sphinx-ci

Conversation

@mtreinish
Copy link
Copy Markdown
Member

@mtreinish mtreinish commented Jun 8, 2021 •
edited
Loading

Summary

This commit disables the use of parallel sphinx by default in CI. This
was originally turned on by default in #4477 to try and improve CI
throughput. While this did improve the docs build time a bit the use of
jupyter-sphinx has proven to not really parallel be safe. There is a
high failure rate caused by jupyter-sphinx racing against itself causing
an error around a zmq address conflict. While the speed boost is useful
in trying to improve CI throughput that benefit is outweighed by the
failure rate.

Details and comments

Fixes #5904

@mtreinish
Disable parallel sphinx builds by default
80d8363 
This commit disables the use of parallel sphinx by default in CI. This
was originally turned on by default in Qiskit#4477 to try and improve CI
throughput. While this did increase the docs build time a bit the use of
jupyter-sphinx has proven to not really parallel be safe. There is a
high failure rate caused by jupyter-sphinx racing against itself causing
an error around a zmq address conflict. While the speed boost is useful
in trying to improve CI throughput that benefit is outweighed by the
failure rate.

Fixes Qiskit#5904
@mtreinish mtreinish requested a review from a team as a code owner June 8, 2021 21:07
@mtreinish mtreinish added the on hold Can not fix yet label Jun 8, 2021
@mtreinish
Copy link
Copy Markdown
Member Author

mtreinish commented Jun 8, 2021

Marking as on hold until CI comes back, I'm not sure how long a docs build will take in CI serially. Hopefully it's not too much longer than a parallel build

@mtreinish
Copy link
Copy Markdown
Member Author

mtreinish commented Jun 8, 2021

Hmm, so it finished in ~40 mins which is definitely slower than with parallel builds, we've been at about the 30min mark for docs builds (parallel sphinx enabled) https://dev.azure.com/qiskit-ci/qiskit-terra/_pipeline/analytics/duration?definitionId=1&contextType=build

I think this is still worth doing to make the jobs more reliable. I think spurious failures cause more lost throughput than our slowest job getting slower.

@mtreinish mtreinish removed the on hold Can not fix yet label Jun 8, 2021
peendebak pushed a commit to peendebak/qiskit-terra that referenced this pull request Jun 10, 2021
@eendebakpt
trigger build, see Qiskit#6539
763c40f
mergify Bot added a commit that referenced this pull request Jun 10, 2021
@peendebak @eendebakpt
@taalexander @mergify
Make limit on waveform amplitude optional (#6345)
ce2f2e7 
* make waveform amplitude limit optional

* change order of Waveform arguments for backwards compatibility

* update black once more

* Update qiskit/pulse/library/waveform.py

Co-authored-by: Thomas Alexander <thomasalexander2718@gmail.com>

* move limit_amplitude from waveform to pulse

* use limit_amplitude in pulses; add tests

* format strings

* fix max line length

* fix line length

* add release notes

* fix reno file

* Update releasenotes/notes/add-option-to-disable-waveform-amplitude-limit-c58e2ec61f6789e6.yaml

Co-authored-by: Thomas Alexander <thomasalexander2718@gmail.com>

* Update releasenotes/notes/add-option-to-disable-waveform-amplitude-limit-c58e2ec61f6789e6.yaml

Co-authored-by: Thomas Alexander <thomasalexander2718@gmail.com>

* Update releasenotes/notes/add-option-to-disable-waveform-amplitude-limit-c58e2ec61f6789e6.yaml

Co-authored-by: Thomas Alexander <thomasalexander2718@gmail.com>

* Remove unnecessary print statement.

* trigger build, see #6539

* trigger build

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Thomas Alexander <thomasalexander2718@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@mtreinish mtreinish added automerge Changelog: None Do not include in the GitHub Release changelog. labels Jul 22, 2021
@mergify mergify Bot merged commit 04563cc into Qiskit:main Jul 22, 2021
@kdk kdk added this to the 0.19 milestone Nov 15, 2021
mtreinish added a commit to mtreinish/qiskit-core that referenced this pull request Jan 6, 2023
@mtreinish
Re-enable parallel sphinx builds by default
831d4f5 
With the recent merge of Qiskit#9346 that stopped using the problematic
jupyter-sphinx plugin we're no long blocked from running sphinx-build in
parallel anymore. We had originally enabled this in Qiskit#4477 to get a speed
boost during docs builds. However, we were forced to revert that in Qiskit#6539
because jupyter-sphinx was unreliable and had a race condition when 2
jupyter-execute blocks were run at the same time (see Qiskit#5904 for more
details). Since we've removed the jupyter-sphinx usage that is no longer
a problem and we can reliably run parallel sphinx builds again. This
commit makes that change and changes the default sphinx-build behavior
when run via tox (as is done via ci) to use all available system cores
to perform a sphinx build. Annecdoteally this provides a nice speed up,
on my local workstation a full docs build went from taking 8min 42
seconds for a serial build to 2min 35sec with a parallel build.
@mtreinish mtreinish mentioned this pull request Jan 6, 2023
Merged
mergify Bot added a commit that referenced this pull request Jan 6, 2023
@mtreinish @mergify
Re-enable parallel sphinx builds by default (#9348)
198be45 
With the recent merge of #9346 that stopped using the problematic
jupyter-sphinx plugin we're no long blocked from running sphinx-build in
parallel anymore. We had originally enabled this in #4477 to get a speed
boost during docs builds. However, we were forced to revert that in #6539
because jupyter-sphinx was unreliable and had a race condition when 2
jupyter-execute blocks were run at the same time (see #5904 for more
details). Since we've removed the jupyter-sphinx usage that is no longer
a problem and we can reliably run parallel sphinx builds again. This
commit makes that change and changes the default sphinx-build behavior
when run via tox (as is done via ci) to use all available system cores
to perform a sphinx build. Annecdoteally this provides a nice speed up,
on my local workstation a full docs build went from taking 8min 42
seconds for a serial build to 2min 35sec with a parallel build.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
king-p3nguin pushed a commit to king-p3nguin/qiskit-terra that referenced this pull request Jan 11, 2023
@mtreinish @mergify @king-p3nguin
Re-enable parallel sphinx builds by default (Qiskit#9348)
654f563 
With the recent merge of Qiskit#9346 that stopped using the problematic
jupyter-sphinx plugin we're no long blocked from running sphinx-build in
parallel anymore. We had originally enabled this in Qiskit#4477 to get a speed
boost during docs builds. However, we were forced to revert that in Qiskit#6539
because jupyter-sphinx was unreliable and had a race condition when 2
jupyter-execute blocks were run at the same time (see Qiskit#5904 for more
details). Since we've removed the jupyter-sphinx usage that is no longer
a problem and we can reliably run parallel sphinx builds again. This
commit makes that change and changes the default sphinx-build behavior
when run via tox (as is done via ci) to use all available system cores
to perform a sphinx build. Annecdoteally this provides a nice speed up,
on my local workstation a full docs build went from taking 8min 42
seconds for a serial build to 2min 35sec with a parallel build.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Cryoris pushed a commit to Cryoris/qiskit-terra that referenced this pull request Jan 12, 2023
@mtreinish @mergify @Cryoris
Re-enable parallel sphinx builds by default (Qiskit#9348)
3c75092 
With the recent merge of Qiskit#9346 that stopped using the problematic
jupyter-sphinx plugin we're no long blocked from running sphinx-build in
parallel anymore. We had originally enabled this in Qiskit#4477 to get a speed
boost during docs builds. However, we were forced to revert that in Qiskit#6539
because jupyter-sphinx was unreliable and had a race condition when 2
jupyter-execute blocks were run at the same time (see Qiskit#5904 for more
details). Since we've removed the jupyter-sphinx usage that is no longer
a problem and we can reliably run parallel sphinx builds again. This
commit makes that change and changes the default sphinx-build behavior
when run via tox (as is done via ci) to use all available system cores
to perform a sphinx build. Annecdoteally this provides a nice speed up,
on my local workstation a full docs build went from taking 8min 42
seconds for a serial build to 2min 35sec with a parallel build.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kdk kdk kdk approved these changes

Assignees

No one assigned

Labels

Changelog: None Do not include in the GitHub Release changelog.

Projects

None yet

Milestone

0.19

Development

Successfully merging this pull request may close these issues.

Run Docs of CI failed due to address collision

2 participants

@mtreinish @kdk