Enable parallel docs builds and bump reno min version#4477
Merged
mergify[bot] merged 4 commits intoQiskit:masterfrom May 20, 2020
Merged
Enable parallel docs builds and bump reno min version#4477mergify[bot] merged 4 commits intoQiskit:masterfrom
mergify[bot] merged 4 commits intoQiskit:masterfrom
Conversation
In the recent reno release 3.1.0 the sphinx extension config option 'parallel_read_safe' was set to true. [1] This makes it possible to run sphinx builds with reno in parallel without a warning being raised. This commit does just that and adds '-j auto' to the default sphinx build command in the tox docs job. This should hopefully speed up the docs builds in CI (although the degree to which is limited by the available resources in the CI nodes) and for local builds. [1] https://opendev.org/openstack/reno/commit/cc09e3d38c2c1046a529270f133de525c0692809
Reno 3.0.0 dropped support for python 3.5 and setting a minimum version above that is causing the python 3.5 jobs to fail. Reno is only used for docs jobs which run with python 3.7 and we don't need to install it as part of a normal test run. This commit makes the reno test requirement only installed on python > 3.5.
1ucian0
approved these changes
May 20, 2020
Member
1ucian0
left a comment
1ucian0
left a comment
There was a problem hiding this comment.
Thanks a lot for being on top of dependencies release notes!
Member
Author
|
Just circling back to show the results here after this merged
The run time for the docs build went from 900-980 seconds without this commit to 670-730 seconds |
faisaldebouni
pushed a commit
to faisaldebouni/qiskit-terra
that referenced
this pull request
Aug 5, 2020
* Enable parallel docs builds and bump reno min version In the recent reno release 3.1.0 the sphinx extension config option 'parallel_read_safe' was set to true. [1] This makes it possible to run sphinx builds with reno in parallel without a warning being raised. This commit does just that and adds '-j auto' to the default sphinx build command in the tox docs job. This should hopefully speed up the docs builds in CI (although the degree to which is limited by the available resources in the CI nodes) and for local builds. [1] https://opendev.org/openstack/reno/commit/cc09e3d38c2c1046a529270f133de525c0692809 * Don't install reno with 3.5 Reno 3.0.0 dropped support for python 3.5 and setting a minimum version above that is causing the python 3.5 jobs to fail. Reno is only used for docs jobs which run with python 3.7 and we don't need to install it as part of a normal test run. This commit makes the reno test requirement only installed on python > 3.5.
mtreinish
added a commit
to mtreinish/qiskit-core
that referenced
this pull request
Jun 8, 2021
This commit disables the use of parallel sphinx by default in CI. This was originally turned on by default in Qiskit#4477 to try and improve CI throughput. While this did increase the docs build time a bit the use of jupyter-sphinx has proven to not really parallel be safe. There is a high failure rate caused by jupyter-sphinx racing against itself causing an error around a zmq address conflict. While the speed boost is useful in trying to improve CI throughput that benefit is outweighed by the failure rate. Fixes Qiskit#5904
mergify Bot
added a commit
that referenced
this pull request
Jul 22, 2021
This commit disables the use of parallel sphinx by default in CI. This was originally turned on by default in #4477 to try and improve CI throughput. While this did increase the docs build time a bit the use of jupyter-sphinx has proven to not really parallel be safe. There is a high failure rate caused by jupyter-sphinx racing against itself causing an error around a zmq address conflict. While the speed boost is useful in trying to improve CI throughput that benefit is outweighed by the failure rate. Fixes #5904 Co-authored-by: Kevin Krsulich <kevin.krsulich@ibm.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
mtreinish
added a commit
to mtreinish/qiskit-core
that referenced
this pull request
Jan 6, 2023
With the recent merge of Qiskit#9346 that stopped using the problematic jupyter-sphinx plugin we're no long blocked from running sphinx-build in parallel anymore. We had originally enabled this in Qiskit#4477 to get a speed boost during docs builds. However, we were forced to revert that in Qiskit#6539 because jupyter-sphinx was unreliable and had a race condition when 2 jupyter-execute blocks were run at the same time (see Qiskit#5904 for more details). Since we've removed the jupyter-sphinx usage that is no longer a problem and we can reliably run parallel sphinx builds again. This commit makes that change and changes the default sphinx-build behavior when run via tox (as is done via ci) to use all available system cores to perform a sphinx build. Annecdoteally this provides a nice speed up, on my local workstation a full docs build went from taking 8min 42 seconds for a serial build to 2min 35sec with a parallel build.
mergify Bot
added a commit
that referenced
this pull request
Jan 6, 2023
With the recent merge of #9346 that stopped using the problematic jupyter-sphinx plugin we're no long blocked from running sphinx-build in parallel anymore. We had originally enabled this in #4477 to get a speed boost during docs builds. However, we were forced to revert that in #6539 because jupyter-sphinx was unreliable and had a race condition when 2 jupyter-execute blocks were run at the same time (see #5904 for more details). Since we've removed the jupyter-sphinx usage that is no longer a problem and we can reliably run parallel sphinx builds again. This commit makes that change and changes the default sphinx-build behavior when run via tox (as is done via ci) to use all available system cores to perform a sphinx build. Annecdoteally this provides a nice speed up, on my local workstation a full docs build went from taking 8min 42 seconds for a serial build to 2min 35sec with a parallel build. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
king-p3nguin
pushed a commit
to king-p3nguin/qiskit-terra
that referenced
this pull request
Jan 11, 2023
With the recent merge of Qiskit#9346 that stopped using the problematic jupyter-sphinx plugin we're no long blocked from running sphinx-build in parallel anymore. We had originally enabled this in Qiskit#4477 to get a speed boost during docs builds. However, we were forced to revert that in Qiskit#6539 because jupyter-sphinx was unreliable and had a race condition when 2 jupyter-execute blocks were run at the same time (see Qiskit#5904 for more details). Since we've removed the jupyter-sphinx usage that is no longer a problem and we can reliably run parallel sphinx builds again. This commit makes that change and changes the default sphinx-build behavior when run via tox (as is done via ci) to use all available system cores to perform a sphinx build. Annecdoteally this provides a nice speed up, on my local workstation a full docs build went from taking 8min 42 seconds for a serial build to 2min 35sec with a parallel build. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Cryoris
pushed a commit
to Cryoris/qiskit-terra
that referenced
this pull request
Jan 12, 2023
With the recent merge of Qiskit#9346 that stopped using the problematic jupyter-sphinx plugin we're no long blocked from running sphinx-build in parallel anymore. We had originally enabled this in Qiskit#4477 to get a speed boost during docs builds. However, we were forced to revert that in Qiskit#6539 because jupyter-sphinx was unreliable and had a race condition when 2 jupyter-execute blocks were run at the same time (see Qiskit#5904 for more details). Since we've removed the jupyter-sphinx usage that is no longer a problem and we can reliably run parallel sphinx builds again. This commit makes that change and changes the default sphinx-build behavior when run via tox (as is done via ci) to use all available system cores to perform a sphinx build. Annecdoteally this provides a nice speed up, on my local workstation a full docs build went from taking 8min 42 seconds for a serial build to 2min 35sec with a parallel build. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In the recent reno release 3.1.0 the sphinx extension config option
'parallel_read_safe' was set to true. [1] This makes it possible to run
sphinx builds with reno in parallel without a warning being raised. This
commit does just that and adds '-j auto' to the default sphinx build
command in the tox docs job. This should hopefully speed up the docs
builds in CI (although the degree to which is limited by the available
resources in the CI nodes) and for local builds.
Details and comments
[1] https://opendev.org/openstack/reno/commit/cc09e3d38c2c1046a529270f133de525c0692809