Don't enable multiprocessing by default on Python 3.9#6524
Don't enable multiprocessing by default on Python 3.9#6524mergify[bot] merged 4 commits intoQiskit:mainfrom
Conversation
We've recently been hitting a slew of job timeouts in python 3.9 CI jobs recently. This was originally reported in Qiskit#6188 but the frequency of these timeouts have spiked since Qiskit#6270 merged which added sympy to the default mulitprocessing path (for pickling symengine expressions by first converting it to a sympy expression). Trying to debug this locally is tricky but when I've reproduced it seems to be just hanging indefinitely on pickling. I think whatever is causing the issue as reported in Qiskit#6188 was just made worse by adding a sympy import to the pickling process (assuming there is a parameter expression in the circuit). To workaround this issue until we can come up with a solution for whatever is causing things to hang this commit disables multiprocessing on python 3.9 by default (it can still be enabled by users if they desire.
|
I'm personally not happy with this solution because it will have noticeable performance impact on linux python 3.9 when transpiling >1 circuit. But I spent a bit too much time trying to come up with another workaround and I haven't been able to come up with anything else except for disabling multiprocessing on my local 3.9 environment. I'll continue to experiment and try to come up with an alternative but to make CI more reliable I feel like we should do this for the time being. We can always revert it if we have a better solution. |
kdk
left a comment
There was a problem hiding this comment.
Same feeling here, but agree this is needed at least temporarily to unblock CI. With parallel already disabled for windows and mac python >= 3.8, this was already a problem in need of a better solution for a substantial number of users.
Summary
We've recently been hitting a slew of job timeouts in python 3.9 CI jobs
recently. This was originally reported in #6188 but the frequency of
these timeouts have spiked since #6270 merged which added sympy to the
default mulitprocessing path (for pickling symengine expressions by
first converting it to a sympy expression). Trying to debug this locally
is tricky but when I've reproduced it seems to be just hanging
indefinitely on pickling. I think whatever is causing the issue as
reported in #6188 was just made worse by adding a sympy import to the
pickling process (assuming there is a parameter expression in the
circuit). To workaround this issue until we can come up with a solution
for whatever is causing things to hang this commit disables
multiprocessing on python 3.9 by default (it can still be enabled by
users if they desire.
Details and comments