Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProcessPoolExecutor behaviour on MacOS in interactive environment changed between Python versions #301

Closed
basnijholt opened this issue Mar 5, 2021 · 0 comments · Fixed by #300
Labels

Comments

@basnijholt
Copy link
Member

basnijholt commented Mar 5, 2021

The following code works in an interactive environment (ipython) on MacOS Python 3.6 and Python 3.7 but not Python 3.8 and 3.9:

from concurrent.futures import ProcessPoolExecutor
def peak(x, offset=0):
    a = 0.01
    return x + a**2 / (a**2 + (x - offset)**2)

if __name__ == '__main__':
    ex = ProcessPoolExecutor(max_workers=1)
    fut = ex.submit(peak, 1)
    fut.result()

Which with Python 3.7 and 3.8 raises:

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/basnijholt/miniconda3/lib/python3.8/concurrent/futures/process.py", line 233, in _process_worker
    call_item = call_queue.get(block=True)
  File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'peak' on <module '__main__' (built-in)>
---------------------------------------------------------------------------
BrokenProcessPool                         Traceback (most recent call last)
<ipython-input-2-1f1b9bfb79b2> in <module>
      7     ex = ProcessPoolExecutor(max_workers=1)
      8     fut = ex.submit(peak, 1)
----> 9     fut.result()
     10

~/miniconda3/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

~/miniconda3/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

This means we need to change the default executor to Loky on MacOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant