multiprocessing.pool.AsyncResult.get() messes up exceptions

[nikratio]

BPO	9400
Nosy	@amauryfa, @schlamar, @dan-blanchard, @applio
Files	bug.py: Test Case issue9400.diff: patch against trunk picklable_process_exception.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/applio'
closed_at = <Date 2016-09-09.20:19:57.831>
created_at = <Date 2010-07-28.16:03:40.739>
labels = ['type-bug', 'library']
title = 'multiprocessing.pool.AsyncResult.get() messes up exceptions'
updated_at = <Date 2019-01-08.14:41:38.117>
user = 'https://bugs.python.org/nikratio'

bugs.python.org fields:

activity = <Date 2019-01-08.14:41:38.117>
actor = 'schlamar'
assignee = 'davin'
closed = True
closed_date = <Date 2016-09-09.20:19:57.831>
closer = 'davin'
components = ['Library (Lib)']
creation = <Date 2010-07-28.16:03:40.739>
creator = 'nikratio'
dependencies = []
files = ['18233', '18309', '25111']
hgrepos = []
issue_num = 9400
keywords = ['patch']
message_count = 18.0
messages = ['111827', '112118', '112150', '112158', '112329', '157433', '157444', '159733', '159752', '159799', '159804', '176431', '274188', '274630', '274636', '274638', '275423', '333232']
nosy_count = 13.0
nosy_names = ['amaury.forgeotdarc', 'bronger', 'jnoller', 'nikratio', 'ysj.ray', 'python-dev', 'schlamar', 'sbt', 'Daniel.Blanchard', 'mdengler', 'viraptor', 'davin', 'Doug Coleman']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9400'
versions = ['Python 2.7']

[nikratio]

The attached test program calls apply_async with a function that will raise CalledProcessError. However, when result.get() is called, it raises a TypeError and the program hangs:

$ ./bug.py
ERROR:root:ops
Traceback (most recent call last):
  File "./bug.py", line 19, in run_with
    dW1 = run_dcon()
  File "./bug.py", line 26, in run_dcon
    subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
  File "/usr/lib/python2.6/subprocess.py", line 498, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['dcon']' returned non-zero exit status 127
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())

[ysjray]

I got "OSError: [Errno 2] No such file or directory" instead of TypeError when running the "bug.py", did I miss something?

[jnoller]

@ray - you probably don't have the "dcon" binary on your path. bug.py calls a subprocess call.

[nikratio]

@ray: Try it with the following dummy dcon program:

$ cat dcon 
#!/bin/sh

exit 127

(and change the path to dcon in bug.py accordingly).

[ysjray]

This is because when an subprocess.CalledProcessError is raised, the CalledProcessError instance is picked through a socket and then read by "parent" process, but in fact CalledProcessError instances cannot be picked correctly. Because CalledProcessError extends PyExc_BaseException, which defines a __reduce__ method, that special method cause the pickle load to call the exception type's __init__ method with packed self.args as arguments. So if a subclass of "Exception" needs to behave correctly in pickling, it should make self.args meats its __init__ method's function signature. That is, ensure the calling to:
self.__init__(*self.args)
has no problem.

But CalledProcessError doesn't meat this requirement. Here is a patch fixing the CalledProcessError.__init__ to call its base class's __init__ to store its arguments in self.args, which can fix this bug.

[dan-blanchard]

I believe I'm still encountering this issue in 2.7:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/opt/python/2.7/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())

[amauryfa]

Here is another patch, with test.

[dan-blanchard]

The patch appears to fix the issue, so is there any chance of this actually getting accepted this time? It seems bizarre that such a simple bug that has been on the books for almost two years now can't get a patch accepted.

[sbt]

There are plenty of other "bad" exception classes apart from CalledProcessError, including TimeoutExpired in the same file. In fact I suspect this is true of the majority of the exception classes in the stdlib which override __init__. So I am not sure how much good it would do to fix just one example.

Python 3.x's Pool wraps bad exception instances in a MaybeEncodingError class which at least lets you see a stringification of the original exception. I am not sure whether you would want to see a backport of this. Even though 2.7 is in bug fix mode, I think a backport would still be appropriate since it stops a pickling error from killing a worker process, causing a hang.

[python-dev]

New changeset 26bbff4562a7 by Richard Oudkerk in branch '2.7':
Issue bpo-9400: Partial backport of fix for bpo-9244
http://hg.python.org/cpython/rev/26bbff4562a7

[sbt]

I have backported the fix for issue bpo-9244 to 2.7. This should fix the hang and produce a traceback containing a representation of the original error.

[sbt]

I am reopening this issue because 26bbff4562a7 only dealt with objects which cannot be pickled. But CalledProcessError instances *can* be pickled: the problem is that the resulting data cannot be unpickled.

Note that in Python 3.3 CalledProcessError can be pickled then unpickled because of the fix for bpo-1692335. But there are other types which will still produce "corrupt" pickles.