[Python] Bug async api call producing "RuntimeError: cannot join current thread"

[coin8086]

Description

There's an Python program making some asynchronous API calls by passing async=True to the generated default API methods. Then an exception was thrown at the end of program(no such error if no async calls):

Exception ignored in: <bound method ApiClient.__del__ of <hpc_acm.api_client.ApiClient object at 0x7f5998219b70>>
Traceback (most recent call last):
  File "/home/.../hpc_acm/api_client.py", line 79, in __del__
    self.pool.join()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 511, in join
    self._task_handler.join()
  File "/usr/lib/python3.5/threading.py", line 1051, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread

openapi-generator version

3.1.2

OpenAPI declaration file content or url

https://raw.githubusercontent.com/EvanCui/hpc-acm/master/src/Swagger/swagger.yaml

Command line used for generation

java -jar ~/openapi-generator-cli.jar -l python -i swagger.yaml -o generated/

Steps to reproduce

I tried to reproduce the problem with a minimum code sample but haven't got it. But the debugging output clearly shows the problem rooted in src/main/resources/python/api_client.mustache

# The following lines are for my debug output
from multiprocessing import util
util.DEFAULT_LOGGING_FORMAT = '[%(levelname)s/%(threadName)s:%(thread)d] %(message)s'
util.log_to_stderr(util.DEBUG)

def __del__(self): 
    # The following two lines are for my debug output
    from multiprocessing import util
    util.debug('Api client is closing!')
    self.pool.close() 
    self.pool.join() 

From the following debug output you can see del was executed on a thread which tried to join itself later, while in my minimum code sample it's on the main thread and then everything is OK. So we can assume that del is called by Python GC and can be run on a random thread besides the main thread.

[DEBUG/MainThread:140428243785472] added worker
[DEBUG/MainThread:140428243785472] added worker
...
[INFO/Dummy-6:140428243785472] process shutting down
[DEBUG/Dummy-6:140428243785472] running all "atexit" finalizers with priority >= 0
[DEBUG/Dummy-6:140428243785472] finalizing pool
[DEBUG/Dummy-6:140428243785472] helping task handler/workers to finish
[DEBUG/Dummy-6:140428243785472] joining worker handler
[DEBUG/Thread-1:140428186494720] worker got sentinel -- exiting
[DEBUG/Thread-1:140428186494720] worker exiting after 48 tasks
[DEBUG/Thread-5:140428152923904] result handler found thread._state=TERMINATE
[DEBUG/Thread-5:140428152923904] result handler exiting: len(cache)=0, thread._state=2
[DEBUG/Thread-2:140428178102016] worker got sentinel -- exiting
[DEBUG/Thread-2:140428178102016] worker exiting after 48 tasks
[DEBUG/Thread-3:140428169709312] worker handler exiting
[DEBUG/Thread-4:140428161316608] task handler got sentinel
[DEBUG/Dummy-6:140428243785472] joining task handler
[DEBUG/Thread-4:140428161316608] task handler sending sentinel to result handler
[DEBUG/Thread-4:140428161316608] task handler sending sentinel to workers
[DEBUG/Thread-4:140428161316608] task handler exiting
[DEBUG/Thread-4:140428161316608] Api client is closing! <-- This is my mark!
[DEBUG/Thread-4:140428161316608] closing pool
[DEBUG/Thread-4:140428161316608] joining pool
Exception ignored in: <bound method ApiClient.__del__ of <hpc_acm.api_client.ApiClient object at 0x7fb7fc518748>>
Traceback (most recent call last):
  File "/home/.../api_client.py", line 82, in __del__
    self.pool.join()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 511, in join
    self._task_handler.join()
  File "/usr/lib/python3.5/threading.py", line 1051, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread
[DEBUG/Dummy-6:140428243785472] joining result handler
[DEBUG/Dummy-6:140428243785472] running the remaining "atexit" finalizers

Related issues/PRs

Suggest a fix/enhancement

I suggest either try to ignore the exception, or make a normal "close" method to be called by user manually.

[coin8086]

See also swagger-api/swagger-codegen#8504

[wing328]

@coin8086 thanks for reporting the issue with the details.

cc @taxpon (2017/07) @frol (2017/07) @mbohlool (2017/07) @cbornet (2017/09) @kenjones-cisco (2017/11) to see if they've any clue.

[hyiltiz]

This would be awesome to get merged. My program is now running single process because I'd rather have tqdm running than to have 3 more cores to process data, but is much slower.

[spacether]

@fabianvf will your PR #5094
also resolve this issue?

[fabianvf]

@spacether it should at least allow the user to manually close the thread properly (or use the context manager). It might still happen atexit but I'm not sure.

[mmourafiq]

Curious to know what others are using as workaround.