Exception by __del__

[coin8086]

swagger-codegen/modules/swagger-codegen/src/main/resources/python/api_client.mustache

Lines 72 to 74 in dbd0a4b

	def __del__(self):
	self.pool.close()
	self.pool.join()

The code causes an exception in generated api client, when an async api call is made. Take mine for example:

[DEBUG/MainThread:140428243785472] added worker
[DEBUG/MainThread:140428243785472] added worker
...
[INFO/Dummy-6:140428243785472] process shutting down
[DEBUG/Dummy-6:140428243785472] running all "atexit" finalizers with priority >= 0
[DEBUG/Dummy-6:140428243785472] finalizing pool
[DEBUG/Dummy-6:140428243785472] helping task handler/workers to finish
[DEBUG/Dummy-6:140428243785472] joining worker handler
[DEBUG/Thread-1:140428186494720] worker got sentinel -- exiting
[DEBUG/Thread-1:140428186494720] worker exiting after 48 tasks
[DEBUG/Thread-5:140428152923904] result handler found thread._state=TERMINATE
[DEBUG/Thread-5:140428152923904] result handler exiting: len(cache)=0, thread._state=2
[DEBUG/Thread-2:140428178102016] worker got sentinel -- exiting
[DEBUG/Thread-2:140428178102016] worker exiting after 48 tasks
[DEBUG/Thread-3:140428169709312] worker handler exiting
[DEBUG/Thread-4:140428161316608] task handler got sentinel
[DEBUG/Dummy-6:140428243785472] joining task handler
[DEBUG/Thread-4:140428161316608] task handler sending sentinel to result handler
[DEBUG/Thread-4:140428161316608] task handler sending sentinel to workers
[DEBUG/Thread-4:140428161316608] task handler exiting
[DEBUG/Thread-4:140428161316608] Api client is closing! <-- This is my mark!
[DEBUG/Thread-4:140428161316608] closing pool
[DEBUG/Thread-4:140428161316608] joining pool
Exception ignored in: <bound method ApiClient.__del__ of <hpc_acm.api_client.ApiClient object at 0x7fb7fc518748>>
Traceback (most recent call last):
  File "/home/.../api_client.py", line 82, in __del__
    self.pool.join()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 511, in join
    self._task_handler.join()
  File "/usr/lib/python3.5/threading.py", line 1051, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread
[DEBUG/Dummy-6:140428243785472] joining result handler
[DEBUG/Dummy-6:140428243785472] running the remaining "atexit" finalizers

My marked code is:

    def __del__(self):
        from multiprocessing import util
        util.debug('Api client is closing!')
        self.pool.close()
        self.pool.join()

And this is the code snippet from /usr/lib/python3.5/multiprocessing/pool.py:

555         util.debug('joining task handler')
556         if threading.current_thread() is not task_handler:
557             task_handler.join()
558
559         util.debug('joining result handler')
560         if threading.current_thread() is not result_handler:
561             result_handler.join()

The __del__ method is expected to run in main thread but here clearly Python runs it on a thread which is task_handler. It's the thread that __del__ runs on, and it makes a thread try to join itself and thus throw an exception.

I suggest either try to ignore the exception, or make it a normal "close" method to be called by user manually.

[coin8086]

See also OpenAPITools/openapi-generator#712

[coin8086]

Has anyone looked into the problem I mentioned? It's definitely a problem when used in a non-hello-world program. I've explained it clearly I believe. Maybe I need to make a pull request for the code change?

[mmourafiq]

Seems like this is stlil going, was hit by this as well.

[hmvp]

This is also related to https://bugs.python.org/issue39360 and can lead to weird behaviour (hanging tests in pytest) under 3.8

Edit: that is filed under #9991