Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Cleanup ThreadPool with atexit rather than __del__ #5094

Merged
merged 6 commits into from
Jan 29, 2020

Conversation

fabianvf
Copy link
Contributor

@fabianvf fabianvf commented Jan 23, 2020

This removes the __del__ function from the generated Python client,
and replaces it with a cleanup function. When a ThreadPool is created,
the cleanup function is registered with the atexit module.

This fixes #5093, where the API client could hang indefinitely at
garbage collection.

Related to swagger-api/swagger-codegen#10002

PR checklist

  • Read the contribution guidelines.
  • If contributing template-only or documentation-only changes which will change sample output, build the project before.
  • Run the shell script(s) under ./bin/ (or Windows batch scripts under.\bin\windows) to update Petstore samples related to your fix. This is important, as CI jobs will verify all generator outputs of your HEAD commit, and these must match the expectations made by your contribution. You only need to run ./bin/{LANG}-petstore.sh, ./bin/openapi3/{LANG}-petstore.sh if updating the code or mustache templates for a language ({LANG}) (e.g. php, ruby, python, etc).
  • File the PR against the correct branch: master, 4.3.x, 5.0.x. Default: master.
  • Copy the technical committee to review the pull request if your PR is targeting a particular programming language.

cc @taxpon @frol @mbohlool @cbornet @kenjones-cisco @tomplus @Jyhess @slash-arun @spacether

@auto-labeler
Copy link

auto-labeler bot commented Jan 23, 2020

👍 Thanks for opening this issue!
🏷 I have applied any labels matching special text in your issue.

The team will review the labels and make any necessary changes.

@spacether
Copy link
Contributor

Thank you for this PR. Can you add a test verifying that the garbage collection is cleaning up the threadpool?

@fabianvf
Copy link
Contributor Author

@spacether hmm, probably, though I'm not totally sure how off the top of my head. I'll dig into it a bit, did you have an approach in mind?

@spacether
Copy link
Contributor

@gaborbernat
Copy link

Thank you for this PR. Can you add a test verifying that the garbage collection is cleaning up the threadpool?

No, please don't. This is not supported in CPython. It's a programming paradigm that might work in other languages, but not CPython. The whole point of this PR is that thread pools should not be handled within a garbage collector as that puts you in undefined space. Remember in Python it's not guaranteed the garbage collector will ever be called. If the user decided that they don't manually close down the threadpool the only fallback is to do it via atexit as the final line of defence. In case you have resources though it's recommended to encourage usage of the objects via the with that guarantees a cleanup operation to be called.

@spacether
Copy link
Contributor

spacether commented Jan 23, 2020

No, please don't. This is not supported in CPython. It's a programming paradigm that might work in other languages, but not CPython. The whole point of this PR is that thread pools should not be handled within a garbage collector

Thank you for letting us know that using the gc library is the wrong way to test this.
My intention is for us to add a test verifying that this is working.

@gaborbernat/@fabianvf what do you think of us also adding __enter__ and __exit__ methods to ApiClient?
That is suggested here: https://stackoverflow.com/questions/865115/how-do-i-correctly-clean-up-a-python-object
And would allow our users to also use with context like

try:
    with petstore_api.PetApi(petstore_api.ApiClient(configuration)) as api_instance:
        body = petstore_api.Pet() # Pet | Pet object that needs to be added to the store
        # Add a new pet to the store
        api_instance.add_pet(body)
except ApiException as e:
    print("Exception when calling PetApi->add_pet: %s\n" % e)

@fabianvf
Can you add a test verifying that the atexit method is working?

Can you also add this feature to the python-experimental generator?

  • In the OpenApi Slack general channel we talked about eventually making python-experimental the primary python generator. Adding helpful fixes like this in both generators would may that future switch easier.

@fabianvf
Copy link
Contributor Author

fabianvf commented Jan 23, 2020

@spacether well I know @gaborbernat is in favor of it, I was worried it would change the PR from a bugfix to a feature and slow it down (we're hitting issues with hanging in the kubernetes python client). I added it in, working on figuring out the test now.

@spacether actually would you mind pointing me to where this sort of test should go? Struggling a bit to find the right place

@fabianvf
Copy link
Contributor Author

Tests added, wasn't totally sure if those are generated tests or not so let me know if I need to put them somewhere else

@spacether
Copy link
Contributor

spacether commented Jan 23, 2020

@fabianvf you put them in the right place, thanks!
Do you think that we should update the docs to use a with context manager for the ApiClient?
Keeping it as is or adding the with both make sense to me.

  • Adding the with encourages the good practice of using the context for ApiClient
  • Keeping it as is: technically we don't need to add the with in the samples because atexit will invoke cleanup when the python program exits

What do you think?

@fabianvf
Copy link
Contributor Author

@spacether probably makes sense to update the docs to encourage the use of the context manager, since there could be longer running programs that create/destroy multiple clients that could still be impacted

@fabianvf fabianvf force-pushed the python3-del-hang branch 2 times, most recently from e185e3d to 32fb9fb Compare January 23, 2020 19:52
if self._pool:
self._pool.close()
self._pool.join()
self._pool = None
if hasattr(atexit, 'unregister'):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like atexit.unregister doesn't exist in Python 2, so I added this guard

This removes the `__del__` function from the generated Python client,
and replaces it with a `cleanup` function. When a ThreadPool is created,
the cleanup function is registered with the `atexit` module.

This fixes OpenAPITools#5093, where the API client could hang indefinitely at
garbage collection.
@spacether
Copy link
Contributor

spacether commented Jan 24, 2020

@fabianvf can you run bin/utils/ensure-up-to-date and update you PR?

  • python-asyncio
  • python-tornado

Need to be updated
That is causing your CircleCi failure

@fabianvf
Copy link
Contributor Author

@spacether thank you, fixed!

@@ -8,22 +8,26 @@ from pprint import pprint
{{#hasAuthMethods}}
Copy link
Contributor

@spacether spacether Jan 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a line above {{#hasAuthMethods}} instantiating configuration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe configuration is instantiated in the {{> python_doc_auth_partial}} snippet

@@ -7,51 +7,55 @@ from pprint import pprint
{{#hasAuthMethods}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a line above {{#hasAuthMethods}} instantiating configuration?

api_instance = {{{packageName}}}.{{{classname}}}({{{packageName}}}.ApiClient(configuration))
{{#allParams}}{{paramName}} = {{{example}}} # {{{dataType}}} | {{{description}}}{{^required}} (optional){{/required}}{{#defaultValue}} (default to {{{.}}}){{/defaultValue}}
{{/allParams}}
# Enter a context with an instance of the API client
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't these blocks {{#hasAuthMethods}}/{{^hasAuthMethods}} identical except for setting
configuration.host?
Please move the identical block contents underneath the {{/hasAuthMethods}} on line 25

@@ -7,18 +7,20 @@ from pprint import pprint
{{> python_doc_auth_partial}}
# Defining host is optional and default to {{{basePath}}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to instantiate configuration here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe configuration is instantiated in the {{> python_doc_auth_partial}} snippet

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually it looks like that snippet doesn't even exit in the experimental directory...

# Enter a context with an instance of the API client
with {{{packageName}}}.ApiClient(configuration) as api_client:
# Create an instance of the API class
api_instance = {{{packageName}}}.{{{classname}}}(api_client)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't these blocks {{#hasAuthMethods}}/{{^hasAuthMethods}} identical except for setting
configuration.host?
Please move the identical block contents underneath the {{/hasAuthMethods}} on line 20

Copy link
Contributor

@spacether spacether left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR. This looks great!
The last requests are non-blocking. This update can be merged in to master.

@spacether spacether merged commit 15345e1 into OpenAPITools:master Jan 29, 2020
@spacether spacether added this to the 4.2.3 milestone Jan 29, 2020
fabianvf added a commit to fabianvf/openapi-generator that referenced this pull request Feb 6, 2020
This is a followup to PR OpenAPITools#5094, which had a few unresolved comments at
merge time. This reduces the amount of redundant lines in the api
example doc templates, and ensures that referenced Configuration objects
are actually instantiated.
spacether pushed a commit that referenced this pull request Feb 6, 2020
* Reduce redundancy in python docs

This is a followup to PR #5094, which had a few unresolved comments at
merge time. This reduces the amount of redundant lines in the api
example doc templates, and ensures that referenced Configuration objects
are actually instantiated.

* Regenerate samples
MikailBag pushed a commit to MikailBag/openapi-generator that referenced this pull request Mar 23, 2020
* Reduce redundancy in python docs

This is a followup to PR OpenAPITools#5094, which had a few unresolved comments at
merge time. This reduces the amount of redundant lines in the api
example doc templates, and ensures that referenced Configuration objects
are actually instantiated.

* Regenerate samples
palnabarun added a commit to palnabarun/python that referenced this pull request Jun 22, 2020
The implementation and tests were already picked up by the upstream OpenAPI
generator [here](OpenAPITools/openapi-generator#5094).
Patching in the tests here for correctness and clarity.

Reference: kubernetes-client#1073

Signed-off-by: Nabarun Pal <[email protected]>
palnabarun added a commit to palnabarun/python that referenced this pull request Jun 22, 2020
The implementation and tests were already picked up by the upstream OpenAPI
generator [here](OpenAPITools/openapi-generator#5094).
Patching in the tests here for correctness and clarity.

Reference: kubernetes-client#1073

Signed-off-by: Nabarun Pal <[email protected]>
palnabarun added a commit to palnabarun/python that referenced this pull request Jul 16, 2020
The implementation and tests were already picked up by the upstream OpenAPI
generator [here](OpenAPITools/openapi-generator#5094).
Patching in the tests here for correctness and clarity.

Reference: kubernetes-client#1073

Signed-off-by: Nabarun Pal <[email protected]>
palnabarun added a commit to palnabarun/python that referenced this pull request Aug 19, 2020
The implementation and tests were already picked up by the upstream OpenAPI
generator [here](OpenAPITools/openapi-generator#5094).
Patching in the tests here for correctness and clarity.

Reference: kubernetes-client#1073

Signed-off-by: Nabarun Pal <[email protected]>
bh717 pushed a commit to bh717/python-dapp that referenced this pull request Apr 1, 2024
The implementation and tests were already picked up by the upstream OpenAPI
generator [here](OpenAPITools/openapi-generator#5094).
Patching in the tests here for correctness and clarity.

Reference: kubernetes-client/python#1073

Signed-off-by: Nabarun Pal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Python] Generated code is out of contract - application hangs indefinitely
3 participants