-
Notifications
You must be signed in to change notification settings - Fork 94
Fix memory leak in AsyncClient #457
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for digging into this! My only question is whether this can be simpler by just shutting down the executor synchronously. It looks like you added the __aenter__
and __aexit__
methods to ensure this autocloses when using it as an AsyncContext?
@@ -64,11 +64,12 @@ def min_version(self, version_str: str) -> bool: | |||
""" | |||
return self.client.min_version(version_str) | |||
|
|||
def close(self): | |||
async def close(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this have to be async? Can we just call the executor.shutdown method synchronously?
This way is more Pythonic. Usually, you open and close clients inside the event loop. If you try to wait for executor shutdown in sync mode it will hang the loop. If you close the thread pool without wait=True, you won't be able to guarantee the graceful shutdown of your application. Here is an example of how it is usually done in other libraries: https://github.com/aio-libs/aiohttp/blob/master/examples/curl.py#L11 |
* Fix memory leak in AsyncClient * Graceful shutdown * fix test * fix tests * fix tests
* Fix memory leak in AsyncClient * Graceful shutdown * fix test * fix tests * fix tests
* Gg/update test jwt (ClickHouse#458) * update test jwt, ensure query params are final * tweak test job * Assume CI "secret" is full JWT * Fix lint * Fix memory leak in AsyncClient (ClickHouse#457) * Fix memory leak in AsyncClient * Graceful shutdown * fix test * fix tests * fix tests * Fix lint (ClickHouse#459) * Exclude 3.8 Aarch64 builds (ClickHouse#460) * Fix lint * Exclude pypy 38 build * Exclude all Python 3.8 builds * Update changelog re Python 3.8 aarch64 wheels * Gg/update test matrix (ClickHouse#464) * Update some tests * Fix lint * Skip JSON buggy test * Fix CI tests with default user (ClickHouse#465) * Replace removal of ; in the loop line with rstrip (ClickHouse#472) * Docker test fixes (ClickHouse#473) * Don't prevent settings that don't change the value * Add docker related config file * Fix typo * Update README.md doc link (ClickHouse#476) * Gg/update test jwt (ClickHouse#458) * update test jwt, ensure query params are final * tweak test job * Assume CI "secret" is full JWT * Fix lint * Fix memory leak in AsyncClient (ClickHouse#457) * Fix memory leak in AsyncClient * Graceful shutdown * fix test * fix tests * fix tests * Fix lint (ClickHouse#459) * Correct typing of create_client(host, username) (ClickHouse#482) The parameters `host` and `username` of `create_client` actually do accept None values, as demonstrated by their default values being `None` and the docstring explaining default behavior when not-set. Correcting these types (by marking as Optional) allows users using `dsn` or default behavior to not see type-checking errors. * Release 0 8 16 (ClickHouse#485) * Check for optional libraries in client methods * Log unexpected http next chunk unexpected * Log unexpected http next chunk unexpected * Updates for 0.8.16 release * Exclude 3.8 Aarch64 builds (ClickHouse#460) * Fix lint * Exclude pypy 38 build * Exclude all Python 3.8 builds * Update changelog re Python 3.8 aarch64 wheels * Gg/update test matrix (ClickHouse#464) * Update some tests * Fix lint * Skip JSON buggy test * Updates for 0.8.17 release (ClickHouse#488) * Updates for 0.8.17 release * Update test matrix * Try to punt on SSL issues * Update TLS test certificates * Fix CI tests with default user (ClickHouse#465) * Add param extra_http_headers to query/command methods (ClickHouse#489) * Add param extra_http_headers to query/command methods * add test, fix dict copy --------- Co-authored-by: Geoff Genz <[email protected]> * Change http_headers to transport settings, add transport settings to async client and insert methods (ClickHouse#490) * wrap sql with text() (ClickHouse#491) * Replace removal of ; in the loop line with rstrip (ClickHouse#472) * Docker test fixes (ClickHouse#473) * Don't prevent settings that don't change the value * Add docker related config file * Fix typo * Update test_dynamic.py for variant and json data types * Update httpclient.py update comments * Update dialect.py * Update client.py * Update test_jwt_auth.py * bring back dbapi, otherwise test fails * diable test_transport_settings in test_client.py * fix the JSON->json data type name issue --------- Co-authored-by: Geoff Genz <[email protected]> Co-authored-by: pufit <[email protected]> Co-authored-by: Sviatoslav Bobryshev <[email protected]> Co-authored-by: Avery Fischer (biggerfisch) <[email protected]> Co-authored-by: Paweł Szczur <[email protected]> Co-authored-by: lakako <[email protected]>
Summary
It's known that ThreadPoolExecutor doesn't deallocate memory without .shutdown being called. Python's GC doesn't stop running thread pools, so the code like
will be leaking memory since each new AsyncClient instance will have a new executor.
Closes #424
Checklist
Delete items not relevant to your PR: