Skip to content

Commit

Permalink
Fix memory leak
Browse files Browse the repository at this point in the history
Creating a new session each time S3Storage is instantiated creates a memory leak. It seems that S3Storage can get created a bunch of times (I'm seeing it get created again and again as my app runs) and a boto3's Session takes loads of memory (see boto/boto3#1670) so my app eventually runs out of memory. This should fix the issue while still avoiding using the same session across different threads.
  • Loading branch information
almost authored Nov 26, 2021
1 parent 2fd1b66 commit abb72ab
Showing 1 changed file with 9 additions and 3 deletions.
12 changes: 9 additions & 3 deletions django_s3_storage/storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,10 +93,17 @@ def open(self, mode="rb"):
if self.closed:
self.file = self._storage.open(self.name, mode).file
return super(S3File, self).open(mode)


class _LocalSession(local):
def __init__(self):
self.session = boto3.session.Session()


local_session = _LocalSession()


class _Local(local):

"""
Thread-local connection manager.
Expand All @@ -116,8 +123,7 @@ def __init__(self, storage):
connection_kwargs["aws_session_token"] = storage.settings.AWS_SESSION_TOKEN
if storage.settings.AWS_S3_ENDPOINT_URL:
connection_kwargs["endpoint_url"] = storage.settings.AWS_S3_ENDPOINT_URL
self.session = boto3.session.Session()
self.s3_connection = self.session.client("s3", config=Config(
self.s3_connection = local_session.session.client("s3", config=Config(
s3={"addressing_style": storage.settings.AWS_S3_ADDRESSING_STYLE},
signature_version=storage.settings.AWS_S3_SIGNATURE_VERSION,
), **connection_kwargs)
Expand Down

0 comments on commit abb72ab

Please sign in to comment.