Memory leak when reading s3 object

[ozturkberkay]

Describe the bug

Using get_object function and reading the Body is causing a memory leak.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The memory should be freed after the object body is not referenced anymore.

Current Behavior

RSS does not go down unless the script runs idle for 40 minutes (see the graph below).
Deleting the python objects, calling gc.collect(), or closing the boto connection are not fixing the problem.

Reproduction Steps

import os

from botocore.config import Config

import boto3

BUCKET = "bucket"
KEY = "key"  # +5 MiB size

session = boto3.session.Session(
    aws_access_key_id="key",
    aws_secret_access_key="secret",
    region_name="region",
)
s3_client = session.client(
    service_name="s3",
    endpoint_url="http://localhost:9000/",
    config=Config(tcp_keepalive=False, max_pool_connections=0),
)


def run():
    for _ in range(50):
        resp = s3_client.get_object(Bucket=BUCKET, Key=KEY)
        with resp["Body"] as stream_resp:
            stream_resp.read()


if __name__ == "__main__":
    run()

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.35.58

Environment details (OS name and version, etc.)

Mac OS 15.1 (24B83), Python 3.11.9

[adev-code]

Hey @ozturkberkay, thanks for reaching out. I have replicated the same steps and able to get the same graph. Observing the graphs you have provided and from my side, there's no indication of memory leak. This is because the graph stays flat and constant and if there was a leak, then the graph would have continuously increased. The way Python's memory allocation works is it will defer memory allocation until absolutely needed by Python's memory manager. Once that's done for large objects, it will generally hold memory, even if the objects are freed for future use. This leaves the blocks free for Python to use and won't return them to the OS. Please let me know if you have any questions. Thank you.

[ozturkberkay]

@adev-code Thanks for taking a look. I just wanted to understand where the overhead is coming from and how can I make sure it is deallocated once not needed. Compare this to reading from a file, you will not observe such behavior.

You can close the issue.