Memory leak in if mp4 container not explicitely closed

[hmaarrfk]

Overview

I think there is a memory leak that occurs if you don't explicitly close a container or stream.

I'm still trying to drill the problem down, but I think I have a minimum reproducing example that I think is worthwhile to share at this stage.

It seems that __dealloc__ isn't called as expected maybe??? https://github.com/PyAV-Org/PyAV/blob/main/av/container/input.pyx#L88

import av
import numpy as np
from tqdm import tqdm
# from tqdm.notebook import tqdm

data = np.ones((256, 256, 3), dtype='uint8')

filename = f"video.mp4"
container = av.open(filename, mode="w")
stream = container.add_stream('libopenh264', rate=30)
stream.height = data.shape[0]
stream.width = data.shape[1]
stream.pix_fmt = "yuv420p"

for j in tqdm(range(100), leave=False):
    frame = av.VideoFrame.from_ndarray(data)
    for packet in stream.encode(frame):
        container.mux(packet)

stream.close()
container.close()

import psutil
import os
virtual_memory = []
memory_used = []

for j in tqdm(range(10000)):
    filename = f"video_{i}.mp4"
    container = av.open(filename, mode="r")
    stream = container.streams.video[0]
    packet_generator = container.demux(stream)
    for packet in packet_generator:
        stream.decode(packet)
    process = psutil.Process(os.getpid())
    memory_info = process.memory_info()
    memory_used.append(memory_info.rss)
    virtual_memory.append(memory_info.vms)
    # Explicitly call close to reduce the memory leak.
    # stream.close()
    # container.close()

import numpy as np
from matplotlib import pyplot as plt
m_used = np.asarray(memory_used)
v_used = np.asarray(virtual_memory)
plt.loglog(np.arange(1, len(m_used)), m_used[1:] - m_used[0], label="Memory Used")
plt.ylabel("Memory Usage (Bytes)")
plt.xlabel("Iteration (count)")
# plt.loglog(np.arange(1, len(m_used) + 1), v_used, label="Virtual Used")
plt.legend()

Expected behavior

That the memory be cleared. If I add the close calls to the loop I get.

Versions

• OS: Ubuntu 22.10 + conda + conda-forge
• PyAV runtime:

python -m av --version
PyAV v10.0.0.post2
library configuration: --prefix=/home/mark/mambaforge/envs/mcam_dev --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-cc --cxx=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-c++ --nm=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-nm --ar=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-ar --disable-doc --disable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libfontconfig --enable-libopenh264 --enable-gnutls --enable-libmp3lame --enable-libvpx --enable-pthreads --enable-vaapi --disable-gpl --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libopus --pkg-config=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/pkg-config
library license: LGPL version 3 or later
libavcodec     59. 37.100
libavdevice    59.  7.100
libavfilter     8. 44.100
libavformat    59. 27.100
libavutil      57. 28.100
libswresample   4.  7.100
libswscale      6.  7.100

• FFmpeg:

ffmpeg version 5.1.2 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11.3.0 (conda-forge gcc 11.3.0-19)
configuration: --prefix=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-cc --cxx=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-c++ --nm=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-nm --ar=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/x86_64-conda-linux-gnu-ar --disable-doc --disable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libfontconfig --enable-libopenh264 --enable-gnutls --enable-libmp3lame --enable-libvpx --enable-pthreads --enable-vaapi --disable-gpl --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libopus --pkg-config=/home/conda/feedstock_root/build_artifacts/ffmpeg_1674566195805/_build_env/bin/pkg-config
libavutil      57. 28.100 / 57. 28.100
libavcodec     59. 37.100 / 59. 37.100
libavformat    59. 27.100 / 59. 27.100
libavdevice    59.  7.100 / 59.  7.100
libavfilter     8. 44.100 /  8. 44.100
libswscale      6.  7.100 /  6.  7.100
libswresample   4.  7.100 /  4.  7.100

My PyAV builds include the patch in #1061 those in master. I really don't think it is related, but if you want, I'll run my plotting code again.

Research

I have done the following:

• Checked the PyAV documentation
• Searched on Google
• Searched on Stack Overflow
• Looked through old GitHub issues
• Asked on PyAV Gitter
• ... and waited 72 hours for a response.

Additional context

Maybe related to:

• OutputContainer.__del__ is never called. #427
• the memory leak is still exists even the latest version 8.0.3 because the InputContainer.close() is not work #838

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hmaarrfk]

No stale

[bfreskura]

hi @hmaarrfk

Have you found any other solution besides explicitly closing containers and streams?

This is my current code that encodes and decodes from the list of numpy array images (imgs_padded). It still produces memory leaks:

 with io.BytesIO() as buf:
            with av.open(buf, "w", format=container_string) as container:
                stream = container.add_stream(codec, rate=rate, options=options)
                stream.height = imgs[0].shape[0]
                stream.width = imgs[0].shape[1]
                stream.pix_fmt = pixel_fmt

                for img in imgs_padded:
                    frame = av.VideoFrame.from_ndarray(img, format="rgb24")
                    frame.pict_type = "NONE"
                    for packet in stream.encode(frame):
                        container.mux(packet)

                # Flush stream
                for packet in stream.encode():
                    container.mux(packet)

                stream.close()

            outputs = []
            with av.open(buf, "r", format=container_string) as video:
                for i, frame in enumerate(video.decode(video=0)):
                    if sequence_length <= i < sequence_length * 2:
                        outputs.append(frame.to_rgb().to_ndarray().astype(np.uint8))

                    if i >= sequence_length * 2:
                        break

                video.streams.video[0].close()

[hmaarrfk]

no, i just explicitly call close.

[bfreskura]

Ok, thanks.

I later found out the memory leak only occurs if I use the libx265 codec. Everything is ok when using libx264, mpeg2video, mpeg1video, libvpx-vp9.

[meakbiyik]

I can also reproduce this, without stream.close() the memory leaks.

It seems to happen when I process the frames in a different process, and stream.close causes everything to hang if I use more than 1 processes. Multiple bugs bundled into one 😅

[RoyaltyLJW]

hi, @meakbiyik
Have you found any solution to deal with the problem?
I find that if i break before all the frame extracted, stream.close causes everything to hang

[meakbiyik]

Hey @RoyaltyLJW, I still use stream.close(), and I was able to fix the deadlock issue by setting the environment variable PYAV_LOGGING=off. Here's that bug: #751

[RoyaltyLJW]

@meakbiyik Thanks a lot. It fix my deadlock issue

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hmaarrfk]

Not stale

[watzmonium]

I also experienced the same issue - when deployed in Docker, not closing the stream causes memory instability. @hmaarrfk you saved my life with this extremely niche bug report.