VOBsub overlay leaks memory after subtitle stream EOF

[Leon]

Summary of the bug:
When overlaying VOBsub (.idx + .sub) onto a video stream using the overlay video filter, ffmpeg starts to use a lot of memory after the subtitle stream has gone EOF.
This behavior seems to be independent of the video codec, I tested it with h264, hevc and vp8, and it happens with all of them.
It doesn't happen with SubRip .srt or MicroDVD .sub subtitles (at least not with the files I tested), so maybe it only happens with bitmap subtitle formats.

Furthermore, this only happens when at least one of shortest=1 or eof_action=endall is used as parameters for the overlay filter.

How to reproduce:
Create a test mkv that will be used as an input for the transcode (lower values for -t suffice, since the subtitles are EOF immediately, but too low values may result in the transcode finishing before a significant increase in memory usage can be observed):

% ffmpeg -f lavfi -i color -f lavfi -i anullsrc=sample_rate=1 -t 5000 -c:v libx264 test.mkv

Download and extract the subtitles I attached to this ticket.
Run the transcode:

% ffmpeg -i test.mkv -i test.idx -map 0:0 -map 0:1 -map -0:0 -filter_complex '[0:0][1:0]overlay=shortest=1' out.mkv

Additional information:

% ffmpeg -version
ffmpeg version N-112854-g44e9cccffa Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13.2.1 (GCC) 20230801
configuration: --prefix=/usr --disable-static --disable-stripping --disable-htmlpages --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
libavutil      58. 32.100 / 58. 32.100
libavcodec     60. 35.100 / 60. 35.100
libavformat    60. 18.100 / 60. 18.100
libavdevice    60.  4.100 / 60.  4.100
libavfilter     9. 13.100 /  9. 13.100
libswscale      7.  6.100 /  7.  6.100
libswresample   4. 13.100 /  4. 13.100
libpostproc    57.  4.100 / 57.  4.100

I also ran the transcode with valgrind massif to try to find out what's allocating that much memory. It seems to be frames of the video stream (maybe they aren't free'd?)
Screenshot of massif visualizer:

You can find the corresponding massif outfile attached to this ticket.

This issue may be related to #6922, as it also uses the overlay filter with eof_action=endall, but I didn't test that.

Please let me know if I can add any additional useful information!
Best Regards,
Leon

[Leon]

VOBsubs for easy reproduction of a memory leak that happens with the overlay filter.

[Leon]

valgrind massif of a memory leak that happens with the overlay filter.

[Elon Musk]

Too old ffmpeg version.

[Leon]

Replying to Elon Musk:

Too old ffmpeg version.

My ffmpeg version was built using the latest commits on the master branch at the time, as shown by ffmpeg —version. Why do you say that it’s too old?

[Elon Musk]

Fixed in a30adf9f96254f4870066c98a6dbf13fc74515a3

[Leon]

Yep, that works. Thanks for the fix!