format + overlay OOM after stream EOF

[Leon]

Summary of the bug:
As in #10698, but now using the format filter and overlay with shortest=1, ffmpeg starts using a lot of memory once one stream goes EOF.

How to reproduce:
Create a test mkv that will be used as an input for the transcode (lower values for -t suffice, since the subtitles are EOF immediately, but too low values may result in the transcode finishing before a significant increase in memory usage can be observed):

% ffmpeg -f lavfi -i color -f lavfi -i anullsrc=sample_rate=1 -t 5000 -c:v libx264 test.mkv

Download the subtitles attached to #10698 and extract them.
Run the transcode:

% ffmpeg -i test.mkv -i test.idx -map 0:0 -map 0:1 -map -0:0 -filter_complex 'format=yuv420p[main];[main][1:0]overlay=shortest=1' out.mkv

This also happens with setparams or scale instead of format (and probably more), e.g.

% ffmpeg -i test.mkv -i test.idx -map 0:0 -map 0:1 -map -0:0 -filter_complex 'setparams[main];[main][1:0]overlay=shortest=1' out.mkv

% ffmpeg -i test.mkv -i test.idx -map 0:0 -map 0:1 -map -0:0 -filter_complex 'scale[main];[main][1:0]overlay=shortest=1' out.mkv

Additional information:

% ffmpeg -version
ffmpeg version N-112876-ga30adf9f96 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13.2.1 (GCC) 20230801
configuration: --prefix=/usr --disable-static --disable-stripping --disable-htmlpages --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-opengl --enable-shared --enable-version3 --enable-vulkan
libavutil      58. 32.100 / 58. 32.100
libavcodec     60. 35.100 / 60. 35.100
libavformat    60. 18.100 / 60. 18.100
libavdevice    60.  4.100 / 60.  4.100
libavfilter     9. 14.100 /  9. 14.100
libswscale      7.  6.100 /  7.  6.100
libswresample   4. 13.100 /  4. 13.100
libpostproc    57.  4.100 / 57.  4.100

[Leon]

Sorry for the mixup, I misidentified the cause to be the scale filter at first.

[Leon]

This OOM case isn't limited to format, it also happens with scale or setparams.

[Elon Musk]

Fixed in d9e41ead82263e96ebd14d4d88d6e7f858dd944c

[Leon]

Thanks for the fix!