Concat filter with `-re` (realtime) inputs does not keep realtimeness

[Brad Isbell]

When you use the concat filter to create a sequence of media from inputs, and when those inputs need to be forced to run in realtime with the -re option, only the first input runs in realtime. Subsequent inputs run faster.

Presumably, the subsequent inputs' realtime keeper is looking at the overall timestamp, and assuming it needs to catch up.

An easy way to test this is with a timer (date command, in my case), and a couple lavfi inputs:

date; ffmpeg -re -f lavfi -i "sine=frequency=1000:duration=30" -re -f lavfi -i "sine=frequency=500:duration=30" -filter_complex "[0:a][1:a] concat=n=2:v=0:a=1" -f null NUL; date

Tuesday, February 21, 2023 1:48:17 PM

ffmpeg version 2023-02-19-git-2aec86695a-full_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libvpl --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      58.  3.100 / 58.  3.100
  libavcodec     60.  4.100 / 60.  4.100
  libavformat    60.  4.100 / 60.  4.100
  libavdevice    60.  2.100 / 60.  2.100
  libavfilter     9.  4.100 /  9.  4.100
  libswscale      7.  2.100 /  7.  2.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
Input #0, lavfi, from 'sine=frequency=1000:duration=30':
  Duration: N/A, start: 0.000000, bitrate: 705 kb/s
  Stream #0:0: Audio: pcm_s16le, 44100 Hz, mono, s16, 705 kb/s
Input #1, lavfi, from 'sine=frequency=500:duration=30':
  Duration: N/A, start: 0.000000, bitrate: 705 kb/s
  Stream #1:0: Audio: pcm_s16le, 44100 Hz, mono, s16, 705 kb/s
Stream mapping:
  Stream #0:0 (pcm_s16le) -> concat
  Stream #1:0 (pcm_s16le) -> concat
  concat:out:a0 -> Stream #0:0 (pcm_s16le)
Press [q] to stop, [?] for help
Output #0, null, to 'NUL':
  Metadata:
    encoder         : Lavf60.4.100
  Stream #0:0: Audio: pcm_s16le, 44100 Hz, mono, s16, 705 kb/s
    Metadata:
      encoder         : Lavc60.4.100 pcm_s16le
size=N/A time=00:00:59.97 bitrate=N/A speed=   2x     0x
video:0kB audio:5168kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

Tuesday, February 21, 2023 1:48:47 PM

Note that both inputs were set to run realtime, and were of 30 seconds each, but the FFmpeg run finished in 30 seconds total. Also note that the total audio output time was ~60 seconds, and that the speed was 2x. Since it's supposed to run realtime, it should have taken 60 seconds and ran at 1x.

[Gyan]

It's working as expected. Each file reader (demuxer) runs in its own thread, and populates that input's packet queue asynchronously at 1x in this command. However, the concat filter only consumes each input when its turn comes for appendage. Since both inputs are of the same duration and readrate limited to the same speed, the 2nd input has been fully ingested into memory when concat gets to it.

Use the realtime filter after concat instead to limit output speed.

[Cigaes]

No, it is not normal. If a filter input is not requesting frames, ffmpeg should not be reading on this input. Doing it will saturate the memory. Reopening, setting component to undetermined, it is possible that threadization of demuxers broke things severely.

[Gyan]

This is related to individual threads for get_input_packet which loops av_read_frame and not Demuxer threads that Anton added this past July. I'm able to reproduce this behaviour in 3.4, the earliest build I have, which dates from 2017.

[Cigaes]

Ok, I checked, and the second input thread correctly blocks after a few frames, causing no flood. The effect of the -re option is visible again after the second input has run for longer than the first input. This is the expected behavior, there is nothing to fix.