-filter_complex doesn't handle well finite video + infinite audio

[Clément Bœsch]

Given a 60 seconds video stream:

./ffmpeg -v 0 -f lavfi -i testsrc=d=60 -y video.mp4

and a 5 seconds audio stream:

./ffmpeg -v 0 -f lavfi -i 'aevalsrc=sin(440*2*PI*t)::d=5' -y audio.wav

We can make a video presentation of 60 seconds with the 5 seconds audio padded with silence with the following command line:

./ffmpeg -f lavfi -i 'movie=video.mp4 [out0]; amovie=audio.wav [audio]; aevalsrc=0 [silence]; [audio][silence] concat=v=0:a=1 [out1]' -y out.avi

Unfortunately, this doesn't work with the -filter_complex option:

% ./ffmpeg -i video.mp4 -i audio.wav -filter_complex 'aevalsrc=0 [silence]; [1:0][silence] concat=v=0:a=1 [padded_audio]' -map 0:0 -map '[padded_audio]' -y out.avi     
ffmpeg version N-43791-g2c7a622 Copyright (c) 2000-2012 the FFmpeg developers
  built on Aug 21 2012 21:01:23 with gcc 4.7.1 (GCC) 20120721 (prerelease)
  configuration: --enable-gpl --enable-fontconfig --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libxvid --enable-libx264 --enable-libvpx --enable-libtheora --enable-x11grab --enable-libopenjpeg --enable-libass --enable-libmodplug --enable-libv4l2 --cc='ccache colorgcc' --samples=/home/ubitux/fate-samples --prefix=/tmp/ffinstall
  libavutil      51. 70.100 / 51. 70.100
  libavcodec     54. 54.100 / 54. 54.100
  libavformat    54. 25.104 / 54. 25.104
  libavdevice    54.  2.100 / 54.  2.100
  libavfilter     3. 13.100 /  3. 13.100
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'video.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf54.25.104
  Duration: 00:01:00.00, start: 0.000000, bitrate: 37 kb/s
    Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 320x240 [SAR 1:1 DAR 4:3], 34 kb/s, 25 fps, 25 tbr, 25 tbn, 50 tbc
    Metadata:
      handler_name    : VideoHandler
Guessed Channel Layout for  Input Stream #1.0 : mono
Input #1, wav, from 'audio.wav':
  Duration: 00:00:05.01, bitrate: 705 kb/s
    Stream #1:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, mono, s16, 705 kb/s
Output #0, avi, to 'out.avi':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    ISFT            : Lavf54.25.104
    Stream #0:0(und): Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 tbn, 25 tbc
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 44100 Hz, mono, s16
Stream mapping:
  Stream #1:0 (pcm_s16le) -> concat:in0:a0 (graph 0)
  Stream #0:0 -> #0:0 (h264 -> mpeg4)
  concat (graph 0) -> Stream #0:1 (libmp3lame)
Press [q] to stop, [?] for help
frame= 1500 fps=177 q=9.6 Lsize=   13088kB time=00:01:00.00 bitrate=1786.9kbits/s    

This creates some kind of infinite loop, with fps slightly decreasing over time (+memory leak)

[Cigaes]

Except for the time that no longer progress (I posted a patch about that yesterday), that is normal and expected, if not desirable.

ffmpeg is still encoding audio silence, and since you privided an infinite quantity of it, it will never stop. The muxer wants to interleave audio and video, but no video is coming, so audio gets queued, thus the memory use growing and not the file. If you quit ffmpeg cleanly (hit q), all the data gets written to the file at once.

There may be ways of enhancing things (maybe some way to signal a stream is finished so that the interleaving does not wait for it), but I do not think there is something fundamentally wrong in ffmpeg's reaction to this hostile situation.

[Elon Musk]

So solution may be to detect such hostile situations and abort muxing.