ffmpeg memory overflow when trimming 4K clips

[Andrew Kaiser]

Summary of the bug:
I have discovered that trimming inputs on large 4K files can cause ffmpeg to run out of memory computer (which has 16GB of ram). This appears to be a bug with input trimming because the command only overflows memory when the inputs are trimmed in the same command that contains a -filter_complex. For comparison, I have ran the same command twice, where the first case has inputs trimmed and saved as separate files. This does not crash. The second case has a single large input which has its inputs trimmed in the same command that renders the final output. This does crash. The two commands are shown below, and log files as well as memory profile graphs are attached.

The video input in both cases is ​https://www.youtube.com/watch?v=voprky8BrPw which was downloaded with youtube-dl.

The first case (that does not crash):

# create separate inputs beforehand
for i in $(seq 0 9)
do
  echo creating output-${i}.mp4
  ffmpeg -v error -ss $(($i * 20)) -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' ./samples/oom-large-4k-file/output-${i}.mp4 -y
done

# stitch them all together
  ffmpeg \
    -v 9 -loglevel 99 \
    -i './samples/oom-large-4k-file/output-0.mp4' \
    -i './samples/oom-large-4k-file/output-1.mp4' \
    -i './samples/oom-large-4k-file/output-2.mp4' \
    -i './samples/oom-large-4k-file/output-3.mp4' \
    -i './samples/oom-large-4k-file/output-4.mp4' \
    -i './samples/oom-large-4k-file/output-5.mp4' \
    -i './samples/oom-large-4k-file/output-6.mp4' \
    -i './samples/oom-large-4k-file/output-7.mp4' \
    -i './samples/oom-large-4k-file/output-8.mp4' \
    -i './samples/oom-large-4k-file/output-9.mp4' \
    -map '[v_out_9]' -filter_complex 'color=s=3840x2160:color=black:duration=100.00900000000001[base];
      [0:v] setpts=PTS-STARTPTS, scale=3840:2160 [v_in_0];
      [0:a] asetpts=PTS-STARTPTS, adelay=0:all=1, volume=1[a_in_0];
      [1:v] setpts=PTS+10.001/TB, scale=3840:2160 [v_in_1];
      [1:a] asetpts=PTS-STARTPTS, adelay=10001:all=1, volume=1[a_in_1];
      [2:v] setpts=PTS+20.002/TB, scale=3840:2160 [v_in_2];
      [2:a] asetpts=PTS-STARTPTS, adelay=20002:all=1, volume=1[a_in_2];
      [3:v] setpts=PTS+30.003/TB, scale=3840:2160 [v_in_3];
      [3:a] asetpts=PTS-STARTPTS, adelay=30003:all=1, volume=1[a_in_3];
      [4:v] setpts=PTS+40.004/TB, scale=3840:2160 [v_in_4];
      [4:a] asetpts=PTS-STARTPTS, adelay=40004:all=1, volume=1[a_in_4];
      [5:v] setpts=PTS+50.004999999999995/TB, scale=3840:2160 [v_in_5];
      [5:a] asetpts=PTS-STARTPTS, adelay=50004.99999999999:all=1, volume=1[a_in_5];
      [6:v] setpts=PTS+60.00599999999999/TB, scale=3840:2160 [v_in_6];
      [6:a] asetpts=PTS-STARTPTS, adelay=60005.99999999999:all=1, volume=1[a_in_6];
      [7:v] setpts=PTS+70.007/TB, scale=3840:2160 [v_in_7];
      [7:a] asetpts=PTS-STARTPTS, adelay=70007:all=1, volume=1[a_in_7];
      [8:v] setpts=PTS+80.00800000000001/TB, scale=3840:2160 [v_in_8];
      [8:a] asetpts=PTS-STARTPTS, adelay=80008.00000000001:all=1, volume=1[a_in_8];
      [9:v] setpts=PTS+90.00900000000001/TB, scale=3840:2160 [v_in_9];
      [9:a] asetpts=PTS-STARTPTS, adelay=90009.00000000001:all=1, volume=1[a_in_9];
      [base][v_in_0] overlay=x=0:y=0:eof_action=pass [v_out_0];
      [v_out_0][v_in_1] overlay=x=0:y=0:eof_action=pass [v_out_1];
      [v_out_1][v_in_2] overlay=x=0:y=0:eof_action=pass [v_out_2];
      [v_out_2][v_in_3] overlay=x=0:y=0:eof_action=pass [v_out_3];
      [v_out_3][v_in_4] overlay=x=0:y=0:eof_action=pass [v_out_4];
      [v_out_4][v_in_5] overlay=x=0:y=0:eof_action=pass [v_out_5];
      [v_out_5][v_in_6] overlay=x=0:y=0:eof_action=pass [v_out_6];
      [v_out_6][v_in_7] overlay=x=0:y=0:eof_action=pass [v_out_7];
      [v_out_7][v_in_8] overlay=x=0:y=0:eof_action=pass [v_out_8];
      [v_out_8][v_in_9] overlay=x=0:y=0:eof_action=pass [v_out_9];
      [a_in_0][a_in_1][a_in_2][a_in_3][a_in_4][a_in_5][a_in_6][a_in_7][a_in_8][a_in_9] amix=inputs=10 [audio]' \
    -map '[audio]' \
    template.mp4 -y \
    2> ffmpeg-separate-inputs-logs.txt

The second case (which does crash)

  ffmpeg \
    -v 9 -loglevel 99 \
    -ss 0 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 20 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 40 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 60 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 80 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 100 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 120 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 140 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 160 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 180 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -map '[v_out_9]' -filter_complex 'color=s=3840x2160:color=black:duration=100.00900000000001[base];
      [0:v] setpts=PTS-STARTPTS, scale=3840:2160 [v_in_0];
      [0:a] asetpts=PTS-STARTPTS, adelay=0:all=1, volume=1[a_in_0];
      [1:v] setpts=PTS+10.001/TB, scale=3840:2160 [v_in_1];
      [1:a] asetpts=PTS-STARTPTS, adelay=10001:all=1, volume=1[a_in_1];
      [2:v] setpts=PTS+20.002/TB, scale=3840:2160 [v_in_2];
      [2:a] asetpts=PTS-STARTPTS, adelay=20002:all=1, volume=1[a_in_2];
      [3:v] setpts=PTS+30.003/TB, scale=3840:2160 [v_in_3];
      [3:a] asetpts=PTS-STARTPTS, adelay=30003:all=1, volume=1[a_in_3];
      [4:v] setpts=PTS+40.004/TB, scale=3840:2160 [v_in_4];
      [4:a] asetpts=PTS-STARTPTS, adelay=40004:all=1, volume=1[a_in_4];
      [5:v] setpts=PTS+50.004999999999995/TB, scale=3840:2160 [v_in_5];
      [5:a] asetpts=PTS-STARTPTS, adelay=50004.99999999999:all=1, volume=1[a_in_5];
      [6:v] setpts=PTS+60.00599999999999/TB, scale=3840:2160 [v_in_6];
      [6:a] asetpts=PTS-STARTPTS, adelay=60005.99999999999:all=1, volume=1[a_in_6];
      [7:v] setpts=PTS+70.007/TB, scale=3840:2160 [v_in_7];
      [7:a] asetpts=PTS-STARTPTS, adelay=70007:all=1, volume=1[a_in_7];
      [8:v] setpts=PTS+80.00800000000001/TB, scale=3840:2160 [v_in_8];
      [8:a] asetpts=PTS-STARTPTS, adelay=80008.00000000001:all=1, volume=1[a_in_8];
      [9:v] setpts=PTS+90.00900000000001/TB, scale=3840:2160 [v_in_9];
      [9:a] asetpts=PTS-STARTPTS, adelay=90009.00000000001:all=1, volume=1[a_in_9];
      [base][v_in_0] overlay=x=0:y=0:eof_action=pass [v_out_0];
      [v_out_0][v_in_1] overlay=x=0:y=0:eof_action=pass [v_out_1];
      [v_out_1][v_in_2] overlay=x=0:y=0:eof_action=pass [v_out_2];
      [v_out_2][v_in_3] overlay=x=0:y=0:eof_action=pass [v_out_3];
      [v_out_3][v_in_4] overlay=x=0:y=0:eof_action=pass [v_out_4];
      [v_out_4][v_in_5] overlay=x=0:y=0:eof_action=pass [v_out_5];
      [v_out_5][v_in_6] overlay=x=0:y=0:eof_action=pass [v_out_6];
      [v_out_6][v_in_7] overlay=x=0:y=0:eof_action=pass [v_out_7];
      [v_out_7][v_in_8] overlay=x=0:y=0:eof_action=pass [v_out_8];
      [v_out_8][v_in_9] overlay=x=0:y=0:eof_action=pass [v_out_9];
      [a_in_0][a_in_1][a_in_2][a_in_3][a_in_4][a_in_5][a_in_6][a_in_7][a_in_8][a_in_9] amix=inputs=10 [audio]' \
    -map '[audio]' \
    template.mp4 -y \
    2> ffmpeg-trimmed-inputs-logs.txt

[Andrew Kaiser]

ffmpeg log file for the crashing ffmpeg command

[Andrew Kaiser]

ffmpeg memory profile graph for the successful ffmpeg command

[Andrew Kaiser]

ffmpeg memory profile graph for the crashing ffmpeg command

[Andrew Kaiser]

ffmpeg log file for the successful ffmpeg command (it had to be zipped, because it was over the size limit)

[Andrew Kaiser]

These two commands are functionally identical, so four times the memory usage (or more possibly) shown in the memory profile graphs seems like an indicator of a bug.

Apologies, I dont know how to edit a ticket description, so I added a comment.

[Elon Musk]

This report is invalid, using non-optimized filter graph, also buy more ram.

[Andrew Kaiser]

Can you share what an optimized filter graph would look like richardpl? I will submit a different ticket if the problem persists. It is hard to know what is un-optimized if you do not explain. If this has to do with using overlays as opposed to a "concat" filter, that is because these ffmpeg commands are not hand-rolled, but are built using a program. The inputs need to be able to be started at any time. This is a simple example for the reproduction.

Outside of my non-optimized commands though, how is this not a bug? Why doesn't trimming clips at input time behave the same as inputting trimmed files? There seems to be no reason that the latter should use upwards of 16GB of ram, while the prior only uses 4GB of ram.

[Elon Musk]

You are messing with timestamps. This can cause big buffering. Use graphmonitor filter to watch for filtergraph memory usage.
Also stop using overlay filter for something that xstack/vstack/hstack can do.

[Andrew Kaiser]

what do you mean by "messing with timestamps"? Are you referring to setpts (like setpts=PTS+20.002/TB)? How else would I start inputs at different times?

[Elon Musk]

Your issue is excessive buffering, this is caused by caching frames because you mess up with timeline. Also does same issue happens without audio?

[Andrew Kaiser]

the same issue does not occur when audio is removed from the filter graph. Why is that? Can you please elaborate on what messing with the timeline means? If I need to allow users to start input videos at arbitrary points in the output, how else am I supposed to accomplish that?

[Elon Musk]

From logs its obvious that something caches huge amounts of memory, probably because of messing with PTS (presentation timestamps) in your graph.

Please consider using agraphmonitor with only audio filters, if it shows big cache numbers than it is bug in some of filters, otherwise it is badly designed filtergraph.

Also I see some frames have NAN timestamps, this will obviously cause big caching issues.

Also please consider uploading short input samples which would allow to reproduce this memory issue.

[Andrew Kaiser]

sample input file. This is a very tiny sample, and not enough to repro. Use youtube-dl '​https://www.youtube.com/watch?v=voprky8BrPw' to download the full input

[Andrew Kaiser]

So apologies, but I think I need some guidance here. I have googled around, and found next to nothing showing how graphmonitor and agraphmonitor are supposed to be used. I dropped it in the video inputs, and see values of zero for everything. I added it to the audio inputs (shown below) and I see no graph monitor display on the output. Also, despite only providing audio streams to the filtergraph, I still see video in the output. Do I have to specify (v=0?) Does the video automatically get inferred?

  ffmpeg \
    -v 9 -loglevel 99 \
    -ss 0 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 20 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 40 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 60 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 80 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 100 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 120 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 140 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 160 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -ss 180 -t 10 -i './samples/oom-large-4k-file/Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' \
    -filter_complex 'color=s=3840x2160:color=black:duration=100.00900000000001[base];
      [0:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=0:all=1, volume=1[a_in_0];
      [1:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=10001:all=1, volume=1[a_in_1];
      [2:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=20002:all=1, volume=1[a_in_2];
      [3:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=30003:all=1, volume=1[a_in_3];
      [4:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=40004:all=1, volume=1[a_in_4];
      [5:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=50004.99999999999:all=1, volume=1[a_in_5];
      [6:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=60005.99999999999:all=1, volume=1[a_in_6];
      [7:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=70007:all=1, volume=1[a_in_7];
      [8:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=80008.00000000001:all=1, volume=1[a_in_8];
      [9:a] agraphmonitor, asetpts=PTS-STARTPTS, adelay=90009.00000000001:all=1, volume=1[a_in_9];
      [base];
      [a_in_0][a_in_1][a_in_2][a_in_3][a_in_4][a_in_5][a_in_6][a_in_7][a_in_8][a_in_9] amix=inputs=10 [audio]' \
    -map '[audio]' \
    template.mp4 -y \
    2> ./samples/oom-large-4k-file/62041061e61fe3c628157f60602dda72/ffmpeg-verbose-logs.txt

This command does not cause the crash. The crash is only caused when both audio and video streams are present in the filtergraph

Also please consider uploading short input samples which would allow to reproduce this memory issue.

As I said, this input is far too large. I will attach a sample of the input, but it is not large enough to cause the crash. To download the input that can cause the crash, run this command: youtube-dl 'https://www.youtube.com/watch?v=voprky8BrPw'. It is 4.2GB large. This requires downloading the utility youtube-dl (​https://ytdl-org.github.io/youtube-dl/index.html)

[Elon Musk]

You are supposed to use single (a)graphmonitor filter as final output.
If OOM happens with only both A+V filtering at same time than it definitely shows problem in your filtergraph. Have you tried also different container?

Not going to download several GB files.

[Elon Musk]

Also Very Important, try with latest ffmpeg version, as bug for amix producing frames with NAN timestamps have been fixed.

[Andrew Kaiser]

Ok thank you. I am using the latest version (4.3.1) but I could try building from source if the amix bug is not part of the latest release.

Have you tried also different container?

The same bug occurs with an mp4 container generated with the following command:

ffmpeg -i 'Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mkv' -c copy -map 0 'Norway 4K - Winter in Norway - Relaxing Music with AMAZING Beautiful Nature and sound-voprky8BrPw.mp4'

If this is a poorly formed filtergraph, could you guide me in how to make it better? Apologies, I know this is not the place for assistance. One idea I have is to replace setpts/asetpts with -itsoffset and enable='between(...)'. This pretty much requires that I use overlay rather than xstack though, since the "enable" arg is tied to each input.

Other than that I can only think to be smarter about using xstack/hstack/vstack and concat filters. The issue is that some of the time, users will want videos to be rendered on a timeline & geometry that does not follow one of those filters, and I'll end up generating filtergraphs that look like this some of the time.

[Elon Musk]

Only master version of FFmpeg is supported here. Release versions have only some security fixes.

[Andrew Kaiser]

Ok I have built on master and the issue appears to be fixed (memory usage now tops out at around 4GB). So this issue can be closed.

Before we close this though, I do want to ask if you could share some advice on how I can write better filter graphs. You pointed out what was inefficient about my filtergraphs, but I do not know what better alternatives would be (especially in regards to "messing with PTS"). I want to write better filters so I dont end up writing more false bug reports. Could you please tell me what an improvement would look like? Does my suggestion here ​https://trac.ffmpeg.org/ticket/9073#comment:12 seem like its on the right track?

[Elon Musk]

Looks like you may better use (a)concat filter for what you doing. xstack/hstack/vstack is useful if you need to display different stacked videos at same time.