"-filter_complex" instrumented error: unexpected video delay

[degradox]

Summary of the bug: using complex filter with alphamerge caused audio-video desync.

How to reproduce:
When converting normally, the output has audio and video synced, like the input:

% ffmpeg -y -i input_good.mp4 -map 0:v -map 0:a -c:v libx264 -profile:v high -level:v 4.2 -pix_fmt yuv420p \
  -r 60 -b:v 600k -preset fast -c:a aac -b:a 80k -ar 48000 -ac 2 -fps_mode cfr -nostdin output_good.mp4

Using filter advances somehow the video. The first minutes it's about 7 seconds. At the end the video ends about 13 seconds earlier than audio, so there's the last frame stuck for that time:

% ffmpeg -y \
  -i input_good.mp4 \
  -filter_complex "\
    [0:v]scale=1920:1080:force_original_aspect_ratio=decrease,setpts=PTS-STARTPTS [scaled_main]; \
    color=c=black@0:s=1280x720:d=10000 [blank_overlay]; \
    [blank_overlay]format=gray [mask_trim]; \
    [blank_overlay][mask_trim]alphamerge [overlay_stream]; \
    [scaled_main][overlay_stream]overlay=x=main_w-overlay_w-0:y=main_h-overlay_h-0:eof_action=pass:shortest=0 [outv]" \
  -map "[outv]" -map 0:a -c:v libx264 -profile:v high -level:v 4.2 -pix_fmt yuv420p -r 60 -b:v 600k -preset fast \
  -c:a aac -b:a 80k -ar 48000 -ac 2 -fps_mode cfr -nostdin output_desynced.mp4

Both outputs have the same duration as the input, 51 minutes 54 seconds.

[degradox]

Sample and output files are all here:
​https://mega.nz/folder/YKAAGLiK#12KyX41EFsOMIXnMEYqCbA

[MasterQuestionable]

͏    I don't think the cause would be ͏"alphamerge".
͏    Probably your ͏"overlay" usage: The sourced ͏"color" input seems to have unmatching duration. [ Also the rate may matter. ]
͏    And the ͏"eof_action" is probably not well-set.

͏    The other possible source is ͏"setpts", which meddles the input timestamps.

[degradox]

@MasterQuestionable can you provide the corrected command to test?

[MasterQuestionable]

͏    It might be somewhat bothersome for me to further process your files.

͏    General guideline:
͏    Try removing various parts of the command, and test if it works.
͏    If the error is reproducible with the 1st 60 s: limit the output duration to 60.

[degradox]

You have the source file there, trim it to your liking to reproduce the issue exactly the same.

It's possible that there are several problems along the timeline, so I don't want to compromise the bug report by mutilating the source files, you probably know much more than me to do any processing, or eventually any developer will do it.

I'm sure it will take you much more time to fix the issue than to just download the file from a direct download.

[MasterQuestionable]

͏    Pardon, I have limited bandwidth processing alike.
͏    Previous conclusion come outright from semantic analysis of your texts. (including log partly)

͏    And I think uploading anything larger than 10 MiB probably wouldn't be of use.

͏    You may do your testing locally, and report the result of your observation.
͏    "-v debug -hide_banner -nostdin -nostats"

͏    Mostly helpful, from your previous report...
͏    ​https://trac.ffmpeg.org/ticket/11146#comment:11

[degradox]

Without setpts=PTS-STARTPTS the issue does not happen. In the ffmpeg conversion log the number of duplicated and dropped frames also is very different:

• With setpts=PTS-STARTPTS: dup=115 drop=837
• Without ,setpts=PTS-STARTPTS: dup=120 drop=0

Adding setpts=PTS-STARTPTS to filter_complex introduces this weird bug. The question is, should this happen, is this expected?

[Cigaes]

Should changing the timestamps of the frames, the ones that are used to synchronize them across streams, change the synchronization of frames across streams?

I'd say yes.

[MasterQuestionable]

͏    Try ͏"-avoid_negative_ts 2"?
͏    Or add "asetpts" in the same manner...

[degradox]

Adding asetpts, video ends about 6 seconds before audio, log is dup=115 drop=837, so something is happening:

% ffmpeg -y \
  -i input_good.mp4 \
  -filter_complex "\
    [0:v]scale=1920:1080:force_original_aspect_ratio=decrease,setpts=PTS-STARTPTS [scaled_main]; \
    color=c=black@0:s=1280x720:d=10000 [blank_overlay]; \
    [blank_overlay]format=gray [mask_trim]; \
    [blank_overlay][mask_trim]alphamerge [overlay_stream]; \
    [scaled_main][overlay_stream]overlay=x=main_w-overlay_w-0:y=main_h-overlay_h-0:eof_action=pass:shortest=0 [outv]; \
    [0:a]asetpts=PTS-STARTPTS [outa]" \
  -map "[outv]" -map "[outa]" \
  -c:v libx264 -profile:v high -level:v 4.2 -pix_fmt yuv420p -r 60 -b:v 600k -preset fast \
  -c:a aac -b:a 80k -ar 48000 -ac 2 -fps_mode cfr -nostdin output_desynced_asetpts.mp4

[degradox]

Not using asetpts but using -avoid_negative_ts 2, log is dup=115 drop=837, video ends about 14 seconds before audio:

% ffmpeg -y \
  -i input_good.mp4 \
  -filter_complex "\
    [0:v]scale=1920:1080:force_original_aspect_ratio=decrease,setpts=PTS-STARTPTS [scaled_main]; \
    color=c=black@0:s=1280x720:d=10000 [blank_overlay]; \
    [blank_overlay]format=gray [mask_trim]; \
    [blank_overlay][mask_trim]alphamerge [overlay_stream]; \
    [scaled_main][overlay_stream]overlay=x=main_w-overlay_w-0:y=main_h-overlay_h-0:eof_action=pass:shortest=0 [outv]" \
  -map "[outv]" -map 0:a \
  -c:v libx264 -profile:v high -level:v 4.2 -pix_fmt yuv420p -r 60 -b:v 600k -preset fast \
  -c:a aac -b:a 80k -ar 48000 -ac 2 -fps_mode cfr \
  -avoid_negative_ts 2 \
  -nostdin output_desynced.mp4

[degradox]

Using both asetpts and -avoid_negative_ts 2, logs is dup=115 drop=837, video ends 5 seconds before audio:

% ffmpeg -y \
  -i input_good.mp4 \
  -filter_complex "\
    [0:v]scale=1920:1080:force_original_aspect_ratio=decrease,setpts=PTS-STARTPTS [scaled_main]; \
    color=c=black@0:s=1280x720:d=10000 [blank_overlay]; \
    [blank_overlay]format=gray [mask_trim]; \
    [blank_overlay][mask_trim]alphamerge [overlay_stream]; \
    [scaled_main][overlay_stream]overlay=x=main_w-overlay_w-0:y=main_h-overlay_h-0:eof_action=pass:shortest=0 [outv]; \
    [0:a]asetpts=PTS-STARTPTS [outa]" \
  -map "[outv]" -map "[outa]" \
  -c:v libx264 -profile:v high -level:v 4.2 -pix_fmt yuv420p -r 60 -b:v 600k -preset fast \
  -c:a aac -b:a 80k -ar 48000 -ac 2 -fps_mode cfr \
  -avoid_negative_ts 2 \
  -nostdin output_desynced.mp4

Same exact results using -avoid_negative_ts 1 instead of -avoid_negative_ts 2.

[MasterQuestionable]

͏    Try "-ignore_editlist"?
͏    ​https://trac.ffmpeg.org/ticket/11250#comment:5