Opened 4 months ago

Last modified 4 months ago

#7101 new defect

Overlay wrongly enabled before specified time

Reported by: slhck Owned by:
Priority: normal Component: avfilter
Version: git-master Keywords: overlay
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

I want to show a "loading" spinner at a given time for a given amount of time, overlaid on top of a video. The video should freeze underneath the spinner.

In the following case, I want to show the spinner as an "initial loading" for 4 seconds, then at 10 seconds (media time) for 0.5 s, etc. At the same time, the video is looped (size = 1 frame) for the same amount of time.

Consider the following command:

ffmpeg \
-y \
-f lavfi -i testsrc=duration=60:size=320x240:rate=60,format=pix_fmts=yuv420p \
-i click_and_count.m4a \
-filter_complex " \
    [0:v] \
        loop=loop=240:size=1:start=0, setpts=N/FRAME_RATE/TB, \
        loop=loop=30:size=1:start=840, setpts=N/FRAME_RATE/TB, \
        loop=loop=84:size=1:start=1140, setpts=N/FRAME_RATE/TB, \
        loop=loop=48:size=1:start=1548, setpts=N/FRAME_RATE/TB \
    [stallvid]; \
        movie=filename=spinner-64-white.png:loop=0, setpts=N/(FRAME_RATE*TB)*2 \
    [spinner]; \
    [stallvid][spinner] \
        overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2:shortest=1:\
        enable='between(t,0,4.0)+between(t,14.0,14.5)+between(t,19.0,20.4)+between(t,25.8,26.6)' \
    [outv];
    [1:a] \
        aloop=loop=192000:size=1:start=0, asetpts=N/SAMPLE_RATE/TB, \
        aloop=loop=24000:size=1:start=672000, asetpts=N/SAMPLE_RATE/TB, \
        aloop=loop=67200:size=1:start=912000, asetpts=N/SAMPLE_RATE/TB, \
        aloop=loop=38400:size=1:start=1238399, asetpts=N/SAMPLE_RATE/TB, \
        volume=0:enable='between(t,0,4.0)+between(t,14.0,14.5)+between(t,19.0,20.4)+between(t,25.8,26.6)' \
    [outa] \
" -shortest -map "[outv]" -map "[outa]" output.mp4

The spinner is indeed shown exactly when the 10 appears in the video (at 00:00:14), but it is also shown for exactly one frame, four frames before the 10 appears.

This should of course not happen, as the overlay is only enabled between 14 and 14.5.

Attachments (2)

spinner-64-white.png (31.4 KB) - added by slhck 4 months ago.
click_and_count.m4a (1002.5 KB) - added by slhck 4 months ago.

Download all attachments as: .zip

Change History (10)

Changed 4 months ago by slhck

Changed 4 months ago by slhck

comment:1 Changed 4 months ago by slhck

ffmpeg output:

ffmpeg version N-90389-g72bb955625-tessus Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.39.2)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
  libavutil      56. 11.100 / 56. 11.100
  libavcodec     58. 14.100 / 58. 14.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 13.100 /  7. 13.100
  libswscale      5.  0.102 /  5.  0.102
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, lavfi, from 'testsrc=duration=60:size=320x240:rate=60,format=pix_fmts=yuv420p':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 60 tbr, 60 tbn, 60 tbc
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'click_and_count.m4a':
  Metadata:
    major_brand     : M4A
    minor_version   : 512
    compatible_brands: isomiso2
    encoder         : Lavf57.83.100
  Duration: 00:01:01.08, start: 0.000000, bitrate: 134 kb/s
    Stream #1:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 132 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 (rawvideo) -> loop
  Stream #1:0 (aac) -> aloop
  overlay -> Stream #0:0 (libx264)
  volume -> Stream #0:1 (aac)
Press [q] to stop, [?] for help
[libx264 @ 0x7fe6f0816e00] using SAR=1/1
[libx264 @ 0x7fe6f0816e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7fe6f0816e00] profile High, level 2.1
[libx264 @ 0x7fe6f0816e00] 264 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=7 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf58.10.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p(progressive), 320x240 [SAR 1:1 DAR 4:3], q=-1--1, 60 fps, 15360 tbn, 60 tbc (default)
    Metadata:
      encoder         : Lavc58.14.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      encoder         : Lavc58.14.100 aac
[out_0_1 @ 0x7fe6efc0f380] 100 buffers queued in out_0_1, something may be wrong.
[out_0_1 @ 0x7fe6efc0f380] 1000 buffers queued in out_0_1, something may be wrong.
Past duration 0.999992 too large       0kB time=00:00:01.70 bitrate=   0.2kbits/s speed=3.41x
Past duration 0.999992 too large       0kB time=00:00:03.52 bitrate=   0.1kbits/s speed=3.52x
    Last message repeated 8 times
Past duration 0.999992 too large       0kB time=00:00:14.14 bitrate=   0.0kbits/s speed= 9.4x
    Last message repeated 11 times
Past duration 0.999992 too large     512kB time=00:00:31.55 bitrate= 132.9kbits/s speed=12.6x
    Last message repeated 24 times
Past duration 0.999992 too large=    1280kB time=00:01:02.12 bitrate= 168.8kbits/s speed=17.7x
    Last message repeated 29 times
[Parsed_testsrc_0 @ 0x7fe6efd02500] EOF timestamp not reliable
Past duration 0.999992 too large
    Last message repeated 2 times
frame= 4002 fps=1079 q=-1.0 Lsize=    1558kB time=00:01:07.71 bitrate= 188.5kbits/s speed=18.3x
video:457kB audio:1003kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 6.756100%
[libx264 @ 0x7fe6f0816e00] frame I:17    Avg QP:16.61  size:  3274
[libx264 @ 0x7fe6f0816e00] frame P:1766  Avg QP:17.44  size:   200
[libx264 @ 0x7fe6f0816e00] frame B:2219  Avg QP:16.02  size:    27
[libx264 @ 0x7fe6f0816e00] consecutive B-frames: 25.5%  1.2%  1.0% 72.2%
[libx264 @ 0x7fe6f0816e00] mb I  I16..4: 65.0%  7.0% 27.9%
[libx264 @ 0x7fe6f0816e00] mb P  I16..4:  1.7%  0.4%  0.1%  P16..4:  9.0%  2.5%  1.2%  0.0%  0.0%    skip:85.0%
[libx264 @ 0x7fe6f0816e00] mb B  I16..4:  0.1%  0.0%  0.0%  B16..8:  2.2%  0.0%  0.0%  direct: 0.0%  skip:97.6%  L0:51.7% L1:47.4% BI: 0.9%
[libx264 @ 0x7fe6f0816e00] 8x8 transform intra:13.5% inter:56.1%
[libx264 @ 0x7fe6f0816e00] coded y,uvDC,uvAC intra: 5.6% 24.2% 11.7% inter: 0.5% 2.6% 1.0%
[libx264 @ 0x7fe6f0816e00] i16 v,h,dc,p: 77%  5%  3% 15%
[libx264 @ 0x7fe6f0816e00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 29% 12% 58%  0%  0%  0%  0%  0%  0%
[libx264 @ 0x7fe6f0816e00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 47% 26% 18%  2%  2%  2%  1%  2%  0%
[libx264 @ 0x7fe6f0816e00] i8c dc,h,v,p: 25% 14% 49% 12%
[libx264 @ 0x7fe6f0816e00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fe6f0816e00] ref P L0: 72.2%  5.6% 18.7%  3.5%
[libx264 @ 0x7fe6f0816e00] ref B L0: 54.0% 40.8%  5.2%
[libx264 @ 0x7fe6f0816e00] ref B L1: 97.6%  2.4%
[libx264 @ 0x7fe6f0816e00] kb/s:56.01
[aac @ 0x7fe6f0814a00] Qavg: 8128.149

comment:2 Changed 4 months ago by Gyan

Does the timing issue occur if you disable all audio processing or perform it in a separate graph?

comment:3 Changed 4 months ago by slhck

(removed comment, issue persists even when disabling audio)

Last edited 4 months ago by slhck (previous) (diff)

comment:4 Changed 4 months ago by slhck

When I handle audio in a separate filter graph, the issue persists:

ffmpeg \
-y \
-f lavfi -i testsrc=duration=60:size=320x240:rate=60,format=pix_fmts=yuv420p \
-i spinners/click_and_count.m4a \
-filter_complex " \
    [0:v] \
        loop=loop=240:size=1:start=0, setpts=N/FRAME_RATE/TB, \
        loop=loop=30:size=1:start=840, setpts=N/FRAME_RATE/TB, \
        loop=loop=84:size=1:start=1140, setpts=N/FRAME_RATE/TB, \
        loop=loop=48:size=1:start=1548, setpts=N/FRAME_RATE/TB \
    [stallvid]; \
        movie=filename=spinners/spinner-64-white.png:loop=0, setpts=N/(FRAME_RATE*TB)*2 \
    [spinner]; \
    [stallvid][spinner] \
        overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2:shortest=1:\
        enable='between(t,0,4.0)+between(t,14.0,14.5)+between(t,19.0,20.4)+between(t,25.8,26.6)' \
    [outv]
" \
-filter_complex " \
    [1:a] \
        aloop=loop=192000:size=1:start=0, asetpts=N/SAMPLE_RATE/TB, \
        aloop=loop=24000:size=1:start=672000, asetpts=N/SAMPLE_RATE/TB, \
        aloop=loop=67200:size=1:start=912000, asetpts=N/SAMPLE_RATE/TB, \
        aloop=loop=38400:size=1:start=1238399, asetpts=N/SAMPLE_RATE/TB, \
        volume=0:enable='between(t,0,4.0)+between(t,14.0,14.5)+between(t,19.0,20.4)+between(t,25.8,26.6)' \
    [outa] \
" \
-shortest -map "[outv]" -map "[outa]" output.mp4

comment:5 Changed 4 months ago by slhck

Actually, even without audio processing (i.e., when adding -an), the issue is the same – the spinner shortly flashes before the specified time.

~/Downloads/ffmpeg \
-y \
-f lavfi -i testsrc=duration=60:size=320x240:rate=60,format=pix_fmts=yuv420p \
-i spinners/click_and_count.m4a \
-filter_complex " \
    [0:v] \
        loop=loop=240:size=1:start=0, setpts=N/FRAME_RATE/TB, \
        loop=loop=30:size=1:start=840, setpts=N/FRAME_RATE/TB, \
        loop=loop=84:size=1:start=1140, setpts=N/FRAME_RATE/TB, \
        loop=loop=48:size=1:start=1548, setpts=N/FRAME_RATE/TB \
    [stallvid]; \
        movie=filename=spinners/spinner-64-white.png:loop=0, setpts=N/(FRAME_RATE*TB)*2 \
    [spinner]; \
    [stallvid][spinner] \
        overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2:shortest=1:\
        enable='between(t,0,4.0)+between(t,14.0,14.5)+between(t,19.0,20.4)+between(t,25.8,26.6)' \
    [outv]
" \
-map "[outv]" -an test/tmp.mp4
ffmpeg version N-90417-ga8c2d375ca-tessus Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.39.2)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
  libavutil      56. 11.100 / 56. 11.100
  libavcodec     58. 15.100 / 58. 15.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 13.100 /  7. 13.100
  libswscale      5.  0.102 /  5.  0.102
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, lavfi, from 'testsrc=duration=60:size=320x240:rate=60,format=pix_fmts=yuv420p':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 60 tbr, 60 tbn, 60 tbc
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'spinners/click_and_count.m4a':
  Metadata:
    major_brand     : M4A
    minor_version   : 512
    compatible_brands: isomiso2
    encoder         : Lavf57.83.100
  Duration: 00:01:01.08, start: 0.000000, bitrate: 134 kb/s
    Stream #1:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 132 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 (rawvideo) -> loop
  overlay -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[libx264 @ 0x7fdd8a008800] using SAR=1/1
[libx264 @ 0x7fdd8a008800] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7fdd8a008800] profile High, level 2.1
[libx264 @ 0x7fdd8a008800] 264 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=7 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'test/tmp.mp4':
  Metadata:
    encoder         : Lavf58.10.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 320x240 [SAR 1:1 DAR 4:3], q=-1--1, 60 fps, 15360 tbn, 60 tbc
    Metadata:
      encoder         : Lavc58.15.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
Past duration 0.999992 too large
    Last message repeated 21 times
Past duration 0.999992 too large       0kB time=00:00:17.16 bitrate=   0.0kbits/s speed=34.3x
    Last message repeated 24 times
Past duration 0.999992 too large=     256kB time=00:00:50.86 bitrate=  41.2kbits/s speed=33.9x
    Last message repeated 32 times
[Parsed_testsrc_0 @ 0x7fdd88c03b00] EOF timestamp not reliable
frame= 4002 fps=2009 q=-1.0 Lsize=     497kB time=00:01:06.65 bitrate=  61.1kbits/s speed=33.5x
video:457kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.911427%
[libx264 @ 0x7fdd8a008800] frame I:17    Avg QP:16.61  size:  3274
[libx264 @ 0x7fdd8a008800] frame P:1766  Avg QP:17.44  size:   200
[libx264 @ 0x7fdd8a008800] frame B:2219  Avg QP:16.02  size:    27
[libx264 @ 0x7fdd8a008800] consecutive B-frames: 25.5%  1.2%  1.0% 72.2%
[libx264 @ 0x7fdd8a008800] mb I  I16..4: 65.0%  7.0% 27.9%
[libx264 @ 0x7fdd8a008800] mb P  I16..4:  1.7%  0.4%  0.1%  P16..4:  9.0%  2.5%  1.2%  0.0%  0.0%    skip:85.0%
[libx264 @ 0x7fdd8a008800] mb B  I16..4:  0.1%  0.0%  0.0%  B16..8:  2.2%  0.0%  0.0%  direct: 0.0%  skip:97.6%  L0:51.7% L1:47.4% BI: 0.9%
[libx264 @ 0x7fdd8a008800] 8x8 transform intra:13.5% inter:56.1%
[libx264 @ 0x7fdd8a008800] coded y,uvDC,uvAC intra: 5.6% 24.2% 11.7% inter: 0.5% 2.6% 1.0%
[libx264 @ 0x7fdd8a008800] i16 v,h,dc,p: 77%  5%  3% 15%
[libx264 @ 0x7fdd8a008800] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 29% 12% 58%  0%  0%  0%  0%  0%  0%
[libx264 @ 0x7fdd8a008800] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 47% 26% 18%  2%  2%  2%  1%  2%  0%
[libx264 @ 0x7fdd8a008800] i8c dc,h,v,p: 25% 14% 49% 12%
[libx264 @ 0x7fdd8a008800] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fdd8a008800] ref P L0: 72.2%  5.6% 18.7%  3.5%
[libx264 @ 0x7fdd8a008800] ref B L0: 54.0% 40.8%  5.2%
[libx264 @ 0x7fdd8a008800] ref B L1: 97.6%  2.4%
[libx264 @ 0x7fdd8a008800] kb/s:56.01

comment:6 Changed 4 months ago by slhck

Upon further investigation it seems to be caused only by the combination of looping the input video and adding the overlay. Adding the overlay on its own does not cause problems.

comment:7 Changed 4 months ago by Gyan

This doesn't seem to be related to looping the input video. If I output the looped testsrc to a file and use that, it still occurs.

But if I append to the PNG filters, fps=60, I don't get the premature overlay.

Last edited 4 months ago by Gyan (previous) (diff)

comment:8 Changed 4 months ago by slhck

That seems to fix it, thanks!

So, I assume that since the PTS are generated for the PNG input, the overlay filter – when seeing two different frame rates – interpolates the timestamps incorrectly?

Note: See TracTickets for help on using tickets.