variable FPS incorrect frame length

[kubabrecka]

I have a sample video to reproduce this bug - see attachment.

The video is in Apple Animation codec (qtrle) with variable frame rate and contains only 5 frames. Frames 1,2,4 and 5 are displayed for a very short time, but frame 3 is much longer.

When I extract all the frames into PNGs with this command:

./ffmpeg -vsync 2 -i apple-animation-variable-fps-bug.mov -vf "[in]showinfo[out]" out/img%04d.png

it's fine, I get 5 PNG files, and the timestamps shown on the output are correct:

n:0 pts:0 pts_time:0 pos:48
n:1 pts:600 pts_time:0.2 pos:4335
n:2 pts:1300 pts_time:0.433333 pos:8286
n:3 pts:7300 pts_time:2.43333 pos:12518
n:4 pts:8100 pts_time:2.7 pos:16642

as you can see, frame n:2 (the third frame) is displayed for much longer than the others. However, when I run this command to dump the frames with a constant FPS:

./ffmpeg -i apple-animation-variable-fps-bug.mov -r 30 out/img%04d.png

the frames are dumped with wrong timing. The "long" frame was 2 seconds long, so it should be in about 60 frames (with FPS 30), but instead it only gets 7 frames. But the next frame which should be very short is dumped into 60 frames. It seems that the conversion between variable FPS into constant FPS incorrectly assigns frame length into the next frame.

[Carl Eugen Hoyos]

For future bug-reports: Please always add complete, uncut console output together with the command line.

Also reproducible with -f mov:

$ ffmpeg -i apple-animation-variable-fps-bug.mov -r 30 out.mov
ffmpeg version N-44854-gbad603c Copyright (c) 2000-2012 the FFmpeg developers
  built on Sep 28 2012 23:13:16 with gcc 4.5.3 (GCC)
  configuration: --cc='/usr/local/gcc-4.5.3/bin/gcc -m32' --enable-gpl
  libavutil      51. 73.101 / 51. 73.101
  libavcodec     54. 61.100 / 54. 61.100
  libavformat    54. 29.105 / 54. 29.105
  libavdevice    54.  3.100 / 54.  3.100
  libavfilter     3. 18.100 /  3. 18.100
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 16.100 /  0. 16.100
  libpostproc    52.  1.100 / 52.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'apple-animation-variable-fps-bug.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 1942-06-22 00:41:40
  Duration: 00:00:02.93, start: 0.000000, bitrate: 58 kb/s
    Stream #0:0(eng): Video: qtrle (rle  / 0x20656C72), rgb24, 112x182, 55 kb/s, 1.69 fps, 30 tbr, 3k tbn, 3k tbc
    Metadata:
      creation_time   : 1942-06-22 00:41:40
      handler_name    : Apple Alias Data Handler
Output #0, mov, to 'out.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    encoder         : Lavf54.29.105
    Stream #0:0(eng): Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 112x182, q=2-31, 200 kb/s, 30 tbn, 30 tbc
    Metadata:
      creation_time   : 1942-06-22 00:41:40
      handler_name    : Apple Alias Data Handler
Stream mapping:
  Stream #0:0 -> #0:0 (qtrle -> mpeg4)
Press [q] to stop, [?] for help
frame=   82 fps=0.0 q=2.0 Lsize=      42kB time=00:00:02.73 bitrate= 125.5kbits/s dup=77 drop=0
video:41kB audio:0kB subtitle:0 global headers:0kB muxing overhead 2.723810%

[Michael Niedermayer]

works with -vf fps

[Roger Pack]

If this is a duplicate of #1925 (which it may be), then it is also reproducible with this:

ffmpeg -f lavfi -i testsrc=duration=3:rate=1 -r 27 out.mp4

(the first frame should display a 0 and last for 1s, instead it just flits by). Interestingly, if you remove the "-r 27" from the output, then it works as expected.

[Zsbán Ambrus]

See also 2664 which is marked as a duplicate of this bug but uses fixed framerate input.

[Anshul]

NOT FROM GIT

It does not work even with -vf fps
Below was my command
ffmpeg -r 1/5 -f concat -i /tmp/tmpsaR4EA -i audio/sound.mp3 -vf "format=yuv420p,scale=640x480,subtitles=/tmp/tmpKu6UJ3.srt,fps=25" "output/Great_Lakes_region_timeshare_camp.mp4"

ffmpeg version 2.1.1 Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec  9 2013 15:48:15 with gcc 4.7 (SUSE Linux)
  configuration: --enable-libass --enable-libfaac --enable-libx264 --enable-nonfree --enable-gpl
  libavutil      52. 48.101 / 52. 48.101
  libavcodec     55. 39.101 / 55. 39.101
  libavformat    55. 19.104 / 55. 19.104
  libavdevice    55.  5.100 / 55.  5.100
  libavfilter     3. 90.100 /  3. 90.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, concat, from '/tmp/tmpsaR4EA':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj444p(pc), 640x480 [SAR 72:72 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Input #1, mp3, from 'audio/sound.mp3':
  Metadata:
    album           : @RdxMob.Com
    encoded_by      : Tagged by Flash Renamer - http://www.rlvision.com
    title           : Sound_Of_Nature::www.RdxMoB.CoM
    artist          : @RdxMob.Com
    track           : Sound_Of_Nature
    TDTG            : 2010-02-13T16:18:41
  Duration: 00:00:29.07, start: 0.000000, bitrate: 128 kb/s
    Stream #1:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
File 'output/Great_Lakes_region_timeshare_camp.mp4' already exists. Overwrite ? [y/N] y
[swscaler @ 0xab9ffa0] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0xaa30ac0] using SAR=1/1
[libx264 @ 0xaa30ac0] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 AVX
[libx264 @ 0xaa30ac0] profile High, level 3.0
[libx264 @ 0xaa30ac0] 264 - core 125 - H.264/MPEG-4 AVC codec - Copyleft 2003-2012 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'output/Great_Lakes_region_timeshare_camp.mp4':
  Metadata:
    encoder         : Lavf55.19.104
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 640x480 [SAR 1:1 DAR 4:3], q=-1--1, 12800 tbn, 25 tbc
    Stream #0:1: Audio: aac (libfaac) ([64][0][0][0] / 0x0040), 44100 Hz, stereo, s16, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg -> libx264)
  Stream #1:0 -> #0:1 (mp3 -> libfaac)
Press [q] to stop, [?] for help
[Parsed_subtitles_2 @ 0xaa27ee0] Neither PlayResX nor PlayResY defined. Assuming 384x288
[Parsed_subtitles_2 @ 0xaa27ee0] fontconfig: Selected font is not the requested one: 'Arimo' != 'Arial'
Input stream #0:0 frame changed from size:640x480 fmt:yuvj444p to size:259x194 fmt:yuvj420p
[swscaler @ 0xab99d40] deprecated pixel format used, make sure you did set range correctly
[Parsed_subtitles_2 @ 0xaa100e0] Neither PlayResX nor PlayResY defined. Assuming 384x288
[Parsed_subtitles_2 @ 0xaa100e0] fontconfig: Selected font is not the requested one: 'Arimo' != 'Arial'
Input stream #0:0 frame changed from size:259x194 fmt:yuvj420p to size:640x480 fmt:yuvj444p
[swscaler @ 0xab99c80] deprecated pixel format used, make sure you did set range correctly
[Parsed_subtitles_2 @ 0xaa0bcc0] Neither PlayResX nor PlayResY defined. Assuming 384x288
[Parsed_subtitles_2 @ 0xaa0bcc0] fontconfig: Selected font is not the requested one: 'Arimo' != 'Arial'
[output stream 0:0 @ 0xaa2b5c0] 100 buffers queued in output stream 0:0, something may be wrong.
[mp3 @ 0xaa2e840] Header missing     390kB time=00:00:27.92 bitrate= 114.4kbits/s    
Error while decoding stream #1:0: Invalid data found when processing input
frame= 2876 fps=513 q=-1.0 Lsize=     634kB time=00:01:54.96 bitrate=  45.2kbits/s    
video:130kB audio:454kB subtitle:0 global headers:0kB muxing overhead 8.601034%
[libx264 @ 0xaa30ac0] frame I:12    Avg QP:19.38  size:  5643
[libx264 @ 0xaa30ac0] frame P:725   Avg QP:27.91  size:    28
[libx264 @ 0xaa30ac0] frame B:2139  Avg QP:28.00  size:    21
[libx264 @ 0xaa30ac0] consecutive B-frames:  0.8%  0.0%  0.0% 99.2%
[libx264 @ 0xaa30ac0] mb I  I16..4: 90.9%  4.1%  4.9%
[libx264 @ 0xaa30ac0] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 0xaa30ac0] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.1%  0.0%  0.0%  direct: 0.0%  skip:99.9%  L0:41.0% L1:59.0% BI: 0.0%
[libx264 @ 0xaa30ac0] 8x8 transform intra:3.8% inter:62.7%
[libx264 @ 0xaa30ac0] coded y,uvDC,uvAC intra: 5.1% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 0xaa30ac0] i16 v,h,dc,p: 95%  2%  3%  0%
[libx264 @ 0xaa30ac0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 31% 25%  7%  2%  4%  3%  3%  6%
[libx264 @ 0xaa30ac0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 23% 24%  3%  5%  4%  6%  4%  5%
[libx264 @ 0xaa30ac0] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0xaa30ac0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0xaa30ac0] ref P L0: 81.5%  8.2%  5.2%  5.2%
[libx264 @ 0xaa30ac0] ref B L0: 81.5% 18.3%  0.2%
[libx264 @ 0xaa30ac0] ref B L1: 98.6%  1.4%
[libx264 @ 0xaa30ac0] kb/s:9.23
[

My 3 starting image were missing, I tried putting fps filter at each place, position of fps filter does not solve my problem

[Anshul]

if I put same fps of input and output, then frame are missing

ffmpeg -r 1/5  -f concat -i /tmp/tmpsaR4EA -i audio/sound.mp3 -vf "format=yuv420p,scale=640x480,subtitles=/tmp/tmpKu6UJ3.srt,fps=1/5" "output/Great_Lakes_region_timeshare_camp.mp4"

but if I don't put the fps filter flag at output then every thing is all right

[Anshul]

NOT FROM GIT
I was looking in code

        if (delta < -1.1)
         {
                 av_log(NULL, AV_LOG_VERBOSE, "2nb_freame%d***Drop! %d----%ldd\n",nb_frames,format_video_sync,delta);
             nb_frames = 0;
         }

and found delta as -2.0 , that makes nb_frame = 0

[Anshul]

NOT FROM GIT
Now When I am looking how delta is calculated, then I found

delta = sync_ipts - ost->sync_opts + duration;

where values were
duration = 1.0000
sync_ipts = 0
ost->sync_opts = 3

delta = 0 -3 +1 = -2.00

and my frame was lost

[Anshul]

git have same calculation
but adding break in below code does not have frame loss

    switch (format_video_sync) {
    case VSYNC_VSCFR:
        if (ost->frame_number == 0 && delta - duration >= 0.5) {
            av_log(NULL, AV_LOG_DEBUG, "Not duplicating %d initial frames\n", (int)lrintf(delta - duration));
            delta = duration;
            ost->sync_opts = lrint(sync_ipts);
        }    
    case VSYNC_CFR:
        // FIXME set to 0.5 after we fix some dts/pts bugs like in avidec.c
        if (delta < -1.1)
            nb_frames = 0; 
        else if (delta > 1.1) 
            nb_frames = lrintf(delta);
        break;

[Roger Pack]

OK offering a $100 USD bounty here, if it's helpful :)

[llogan]

Replying to rogerdpack:

OK offering a $100 USD bounty here, if it's helpful :)

You can also post the bounty at the FFmpeg Bountysource page:

​https://www.bountysource.com/issues/1414359-variable-fps-incorrect-frame-length

[Carl Eugen Hoyos]

Replying to rogerdpack:

OK offering a $100 USD bounty here, if it's helpful :)

I don't understand, is there an issue if you are using -vf fps?

[Roger Pack]

Seems to work ok if using -vf fps. Only -r 27 (like ffmpeg.exe -f lavfi -i testsrc=duration=30:rate=1 -r 27 out.mp4) seems to fail for me.

Also saw this note on #3164 "Note that the pullup filter does not work correctly with the fps filter, it currently needs -r." Thanks!

[Carl Eugen Hoyos]

So what do you want to spend your $100 for (exactly)?

[Roger Pack]

In this case it would be just to make "ffmpeg -f lavfi -i testsrc=duration=30:rate=1 -r 27 out.mp4" display the first frame for a full second correctly [like -vf fps does correctly today]. Currently the first frame is ther but only for a split second, which seems off somehow...I'd expect it to be there for a second...thanks for your interest!

[Michael Niedermayer]

Fixed in 9f6d48d696d679de77e8cb513d5f64cd708ed86f