Duration estimation regression for h264 in mov

[Peter Z]

Summary of the bug:
There appears to be a regression with guesstimation of packet durations in ffmpeg v5 compared to v4.4.1. Tested on latest master. The problem can be seen both with ffprobe and by doing a remux of the content, with the remux the total file duration will be affected.

How to reproduce:
Create a test file with a frame rate of 30fps with a timescale double that.

% ./ffmpeg -f lavfi -i testsrc=duration=10:size=1728x720:rate=30 -c:v libx264 -b:v 1000000 -video_track_timescale 60 -movie_timescale 60 repro.mp4
ffmpeg version N-109048-g65f3bc9e7e Copyright (c) 2000-2022 the FFmpeg developers
  built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  configuration: --enable-static --enable-ffmpeg --enable-ffprobe --disable-shared --enable-gpl --enable-gray --enable-nonfree --enable-iconv --enable-libxml2 --enable-libmp3lame --enable-libfdk-aac --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxml2 --disable-stripping --enable-libfreetype --enable-libsoxr --extra-cflags=-I/opt/homebrew/include --extra-ldflags=-L/opt/homebrew/lib
  libavutil      57. 42.100 / 57. 42.100
  libavcodec     59. 52.100 / 59. 52.100
  libavformat    59. 34.101 / 59. 34.101
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 50.100 /  8. 50.100
  libswscale      6.  8.112 /  6.  8.112
  libswresample   4.  9.100 /  4.  9.100
  libpostproc    56.  7.100 / 56.  7.100
Input #0, lavfi, from 'testsrc=duration=10:size=1728x720:rate=30':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 1728x720 [SAR 1:1 DAR 12:5], 30 fps, 30 tbr, 30 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x1452061e0] using SAR=1/1
[libx264 @ 0x1452061e0] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x1452061e0] profile High 4:4:4 Predictive, level 3.2, 4:4:4, 8-bit
[libx264 @ 0x1452061e0] 264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=15 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=1000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'repro.mp4':
  Metadata:
    encoder         : Lavf59.34.101
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv444p(tv, progressive), 1728x720 [SAR 1:1 DAR 12:5], q=2-31, 1000 kb/s, 30 fps, 60 tbn
    Metadata:
      encoder         : Lavc59.52.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: N/A
frame=  300 fps=0.0 q=-1.0 Lsize=     547kB time=00:00:09.90 bitrate= 452.9kbits/s speed=11.6x
video:543kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.804443%
[libx264 @ 0x1452061e0] frame I:2     Avg QP:11.65  size: 12071
[libx264 @ 0x1452061e0] frame P:76    Avg QP: 4.77  size:  2199
[libx264 @ 0x1452061e0] frame B:222   Avg QP: 4.38  size:  1640
[libx264 @ 0x1452061e0] consecutive B-frames:  1.3%  0.0%  0.0% 98.7%
[libx264 @ 0x1452061e0] mb I  I16..4: 72.3% 23.5%  4.2%
[libx264 @ 0x1452061e0] mb P  I16..4:  2.1%  0.5%  0.6%  P16..4:  3.1%  0.5%  0.1%  0.0%  0.0%    skip:93.0%
[libx264 @ 0x1452061e0] mb B  I16..4:  0.0%  0.1%  0.0%  B16..8:  3.6%  0.2%  0.0%  direct: 1.6%  skip:94.4%  L0:52.6% L1:46.7% BI: 0.7%
[libx264 @ 0x1452061e0] final ratefactor: 0.18
[libx264 @ 0x1452061e0] 8x8 transform intra:21.2% inter:84.3%
[libx264 @ 0x1452061e0] coded y,u,v intra: 6.2% 7.1% 6.3% inter: 0.1% 1.3% 1.2%
[libx264 @ 0x1452061e0] i16 v,h,dc,p: 89%  9%  1%  2%
[libx264 @ 0x1452061e0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 62%  4% 33%  0%  0%  0%  0%  0%  0%
[libx264 @ 0x1452061e0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 47% 34%  0%  0%  0%  0%  0%  0%
[libx264 @ 0x1452061e0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x1452061e0] ref P L0: 84.6%  0.8% 10.9%  3.6%
[libx264 @ 0x1452061e0] ref B L0: 93.8%  6.2%  0.1%
[libx264 @ 0x1452061e0] ref B L1: 98.2%  1.8%
[libx264 @ 0x1452061e0] kb/s:444.27

Check the duration of the packets with ffprobe:

% ffprobe -show_packets repro.mp4
 --- full ffprobe output in attached file --- 

With ffmpeg 4.4.1 this would show packet duration as 2 (2/60) but with ffmpeg 5.x all the packets get duration 1 (1/60).

The consequences of this can also be observed by remuxing the file:

% ffmpeg -i repro.mp4 -c copy remux.mp4

Now the probed duration is no longer 10s but 1/60 less 9.983333.

I think this problem might have appeared after H264 parsing was introduced with this commit:
8a3f8afa4e46011e9c5849f8e0d57ec9b53deef7

At least I have observed that during the execution of the compute_frame_duration function there is in ffmpeg5 a valid pointer to a AVCodecParserContext set but not in version 4.4.1 of ffmpeg.
​https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/demux.c#L668
Thus the heuristic for guessing the frame duration is now taking the second branch and going purely on timebase.

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Carl Eugen Hoyos]

Please test current FFmpeg git head and provide the command line you tested together with the complete, uncut console output to make this a valid ticket.

[Carl Eugen Hoyos]

So the problem you see is that the file repro.mp4 is not created correctly?

[Peter Z]

Replying to Carl Eugen Hoyos:

So the problem you see is that the file repro.mp4 is not created correctly?

No, as far as I can tell it is not actually the file produced that is incorrect but rather how ffmpeg later interprets the file. I think the problem is with demuxing as it can be observed both with ffprobe and by remuxing the file. With the remux the remuxed file will be incorrect.

[Balling]

The first file has STTS duration of last frame. Unforunately that info is lost when you c copy. We do not support reading that metadata: ​https://trac.ffmpeg.org/ticket/7828#comment:8

Patch is here to use stts atom. ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190429225027.81295-1-fumoboy007@me.com/