time_scale / num_units_in_tick is not infinite precision

[Balling]

Summary of the bug: as you can read here ​https://www.ffmpeg.org/ffmpeg-bitstream-filters.html#h264_005fmetadata H.264 has time_scale / num_units_in_tick in the VUI parameters. That is how one writes frame rate (besides timing SEI that is) in Annex B files. The problem is that (not touching very broken mkv here with only ms precision and only PTS in the form of duration, no DTS) when you go to mp4 you are supposed to make it CFR and use the nice tbn. But even if you will use vfrdet on annex b file it will not be CFR. This sample in fact has even fixed_frame_rate_flag set! So this is a bug, ffplay also cannot even seek in this annex b file! Sample is here: ​https://github.com/wang-bin/QtAV/files/1132086/example.zip

How to reproduce:

ffmpeg -i  example.h264 -vf vfrdet -f null NUL
ffmpeg version N-107098-g4d45f5acbd-20220613 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.2.0 (crosstool-NG 1.24.0.533_681aaef)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --enable-shared --disable-static --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20220613
  libavutil      57. 26.100 / 57. 26.100
  libavcodec     59. 33.100 / 59. 33.100
  libavformat    59. 24.100 / 59. 24.100
  libavdevice    59.  6.100 / 59.  6.100
  libavfilter     8. 40.100 /  8. 40.100
  libswscale      6.  6.100 /  6.  6.100
  libswresample   4.  6.100 /  4.  6.100
  libpostproc    56.  5.100 / 56.  5.100
Input #0, h264, from 'example.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1200k tbn
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'NUL':
  Metadata:
    encoder         : Lavf59.24.100
  Stream #0:0: Video: wrapped_avframe, yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn
    Metadata:
      encoder         : Lavc59.33.100 wrapped_avframe
frame= 1082 fps=912 q=-0.0 Lsize=N/A time=00:00:36.10 bitrate=N/A speed=30.4x
video:499kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_vfrdet_0 @ 000001a582b7f900] VFR:0.799260 (864/217) min: 40040 max: 40041 avg: 40040

What it should be as if -r 29.97002997 was before input.

In fact Mediainfo can see all of it (both the timetamps of mp4 and time_scale / num_units_in_tick) in -c copy to mp4 with -r 29.97002997 and without.

There were quite a lot of similar bugs in that regard, but Annex B to mp4 is a clear case and is a clear bug.

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Balling]

What I want is the file be default be as if ffmpeg.exe -r 30000/1001 -i example.h264 -c copy fnmafa3.mp4

is used...

For that you need to increase 30000/1001 precision.

[Elon Musk]

Issue is that your timebase is generic one that can not reproduce your fps perfectly, and .h264 does not store pts or timebase at all.

ffmpeg -i example.h264 -vf settb=1001/30000,vfrdet -f null -

This shows that file is "CFR", but that is meaningless as format never store timestamps or timebase, its just put by ffmpeg code so it can process it somehow.

[Balling]

Issue is that your timebase is generic one that can not reproduce your fps perfectly

I need 30k tbn, not 1200k tbn BY DEFAULT. Nice that you understand. Changing to feature request.

Why does it report

[mp4 @ 0000022d9775a040] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

That is just strange. What does "Fix your code" mean??

and .h264 does not store pts or timebase at all.

It does have timing SEI, but not in my sample. Anyway, it sets fixed_frame_rate_flag, yet after this genius remux it has CFR with a different frame rate. Like how strange is that.

Your command with -c copy reports:

Filtergraph 'settb=1001/30000' was defined for video output stream 0:0 but codec copy was selected.
Filtering and streamcopy cannot be used together.

So that is useless.

[Balling]

Issue is that your timebase is generic one

But we have fixed_frame_rate_flag set. So in reality we can assume it is CFR and THAT IS WHAT mov/mp4 muxer does. It sets Frame rate mode: Constant. Okay? Yet, because of bad code somewhere, it does paste 29.97 instead of 29.97002997. Okay?? This is what was happening with drop-frame timecode! That is why it is not used anymore.

That is why I said

For that you need to increase 30000/1001 precision.

[Elon Musk]

bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
scale=1000
30000/1001
29.97002997002997002997002997002997002997002997002997002997002997002\
99700299700299700299700299700299700299700299700299700299700299700299\
70029970029970029970029970029970029970029970029970029970029970029970\
02997002997002997002997002997002997002997002997002997002997002997002\
99700299700299700299700299700299700299700299700299700299700299700299\
70029970029970029970029970029970029970029970029970029970029970029970\
02997002997002997002997002997002997002997002997002997002997002997002\
99700299700299700299700299700299700299700299700299700299700299700299\
70029970029970029970029970029970029970029970029970029970029970029970\
02997002997002997002997002997002997002997002997002997002997002997002\
99700299700299700299700299700299700299700299700299700299700299700299\
70029970029970029970029970029970029970029970029970029970029970029970\
02997002997002997002997002997002997002997002997002997002997002997002\
99700299700299700299700299700299700299700299700299700299700299700299\
700299700299700299700299700299700299700299700299700

So timebase could be set to reverse of detected framerate. Need to inspect this.

[Balling]

That is just 29.(970029), periodic... The point is at some moment this becomes 29.97 which is wrong, and it is very simple to isolate as opposed to the whole mp4 to mp4 demux.

[Balling]

So can you make 30k tbn the default in this case. Please?

[Balling]

Please?

[Balling]

[trace_headers @ 0000027e73374480] 119 num_units_in_tick 00000000000000000000111110100100 = 4004
[trace_headers @ 0000027e73374480] 151 time_scale 00000000000000111010100110000000 = 240000
[trace_headers @ 0000027e73374480] 183 fixed_frame_rate_flag 1 = 1

My file has and that means 59.94 tbr is wrong. It is 59.9400599400... tbr, not 59.94 tbr.

[Balling]

Something broke recently, and this file is somehow recognized as 25 fps, which is wrong. Seriously?

[MasterQuestionable]

͏    I fear this is actually none issue:
͏    No system can be of infinite precision.
͏    And the actual playback experience mostly wouldn't differ. [1]
͏    Much comparable to those use ns accuracy for regular timestamps...

͏    See also: ​https://github.com/MasterInQuestion/talk/discussions/3#issuecomment-1546820018-3

͏    As for the recent further break [ #11192 ]: much the result of "Codec container wrestling each other..."

[ [1]
͏    Math, if 30,000/1,001 did become 29.97:
͏    The difference would be:
͏    1 / (30,000/1,001 - 29.97) * 29.97
͏    = 999,999 s (11::13:46:39)
͏    .
͏    Which means 1 s difference (desync) every such interval. ]

͏    But probably the 29.97 is just display error?..
͏    Real presentation time data may not be such simple. (effective VFR)

͏    Probably similar:
͏    ​https://trac.ffmpeg.org/ticket/11055#comment:48

[Balling]

No system can be of infinite precision.

LOL
​https://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic

As for the recent further break: much the result of "Codec container wrestling each other..."

There is no container here. That is the point

[MasterQuestionable]

͏    Learn entropy theory.
͏    Infinite precision demands infinite resource:
͏    Something impossible to practical systems.

͏    The point is, container may also deliver presentation info:
͏    And which is generally more favored.

[Balling]

"Infinite precision demands infinite resource"

No, it does not. E.g. you can calculate each digit of pi without relying on previous digits in constand time.

[MasterQuestionable]

͏    Where to store the infinite result?
͏    Discard outright..?

[Balling]

Where to store the infinite result

Erm, the 29.970 is what was used when Timecode was used as main timestamp information. It was accumuating 1 frame every 9 hours 15 minutes because it was saying 29.970 instead of actual 30/1.001. Timecodes were corrected every time.

This is a rather basic scenario, and literally what this issue is

[MasterQuestionable]

͏    <&~~>Your math is wrong:
͏    Mistook calculated number of frames as time (in s).</&>

͏    Pardon the previous Math on Mars...
͏    Check comment:21.

[Balling]

I am just quoting wikipedia. ​https://en.wikipedia.org/wiki/SMPTE_timecode#cite_ref-5

This says 1 frame every 9 hours 15 minutes.

[Balling]

Achieving 30 × 0.999 = 29.97 frame/s. This is very slightly slower than the true NTSC frame rate of ⁠
"30
/
1.001
⁠ = 29.970029 frame/s. The difference is one additional NTSC frame per 1,000,000 timecode frames, a residual timing error of 1.0 ppm or roughly 2.6 frames (86.4 milliseconds) per day which is considered negligible."

It was not negligible, BTW, TV channels had to actually correct this.

[MasterQuestionable]

͏    Pardon. My math went awry.
͏    It seems...
͏    1/${FPS} not 1/${Time}.

͏    Now fixed.

͏    ----

͏    Negligible or not, no system can be perfectly CFR.
͏    The correction regardless has to occur.

[Balling]

Negligible or not, no system can be perfectly CFR.

In mkv with milliseconds it cannot be, but in mov/mp4 it can be (it literally is if you do ffmpeg.exe -r 30000/1001 -i example.h264 -c copy fnmafa3.mp4), microseconds.

[MasterQuestionable]

͏    Actual a repetition of...
͏    ​https://trac.ffmpeg.org/ticket/11055#comment:58

͏    "Concerning actual playback experience":
͏    Wouldn't your system ever jitter or drop frame..? How to ensure that?

[Balling]

The frames are dropped because of this bug... This is not Hollywood when 24.000 fps gets converted to 24/1.001 with an optical flow.

Chrome and others just drop frames. But dedicated apps do not, like Youtube. If you ever tried Youtube app on a TV 0 frames are dropped, that is because LG tvs can switch from 24 to 24/1.001 and so do not have to drop any frames ever.

[Balling]

"Wouldn't your system ever jitter or drop frame..?"

Jitter is introduced by mkv container, that has millisecond durations, no PTS or DTS. In mkv fork webm they actually introduced some funny way how to signal constant frame rate, apparently bug in firefox that it does not support that in youtube, lol ​https://bugzilla.mozilla.org/show_bug.cgi?id=1903466

In practice some files on Youtube somehow have this strange issue with 29970/1000 fps instead of 30/1.001, maybe this bug.

[MasterQuestionable]

͏    Even if targeted specific frame rate: maintaining which constantly may not be practical.
͏    And absolute CFR is simply impossible: that demands absolute precision, no system could ever be capable of.

͏    And delay within 10 ms is generally indiscernable to human.

[Balling]

Your commit broke Annex B fps propogation. Please fix it, Anton. And fix it so that it is 30/1.001, not 29970/1000.

[Gyan]

Which commit?

[Balling]

Some commit right before 9 July. 10185e2d4c1e9839bc58a1d6a63c861677b13fd0 from the look of it. But that was was apparently fixed, in fact

.\bin\ffmpeg -i example.h264 -vf vfrdet -f null NUL

shows constant framerate now, still not written with correct 30k tbn so that you get 30/1.001.

[Balling]

͏ And absolute CFR is simply impossible: that demands absolute precision, no system could ever be capable of.

Erm, you can just set the flag saying it is constant framerate. Whether on some segment of the file or on the whole file. And again, maybe you have terrible HW decoders, but any NORMAL decoder will never drop frames, unless that is needed due to frame rate mismatch.

[MasterQuestionable]

͏    Similar to the predicament of 100% reliability... Possible?