24fps nut file converts to 24.39fps mkv?

[Laurence Tratt]

I created a simple video creation system that uses ffmpeg for everything
(recording, editing, etc) called Aeschylus ​https://github.com/ltratt/aeschylus

As part of that I seem to have found a case where ffmpeg 4.3.2 converts a 24fps
nut file to a 24.39fps mkv although, weirdly, it seems to lead to a 24fps mp4.
You can see the problem with the video at ​https://tratt.net/scene_1.nut

Output from ffprobe for the original .nut file:

  Input #0, nut, from 'scene_1.nut':
    Metadata:
      encoder         : Lavf58.45.100
    Duration: 00:01:08.49, start: 0.000000, bitrate: 7657 kb/s
      Stream #0:0: Audio: flac ([172][241][0][0] / 0xF1AC), 44100 Hz, mono, s16 (default)
      Metadata:
        encoder         : Lavc58.91.100 flac
      Stream #0:1: Video: h264 (High 4:4:4 Predictive) (H264 / 0x34363248), yuv420p, 1920x1080, 24 fps, 24 tbr, 49152 tbn, 48 tbc (default)
      Metadata:
        encoder         : Lavc58.91.100 libx264

and then if we look at the first couple of video packets we can see that each
has a duration of 0.041667 seconds (i.e. 1/24 of a second):

  frames.frame.0.media_type="video"
  frames.frame.0.stream_index=1
  frames.frame.0.key_frame=1
  frames.frame.0.pkt_pts=0
  frames.frame.0.pkt_pts_time="0.000000"
  frames.frame.0.pkt_dts=0
  frames.frame.0.pkt_dts_time="0.000000"
  frames.frame.0.best_effort_timestamp=0
  frames.frame.0.best_effort_timestamp_time="0.000000"
  frames.frame.0.pkt_duration=2048
  frames.frame.0.pkt_duration_time="0.041667"
  frames.frame.0.pkt_pos="1328"
  frames.frame.0.pkt_size="239333"
  frames.frame.0.width=1920
  frames.frame.0.height=1080
  frames.frame.0.pix_fmt="yuv420p"
  frames.frame.0.sample_aspect_ratio="1:1"
  frames.frame.0.pict_type="I"
  frames.frame.0.coded_picture_number=0
  frames.frame.0.display_picture_number=0
  frames.frame.0.interlaced_frame=0
  frames.frame.0.top_field_first=0
  frames.frame.0.repeat_pict=0
  frames.frame.0.color_range="unknown"
  frames.frame.0.color_space="unknown"
  frames.frame.0.color_primaries="unknown"
  frames.frame.0.color_transfer="unknown"
  frames.frame.0.chroma_location="left"
  frames.frame.1.media_type="video"
  frames.frame.1.stream_index=1
  frames.frame.1.key_frame=0
  frames.frame.1.pkt_pts=2048
  frames.frame.1.pkt_pts_time="0.041667"
  frames.frame.1.pkt_dts=2048
  frames.frame.1.pkt_dts_time="0.041667"
  frames.frame.1.best_effort_timestamp=2048
  frames.frame.1.best_effort_timestamp_time="0.041667"
  frames.frame.1.pkt_duration=2048
  frames.frame.1.pkt_duration_time="0.041667"
  frames.frame.1.pkt_pos="241483"
  frames.frame.1.pkt_size="215575"
  frames.frame.1.width=1920
  frames.frame.1.height=1080
  frames.frame.1.pix_fmt="yuv420p"
  frames.frame.1.sample_aspect_ratio="1:1"
  frames.frame.1.pict_type="P"
  frames.frame.1.coded_picture_number=1
  frames.frame.1.display_picture_number=0
  frames.frame.1.interlaced_frame=0
  frames.frame.1.top_field_first=0
  frames.frame.1.repeat_pict=0
  frames.frame.1.color_range="unknown"
  frames.frame.1.color_space="unknown"
  frames.frame.1.color_primaries="unknown"
  frames.frame.1.color_transfer="unknown"
  frames.frame.1.chroma_location="left"

If we convert it into an mp4 with:

  $ ffmpeg -i scene_1.nut out.mp4

we get:

  Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'out.mp4':
    Metadata:
      major_brand     : isom
      minor_version   : 512
      compatible_brands: isomiso2avc1mp41
      encoder         : Lavf58.45.100
    Duration: 00:01:08.51, start: 0.000000, bitrate: 320 kb/s
      Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 244 kb/s, 24 fps, 24 tbr, 12288 tbn, 48 tbc (default)
      Metadata:
        handler_name    : VideoHandler
      Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 69 kb/s (default)
      Metadata:
        handler_name    : SoundHandler

and the first couple of packets again we have frames 1/24 of a second long:

  frames.frame.0.media_type="video"
  frames.frame.0.stream_index=0
  frames.frame.0.key_frame=1
  frames.frame.0.pkt_pts=0
  frames.frame.0.pkt_pts_time="0.000000"
  frames.frame.0.pkt_dts=0
  frames.frame.0.pkt_dts_time="0.000000"
  frames.frame.0.best_effort_timestamp=0
  frames.frame.0.best_effort_timestamp_time="0.000000"
  frames.frame.0.pkt_duration=512
  frames.frame.0.pkt_duration_time="0.041667"
  frames.frame.0.pkt_pos="48"
  frames.frame.0.pkt_size="15038"
  frames.frame.0.width=1920
  frames.frame.0.height=1080
  frames.frame.0.pix_fmt="yuv420p"
  frames.frame.0.sample_aspect_ratio="N/A"
  frames.frame.0.pict_type="I"
  frames.frame.0.coded_picture_number=0
  frames.frame.0.display_picture_number=0
  frames.frame.0.interlaced_frame=0
  frames.frame.0.top_field_first=0
  frames.frame.0.repeat_pict=0
  frames.frame.0.color_range="unknown"
  frames.frame.0.color_space="unknown"
  frames.frame.0.color_primaries="unknown"
  frames.frame.0.color_transfer="unknown"
  frames.frame.0.chroma_location="left"
  frames.frame.1.media_type="video"
  frames.frame.1.stream_index=0
  frames.frame.1.key_frame=0
  frames.frame.1.pkt_pts=512
  frames.frame.1.pkt_pts_time="0.041667"
  frames.frame.1.pkt_dts=512
  frames.frame.1.pkt_dts_time="0.041667"
  frames.frame.1.best_effort_timestamp=512
  frames.frame.1.best_effort_timestamp_time="0.041667"
  frames.frame.1.pkt_duration=512
  frames.frame.1.pkt_duration_time="0.041667"
  frames.frame.1.pkt_pos="19190"
  frames.frame.1.pkt_size="254"
  frames.frame.1.width=1920
  frames.frame.1.height=1080
  frames.frame.1.pix_fmt="yuv420p"
  frames.frame.1.sample_aspect_ratio="N/A"
  frames.frame.1.pict_type="B"
  frames.frame.1.coded_picture_number=3
  frames.frame.1.display_picture_number=0
  frames.frame.1.interlaced_frame=0
  frames.frame.1.top_field_first=0
  frames.frame.1.repeat_pict=0
  frames.frame.1.color_range="unknown"
  frames.frame.1.color_space="unknown"
  frames.frame.1.color_primaries="unknown"
  frames.frame.1.color_transfer="unknown"
  frames.frame.1.chroma_location="left"

However if we convert it to a .mkv:

  $ ffmpeg -i scene_1.nut out.mkv

then we get:

  Input #0, matroska,webm, from 'out.mkv':
    Metadata:
      ENCODER         : Lavf58.45.100
    Duration: 00:01:08.49, start: 0.000000, bitrate: 322 kb/s
      Stream #0:0: Video: h264 (High), yuv420p(progressive), 1920x1080, 24 fps, 24 tbr, 1k tbn, 48 tbc (default)
      Metadata:
        ENCODER         : Lavc58.91.100 libx264
        DURATION        : 00:01:08.459000000
      Stream #0:1: Audio: vorbis, 44100 Hz, mono, fltp (default)
      Metadata:
        ENCODER         : Lavc58.91.100 libvorbis
        DURATION        : 00:01:08.488000000

and the first two packets have a duration of 0.41000 seconds (roughly
equivalent to 24.36fps):

  frames.frame.0.media_type="video"
  frames.frame.0.stream_index=0
  frames.frame.0.key_frame=1
  frames.frame.0.pkt_pts=0
  frames.frame.0.pkt_pts_time="0.000000"
  frames.frame.0.pkt_dts=0
  frames.frame.0.pkt_dts_time="0.000000"
  frames.frame.0.best_effort_timestamp=0
  frames.frame.0.best_effort_timestamp_time="0.000000"
  frames.frame.0.pkt_duration=41
  frames.frame.0.pkt_duration_time="0.041000"
  frames.frame.0.pkt_pos="4124"
  frames.frame.0.pkt_size="15038"
  frames.frame.0.width=1920
  frames.frame.0.height=1080
  frames.frame.0.pix_fmt="yuv420p"
  frames.frame.0.sample_aspect_ratio="N/A"
  frames.frame.0.pict_type="I"
  frames.frame.0.coded_picture_number=0
  frames.frame.0.display_picture_number=0
  frames.frame.0.interlaced_frame=0
  frames.frame.0.top_field_first=0
  frames.frame.0.repeat_pict=0
  frames.frame.0.color_range="unknown"
  frames.frame.0.color_space="unknown"
  frames.frame.0.color_primaries="unknown"
  frames.frame.0.color_transfer="unknown"
  frames.frame.0.chroma_location="left"
  frames.frame.1.media_type="video"
  frames.frame.1.stream_index=0
  frames.frame.1.key_frame=0
  frames.frame.1.pkt_pts=42
  frames.frame.1.pkt_pts_time="0.042000"
  frames.frame.1.pkt_dts=42
  frames.frame.1.pkt_dts_time="0.042000"
  frames.frame.1.best_effort_timestamp=42
  frames.frame.1.best_effort_timestamp_time="0.042000"
  frames.frame.1.pkt_duration=41
  frames.frame.1.pkt_duration_time="0.041000"
  frames.frame.1.pkt_pos="23299"
  frames.frame.1.pkt_size="254"
  frames.frame.1.width=1920
  frames.frame.1.height=1080
  frames.frame.1.pix_fmt="yuv420p"
  frames.frame.1.sample_aspect_ratio="N/A"
  frames.frame.1.pict_type="B"
  frames.frame.1.coded_picture_number=3
  frames.frame.1.display_picture_number=0
  frames.frame.1.interlaced_frame=0
  frames.frame.1.top_field_first=0
  frames.frame.1.repeat_pict=0
  frames.frame.1.color_range="unknown"
  frames.frame.1.color_space="unknown"
  frames.frame.1.color_primaries="unknown"
  frames.frame.1.color_transfer="unknown"
  frames.frame.1.chroma_location="left"

Notice also that the timestamp of the second packet is 0.042000. It feels like,
perhaps, the duration of the first frame is being rounded down but the
timestamp of the second frame is being rounded up? This video appears notably
desynchronised when played, presumably because of this problem.

It is quite probable that I'm doing something stupid somewhere in the pipeline,
but I suspect this is still a surprising outcome?

Laurie

[Balling]

mkv just has worse timestamp precision than mp4. That will fix mkv, but I am not so sure what will support it outside of ffmpeg. ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210113174623.1397914-1-tfoucu@gmail.com/

Why not just convert to mp4 and then -c copy to mkv? Also you need to check for variable framerate.

[Balling]

Yeah, it is indeed a bug. If you will then -c copy that broken mkv to mp4 it will show:

Frame rate mode             : Variable
Frame rate                  : 24.000 FPS
Minimum frame rate          : 23.634 FPS
Maximum frame rate          : 24.390 FPS

that IS wrong and does not happen with nut convert to mp4 -c copy to mkv -c copy to mp4 and happens because (as ffmpeg warns on that broken mkv -c copy to mp4 ):

[mp4 @ 0000014f82a864c0] track 1: codec frame size is not set

[Laurence Tratt]

Thanks for looking into this! There seems to be something even weirder going on than I first realised. mpv seems capable of playing at least the nut and mp4 files with proper synchronisation (to my eyes it drifts slightly with the mkv, but I might be seeing double at this point!). vlc, however, struggles with all of them (nut, mkv, mp4), drifting noticeably over time. So does youtube, for which the audio is heavily (unusably!) out of sync for the nut, mkv, and mp4 files. I have no idea if this has any bearing on this issue or not, but I thought it might be worth mentioning!

[Carl Eugen Hoyos]

Please test current FFmpeg git head - the only version supported here - and provide the command line that allows to reproduce the issue together with the complete, uncut console output to make this a valid ticket.

[Carl Eugen Hoyos]

Replying to Balling:

Please stop setting tickets to open where all necessary information is missing!

[Balling]

There are so many things wrong with this: while mp4 converts FLAC (from your source nut file) to aac, mkv converts it to Vorbis (also mp4 does not support flac, only with -strict -2 -c copy it will work, while mkv converts it to Vorbis of all things, while it supports flac). But because flac from your file is very interesting as it is longer than video:

Stream #0:0: Video: h264 (High 4:4:4 Predictive), yuv420p(progressive), 1920x1080, 24 fps, 24 tbr, 1k tbn, 48 tbc (default)
    Metadata:
      ENCODER         : Lavc58.91.100 libx264
      DURATION        : 00:01:08.459000000
  Stream #0:1: Audio: flac, 44100 Hz, mono, s16 (default)
    Metadata:
      ENCODER         : Lavc58.91.100 flac
      DURATION        : 00:01:08.470000000

it breaks mkv. Well... I suppose this is WontFix.

Please test current FFmpeg git head - the only version supported here

I already did it. Really?

[Carl Eugen Hoyos]

Please reopen this ticket if you can provide the command line you tested together with the complete, uncut console output.

[Laurence Tratt]

Here's a version tested against the latest FFmpeg from git:

$ wget https://tratt.net/scene_1.nut
$ ./ffmpeg -i scene_1.nut out.mkv
ffmpeg version N-101291-gd3d99a0a06 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: 
  libavutil      56. 66.100 / 56. 66.100
  libavcodec     58.125.100 / 58.125.100
  libavformat    58. 68.100 / 58. 68.100
  libavdevice    58. 12.100 / 58. 12.100
  libavfilter     7.107.100 /  7.107.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
Input #0, nut, from 'scene_1.nut':
  Metadata:
    encoder         : Lavf58.45.100
  Duration: 00:01:08.49, start: 0.000000, bitrate: 7657 kb/s
  Stream #0:0: Audio: flac ([172][241][0][0] / 0xF1AC), 44100 Hz, mono, s16 (default)
    Metadata:
      encoder         : Lavc58.91.100 flac
  Stream #0:1: Video: h264 (High 4:4:4 Predictive) (H264 / 0x34363248), yuv420p, 1920x1080, 24 fps, 24 tbr, 49152 tbn, 48 tbc (default)
    Metadata:
      encoder         : Lavc58.91.100 libx264
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (native) -> mpeg4 (native))
  Stream #0:0 -> #0:1 (flac (native) -> ac3 (native))
Press [q] to stop, [?] for help
Output #0, matroska, to 'out.mkv':
  Metadata:
    encoder         : Lavf58.68.100
  Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p(progressive), 1920x1080, q=2-31, 200 kb/s, 24 fps, 1k tbn (default)
    Metadata:
      encoder         : Lavc58.125.100 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
  Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, mono, fltp (16 bit), 96 kb/s (default)
    Metadata:
      encoder         : Lavc58.125.100 ac3
Error while decoding stream #0:0: Invalid argument:01:02.44 bitrate= 772.4kbits/s speed=13.6x    
frame= 1643 fps=327 q=31.0 Lsize=    6737kB time=00:01:08.47 bitrate= 806.1kbits/s speed=13.6x    
video:5904kB audio:803kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.455408%
$ ./ffprobe -i out.mkv -select_streams v:0 -print_format flat -show_frames 2>&1 | head -n 75
ffprobe version N-101291-gd3d99a0a06 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: 
  libavutil      56. 66.100 / 56. 66.100
  libavcodec     58.125.100 / 58.125.100
  libavformat    58. 68.100 / 58. 68.100
  libavdevice    58. 12.100 / 58. 12.100
  libavfilter     7.107.100 /  7.107.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
Input #0, matroska,webm, from 'out.mkv':
  Metadata:
    ENCODER         : Lavf58.68.100
  Duration: 00:01:08.51, start: 0.000000, bitrate: 805 kb/s
  Stream #0:0: Video: mpeg4 (Simple Profile), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 24 fps, 24 tbr, 1k tbn, 24 tbc (default)
    Metadata:
      ENCODER         : Lavc58.125.100 mpeg4
      DURATION        : 00:01:08.462000000
  Stream #0:1: Audio: ac3, 44100 Hz, mono, fltp, 96 kb/s (default)
    Metadata:
      ENCODER         : Lavc58.125.100 ac3
      DURATION        : 00:01:08.511000000
frames.frame.0.media_type="video"
frames.frame.0.stream_index=0
frames.frame.0.key_frame=1
frames.frame.0.pkt_pts=3
frames.frame.0.pkt_pts_time="0.003000"
frames.frame.0.pkt_dts=3
frames.frame.0.pkt_dts_time="0.003000"
frames.frame.0.best_effort_timestamp=3
frames.frame.0.best_effort_timestamp_time="0.003000"
frames.frame.0.pkt_duration=41
frames.frame.0.pkt_duration_time="0.041000"
frames.frame.0.pkt_pos="1224"
frames.frame.0.pkt_size="36139"
frames.frame.0.width=1920
frames.frame.0.height=1080
frames.frame.0.pix_fmt="yuv420p"
frames.frame.0.sample_aspect_ratio="1:1"
frames.frame.0.pict_type="I"
frames.frame.0.coded_picture_number=0
frames.frame.0.display_picture_number=0
frames.frame.0.interlaced_frame=0
frames.frame.0.top_field_first=0
frames.frame.0.repeat_pict=0
frames.frame.0.color_range="unknown"
frames.frame.0.color_space="unknown"
frames.frame.0.color_primaries="unknown"
frames.frame.0.color_transfer="unknown"
frames.frame.0.chroma_location="left"
frames.frame.1.media_type="video"
frames.frame.1.stream_index=0
frames.frame.1.key_frame=0
frames.frame.1.pkt_pts=45
frames.frame.1.pkt_pts_time="0.045000"
frames.frame.1.pkt_dts=45
frames.frame.1.pkt_dts_time="0.045000"
frames.frame.1.best_effort_timestamp=45
frames.frame.1.best_effort_timestamp_time="0.045000"
frames.frame.1.pkt_duration=41
frames.frame.1.pkt_duration_time="0.041000"
frames.frame.1.pkt_pos="37795"
frames.frame.1.pkt_size="10590"
frames.frame.1.width=1920
frames.frame.1.height=1080
frames.frame.1.pix_fmt="yuv420p"
frames.frame.1.sample_aspect_ratio="1:1"
frames.frame.1.pict_type="P"
frames.frame.1.coded_picture_number=1
frames.frame.1.display_picture_number=0
frames.frame.1.interlaced_frame=0
frames.frame.1.top_field_first=0
frames.frame.1.repeat_pict=0
frames.frame.1.color_range="unknown"
frames.frame.1.color_space="unknown"

This is giving me the same pkt_duration_times as before, but slightly different best_effort_timestamp_times -- both though, I think, still incorrect.

[Carl Eugen Hoyos]

You cannot divide 1000 / 24
Why do you think FFmpeg can improve this?

[Balling]

Replying to cehoyos:

You cannot divide 1000 / 24
Why do you think FFmpeg can improve this?

You can, obviously, even in ms (as in mkv, but then again that will be fixed sooner or later), both for integer 24 and non-integer 24/1.001. This issue has nothing to with this. It has everything to do with the fact it tries to resize flac, affecting video and doing video variable framerate (and it was NOT VFR, you can check with -vf vfrdet -an -f null -).

[Balling]

So I further checked this one, it has nothing to do with flac, if I add -an it is still a problem.

I suppose that is just the way it is: all mkv's with either variant of 24p are seen as VFR, which is very wrong of course. See #7927. Also see #8909.

[Carl Eugen Hoyos]

I am not convinced that this can be improved in FFmpeg.

[Balling]

Replying to cehoyos:

I am not convinced that this can be improved in FFmpeg.

There is a standard way of rounding to preserve timetsamps (or more accurately time durations) in mkv (mkv also does not have DTS, only PTS). I suppose, you can look into Mkvtoolnix or even reference mkv code for that.

While there, our mail in this trac mailing list did cause at least two patches in devel. See:

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/1616537325-620485-1-git-send-email-andrey.rikunov@gmail.com/
​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210318030238.561057-1-jerome@percipient.ai/

[Carl Eugen Hoyos]

Neither is acceptable though...