Raw bitstream export/import makes video stuttery

[Linus Kardell]

Summary of the bug:
Trying to export and import a raw h.264 bitstream as described in ​https://trac.ffmpeg.org/wiki/How%20to%20speed%20up%20/%20slow%20down%20a%20video#rawbitstreammethod makes the video extremely stuttery when viewing it in a browser. The resulting file plays fine when viewing in mpv or VLC, but if I play it in Firefox it looks like it's running at 5 FPS or something and if I play it in Chrome it jitters back and forth like the frames are played out of order.
How to reproduce:

linus@feja:/hdd/linus/Hämtningar> ffmpeg -i big_buck_bunny_480p_h264.mov -map 0:v -c:v copy -bsf:v h264_mp4toannexb raw.h264
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14 (SUSE Linux)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --incdir=/usr/include/ffmpeg --extra-cflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --optflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --enable-version3 --enable-libsmbclient --disable-openssl --enable-gnutls --enable-ladspa --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdc1394 --enable-libdrm --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libjack --enable-libjxl --enable-librist --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopenh264-dlopen --enable-libopus --enable-libpulse --enable-librav1e --enable-librubberband --enable-libsvtav1 --enable-libsoxr --enable-libspeex --enable-libssh --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lto --enable-lv2 --enable-libvpl --enable-vaapi --enable-vdpau --enable-version3 --enable-libfdk-aac-dlopen --enable-nonfree --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libx264 --enable-libx265 --enable-libxvid
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'big_buck_bunny_480p_h264.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2008-05-27T18:32:32.000000Z
    com.apple.quicktime.player.movie.audio.gain: 1.000000
    com.apple.quicktime.player.movie.audio.treble: 0.000000
    com.apple.quicktime.player.movie.audio.bass: 0.000000
    com.apple.quicktime.player.movie.audio.balance: 0.000000
    com.apple.quicktime.player.movie.audio.pitchshift: 0.000000
    com.apple.quicktime.player.movie.audio.mute: 
    com.apple.quicktime.player.movie.visual.brightness: 0.000000
    com.apple.quicktime.player.movie.visual.color: 1.000000
    com.apple.quicktime.player.movie.visual.tint: 0.000000
    com.apple.quicktime.player.movie.visual.contrast: 1.000000
    com.apple.quicktime.player.version: 7.4.1 (14)
    com.apple.quicktime.version: 7.4.1 (14) 0x7418000 (Mac OS X, 10.5.2, 9C31)
    timecode        : 00:00:00:00
  Duration: 00:09:56.46, start: 0.000000, bitrate: 3342 kb/s
  Stream #0:0[0x1](eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 853x480, 2899 kb/s, 24 fps, 24 tbr, 2400 tbn (default)
      Metadata:
        creation_time   : 2008-05-27T18:32:32.000000Z
        handler_name    : Apple Video Media Handler
        vendor_id       : appl
        encoder         : H.264
      Side data:
        displaymatrix: rotation of -0.00 degrees
  Stream #0:1[0x2](eng): Data: none (tmcd / 0x64636D74) (default)
      Metadata:
        creation_time   : 2008-05-27T18:32:32.000000Z
        handler_name    : Time Code Media Handler
        timecode        : 00:00:00:00
  Stream #0:2[0x4](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 437 kb/s (default)
      Metadata:
        creation_time   : 2008-05-27T18:32:32.000000Z
        handler_name    : Apple Sound Media Handler
        vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Output #0, h264, to 'raw.h264':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    timecode        : 00:00:00:00
    com.apple.quicktime.player.movie.audio.gain: 1.000000
    com.apple.quicktime.player.movie.audio.treble: 0.000000
    com.apple.quicktime.player.movie.audio.bass: 0.000000
    com.apple.quicktime.player.movie.audio.balance: 0.000000
    com.apple.quicktime.player.movie.audio.pitchshift: 0.000000
    com.apple.quicktime.player.movie.audio.mute: 
    com.apple.quicktime.player.movie.visual.brightness: 0.000000
    com.apple.quicktime.player.movie.visual.color: 1.000000
    com.apple.quicktime.player.movie.visual.tint: 0.000000
    com.apple.quicktime.player.movie.visual.contrast: 1.000000
    com.apple.quicktime.player.version: 7.4.1 (14)
    com.apple.quicktime.version: 7.4.1 (14) 0x7418000 (Mac OS X, 10.5.2, 9C31)
    encoder         : Lavf61.7.100
  Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 853x480, q=2-31, 2899 kb/s, 24 fps, 24 tbr, 2400 tbn (default)
      Metadata:
        creation_time   : 2008-05-27T18:32:32.000000Z
        handler_name    : Apple Video Media Handler
        vendor_id       : appl
        encoder         : H.264
      Side data:
        displaymatrix: rotation of -0.00 degrees
Press [q] to stop, [?] for help
[out#0/h264 @ 0x561cd3297140] video:211089KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.000000%
frame=14315 fps=0.0 q=-1.0 Lsize=  211089KiB time=00:09:56.41 bitrate=2899.4kbits/s speed=4.07e+03x    
linus@feja:/hdd/linus/Hämtningar> ffmpeg -fflags +genpts -r 24 -i raw.h264 -c:v copy output.mp4
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14 (SUSE Linux)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --incdir=/usr/include/ffmpeg --extra-cflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --optflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --enable-version3 --enable-libsmbclient --disable-openssl --enable-gnutls --enable-ladspa --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdc1394 --enable-libdrm --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libjack --enable-libjxl --enable-librist --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopenh264-dlopen --enable-libopus --enable-libpulse --enable-librav1e --enable-librubberband --enable-libsvtav1 --enable-libsoxr --enable-libspeex --enable-libssh --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lto --enable-lv2 --enable-libvpl --enable-vaapi --enable-vdpau --enable-version3 --enable-libfdk-aac-dlopen --enable-nonfree --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libx264 --enable-libx265 --enable-libxvid
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, h264, from 'raw.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 854x480, 25 fps, 1200k tbr, 1200k tbn
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf61.7.100
  Stream #0:0: Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 854x480, q=2-31, 24 fps, 1200k tbr, 12288 tbn
Press [q] to stop, [?] for help
[mp4 @ 0x55d7a0abb7c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0x55d7a0abb7c0] pts has no value
    Last message repeated 14314 times
[out#0/mp4 @ 0x55d7a0ab7740] video:211089KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.062597%
frame=14315 fps=0.0 q=-1.0 Lsize=  211222KiB time=N/A bitrate=N/A speed=N/A

[MasterQuestionable]

͏    I doubt raw streams are really supported in browsers...
͏    ​https://github.com/richtr/NoSleep.js/issues/157#issuecomment-1529149759
͏    "Browsers for security considerations may choose to only support a limited set of medias to reduce attack surface."

[James]

Replying to MasterQuestionable:

I doubt raw streams are really supported in browsers...

That's not what he's trying to do.

[Linus Kardell]

I'm playing the resulting output.mp4

[MasterQuestionable]

͏    Reinterpreted:
͏    ffmpeg -i "big_buck_bunny_480p_h264.mov" -map 0:v -c:v copy -bsf:v h264_mp4toannexb "raw.h264"
͏    ffmpeg -fflags +genpts -r 24 -i "raw.h264" -c:v copy "output.mp4"

͏    Mostly something wrong with the generated timestamps.

[Balling]

Does it happen with mkv, rename mkv to mp4, because Chrome no longer opens mkv extension.

[Linus Kardell]

I can't import the raw bitstream directly into mkv, I get:

[matroska @ 0x55fd2f62e4c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[matroska @ 0x55fd2f62e4c0] Can't write packet with unknown timestamp
[vost#0:0/copy @ 0x55fd2f8d0f00] Error submitting a packet to the muxer: Invalid argument
    Last message repeated 1 times
[out#0/matroska @ 0x55fd2f62a440] Error muxing a packet
[out#0/matroska @ 0x55fd2f62a440] Task finished with error code: -22 (Invalid argument)
[out#0/matroska @ 0x55fd2f62a440] Terminating thread with return code -22 (Invalid argument)
[out#0/matroska @ 0x55fd2f62a440] video:1KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=    1 fps=0.0 q=-1.0 Lsize=       1KiB time=N/A bitrate=N/A speed=N/A    
Conversion failed!

Remuxing the mp4 into mkv makes no difference to the behavior when playing it in Chrome.