videotoolbox decoding HEVC too slow for realtime

[der_richter]

following 4k HEVC sample decodes too slow with videotoolbox for realtime playback.

sample file can be found here: ​https://0x0.st/iVhx.mkv
related mpv issue: ​https://github.com/mpv-player/mpv/issues/5609

ffmpeg -hwaccel videotoolbox -i "/Volumes/AkemiData/Downloads/ffmpeg_videotoolbox_issues/iVhx.mkv" -f null -
ffmpeg version git-2020-06-11-bc8ab084 Copyright (c) 2000-2020 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.12)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/HEAD-bc8ab08 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=/usr/bin/clang --host-cflags='-Os -w -pipe -march=nehalem -mmacosx-version-min=10.14 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -fno-stack-check' --host-ldflags='-L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/libxml2/lib -L/usr/local/opt/python@3.8/lib -L/usr/local/opt/sqlite/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/lib -F/usr/local/Frameworks -Wl,-headerpad_max_install_names -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk' --enable-ffplay --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-gpl --enable-nonfree --enable-hardcoded-tables --enable-libfdk-aac --enable-libgme --enable-libbs2b --enable-libgsm --enable-libxml2 --enable-openssl --enable-libwavpack --enable-libzmq --enable-libzimg --enable-opencl
  libavutil      56. 54.100 / 56. 54.100
  libavcodec     58. 92.100 / 58. 92.100
  libavformat    58. 46.101 / 58. 46.101
  libavdevice    58. 11.100 / 58. 11.100
  libavfilter     7. 86.100 /  7. 86.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, matroska,webm, from '/Volumes/AkemiData/Downloads/d2/BladeRunner2049-1.mkv':
  Metadata:
    encoder         : libebml v1.3.7 + libmatroska v1.5.0
    creation_time   : 2020-06-12T13:35:58.000000Z
  Duration: 00:00:16.67, start: 0.025000, bitrate: 52817 kb/s
    Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 23.98 tbc (default)
    Metadata:
      BPS-eng         : 53609224
      DURATION-eng    : 00:00:16.225000000
      NUMBER_OF_FRAMES-eng: 389
      NUMBER_OF_BYTES-eng: 108726209
      _STATISTICS_WRITING_APP-eng: mkvmerge v34.0.0 ('Sight and Seen') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-06-12 13:35:58
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s
    Metadata:
      BPS-eng         : 640000
      DURATION-eng    : 00:00:16.672000000
      NUMBER_OF_FRAMES-eng: 521
      NUMBER_OF_BYTES-eng: 1333760
      _STATISTICS_WRITING_APP-eng: mkvmerge v34.0.0 ('Sight and Seen') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-06-12 13:35:58
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (ac3 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[hevc @ 0x7fb9c3021a00] First slice in a frame missing.
    Last message repeated 6 times
[hevc @ 0x7fb9c302cc00] First slice in a frame missing.
    Last message repeated 6 times
Output #0, null, to 'pipe:':ze=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A    
  Metadata:
    encoder         : Lavf58.46.101
    Stream #0:0: Video: wrapped_avframe, p010le, 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc (default)
    Metadata:
      BPS-eng         : 53609224
      DURATION-eng    : 00:00:16.225000000
      NUMBER_OF_FRAMES-eng: 389
      NUMBER_OF_BYTES-eng: 108726209
      _STATISTICS_WRITING_APP-eng: mkvmerge v34.0.0 ('Sight and Seen') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-06-12 13:35:58
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      encoder         : Lavc58.92.100 wrapped_avframe
    Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, 5.1(side), s16, 4608 kb/s
    Metadata:
      BPS-eng         : 640000
      DURATION-eng    : 00:00:16.672000000
      NUMBER_OF_FRAMES-eng: 521
      NUMBER_OF_BYTES-eng: 1333760
      _STATISTICS_WRITING_APP-eng: mkvmerge v34.0.0 ('Sight and Seen') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-06-12 13:35:58
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      encoder         : Lavc58.92.100 pcm_s16le
frame=  387 fps= 17 q=-0.0 Lsize=N/A time=00:00:16.67 bitrate=N/A speed=0.726x    
video:203kB audio:9378kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

[Carl Eugen Hoyos]

Why do you believe that this is an issue that can be fixed in FFmpeg?

[der_richter]

i can't tell you if it can be fixed within ffmpeg or if it is just a limitation of the videotoolbox API provided by Apple. that is a question for someone who actually worked on the code in ffmpeg and has more experience with the videotoolbox API and its limitations. so if you think it's not an issue with ffmpeg but upstream i am perfectly fine if this ticket is closed.

as a simple user though, i would expect a normal 4K UHDBD to playback/decode in realtime when any hwdec is used and advertised as supported. otherwise it should fallback to sw decoding. VLC for example gets a "videotoolbox warning: vt session error: 'kVTInvalidSessionErr'" and falls back to sw decoding.

also on a side note, copying the video stream into a mp4 container to make it readable with quicktime player also shields a "The file isn’t compatible with QuickTime Player." error, and it won't be played back.

[Carl Eugen Hoyos]

What you write may (or may not) be true for playback software, a transcoder should use the hardware decoder if this was requested by the user (as you did in your command line).

[der_richter]

yeah, i can see that reasoning.