videotoolbox decoding h264 60fps too slow for realtime

[der_richter]

following h264 60fps sample can't be decoded in realtime with videotoolbox decoding. quicktime player and vlc both can decode (with videotoolbox/hwdec) and playback that sample without issues.

sample file: ​http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_2160p_60fps_normal.mp4
mpv issue: ​https://github.com/mpv-player/mpv/issues/5536

akemi$ ffmpeg -hwaccel videotoolbox -i "/Volumes/AkemiData/Downloads/bbb_sunflower_2160p_60fps_normal.mp4" -f null -
ffmpeg version git-2020-06-11-bc8ab084 Copyright (c) 2000-2020 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.12)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/HEAD-bc8ab08 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=/usr/bin/clang --host-cflags='-Os -w -pipe -march=nehalem -mmacosx-version-min=10.14 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -fno-stack-check' --host-ldflags='-L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/libxml2/lib -L/usr/local/opt/python@3.8/lib -L/usr/local/opt/sqlite/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/lib -F/usr/local/Frameworks -Wl,-headerpad_max_install_names -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk' --enable-ffplay --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-gpl --enable-nonfree --enable-hardcoded-tables --enable-libfdk-aac --enable-libgme --enable-libbs2b --enable-libgsm --enable-libxml2 --enable-openssl --enable-libwavpack --enable-libzmq --enable-libzimg --enable-opencl
  libavutil      56. 54.100 / 56. 54.100
  libavcodec     58. 92.100 / 58. 92.100
  libavformat    58. 46.101 / 58. 46.101
  libavdevice    58. 11.100 / 58. 11.100
  libavfilter     7. 86.100 /  7. 86.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Volumes/AkemiData/Downloads/bbb_sunflower_2160p_60fps_normal.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2013-12-17T16:40:26.000000Z
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
    composer        : Sacha Goedegebure
  Duration: 00:10:34.53, start: 0.000000, bitrate: 8487 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], 8002 kb/s, 60 fps, 60 tbr, 60k tbn, 120 tbc (default)
    Metadata:
      creation_time   : 2013-12-17T16:40:26.000000Z
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
    Metadata:
      creation_time   : 2013-12-17T16:40:28.000000Z
      handler_name    : GPAC ISO Audio Handler
    Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
    Metadata:
      creation_time   : 2013-12-17T16:40:28.000000Z
      handler_name    : GPAC ISO Audio Handler
    Side data:
      audio service type: main
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:2 -> #0:1 (ac3 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
GVA info: Successfully connected to the Intel plugin, offline Gen9 
Output #0, null, to 'pipe:':ze=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A    
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    composer        : Sacha Goedegebure
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
    encoder         : Lavf58.46.101
    Stream #0:0(und): Video: wrapped_avframe, nv12(progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2013-12-17T16:40:26.000000Z
      handler_name    : GPAC ISO Video Handler
      encoder         : Lavc58.92.100 wrapped_avframe
    Stream #0:1(und): Audio: pcm_s16le, 48000 Hz, 5.1(side), s16, 4608 kb/s (default)
    Metadata:
      creation_time   : 2013-12-17T16:40:28.000000Z
      handler_name    : GPAC ISO Audio Handler
      encoder         : Lavc58.92.100 pcm_s16le
    Side data:
      audio service type: main
frame=  592 fps= 40 q=-0.0 Lsize=N/A time=00:00:09.90 bitrate=N/A speed=0.676x   

[Carl Eugen Hoyos]

If there is an issue (which is difficult to prove without a patch), this certainly a duplicate of ticket #8731.

[der_richter]

i don't know how those can be related. first both issue use a different codec h264/AVC vs h265/HEVC. also please note VLC and quicktime play this sample without problems in realtime with hwdec/videtoolbox. i think this should be prove enough that there is an issue.

[Carl Eugen Hoyos]

Did you transcode the input file with vlc? If not, the comparison makes little sense (vlc does suppport transcoding).

[jeeb]

Replying to cehoyos:

Did you transcode the input file with vlc? If not, the comparison makes little sense (vlc does suppport transcoding).

Could you please note why you think transcoding/non-transcoding is relevant in this case? The issue is just being replicated through ffmpeg.c as that is the simplest way to utilize FFmpeg's internal tooling as a way of reproduction for an issue ticket. It is also visible in another application utilizing VT through the the FFmpeg hwaccel interface.

This can be a difference in the VT API usage between VLC's and FFmpeg's hwdec interfaces, yes. If that is what you are pointing towards, I would note that mentioning it in this way is with quite the detour (at least in my opinion).

[Carl Eugen Hoyos]

Replying to JEEB:

Replying to cehoyos:

Did you transcode the input file with vlc? If not, the comparison makes little sense (vlc does suppport transcoding).

Could you please note why you think transcoding/non-transcoding is relevant in this case?

Because the download of the frames is non-free and you cannot display with ffmpeg.