Failed setup for format videotoolbox_vld - Video Decoding (H264 yuv420p)

[David Franco]

Summary of the bug:
This bug can be reproduced in both the 5.4 and the latest ffmpeg version.
While trying to decode the video time_cut.mp4 (I transmuxed this and cut it to 5s to prevent copyright issues), the videotoolbox hardware acceleration fails to initialize.

The video specs are:

• H264
• 1920x1080
• yuv420p
• The video contains some NON-IDR I Frames

My system specs are:

• Mac Book Pro 2022
• Apple M2
• 16GB
• Sonoma 14.1

I tried this video with VLC and it also produces a "vt session error: 'kVTVideoDecoderMalfunctionErr'" in the logs.
The decoding by CPU works as it should.

I am unaware if this is an issue with the parameters passed to the decoder or a videotoolbox library issue. I apologize if it's the last one.

Video download:
​https://drive.google.com/file/d/1IxTyGoVCJ8EQEbyhyZRBKyokZB4ioMBm/view?usp=sharing

How to reproduce:

% ./ffmpeg -hwaccel videotoolbox -i time_cut.mp4 -f null -                    
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with Apple clang version 13.1.6 (clang-1316.0.21.2.5)
  configuration: --prefix=/Volumes/tempdisk/sw --extra-cflags=-fno-stack-check --arch=arm64 --cc=/usr/bin/clang --enable-gpl --enable-libvmaf --enable-libopenjpeg --enable-libopus --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libvpx --enable-libwebp --enable-libass --enable-libfreetype --enable-fontconfig --enable-libtheora --enable-libvorbis --enable-libsnappy --enable-libaom --enable-libvidstab --enable-libzimg --enable-libsvtav1 --enable-libharfbuzz --enable-libkvazaar --pkg-config-flags=--static --enable-ffplay --enable-postproc --enable-neon --enable-runtime-cpudetect --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'time_cut.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomdby1iso2avc1mp41
    encoder         : Lavf61.7.100
  Duration: 00:00:05.02, start: 0.000000, bitrate: 12240 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv), 1920x1080 [SAR 1:1 DAR 16:9], 12097 kb/s, 25 fps, 25 tbr, 16k tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](chi): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, stereo, fltp, 192 kb/s (default)
      Metadata:
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]
      Side data:
        audio service type: main
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (ac3 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[h264 @ 0x159606140] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomdby1iso2avc1mp41
    encoder         : Lavf61.7.100
  Stream #0:0(und): Video: wrapped_avframe, yuv420p(tv, top coded first (swapped)), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.100 wrapped_avframe
  Stream #0:1(chi): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default)
      Metadata:
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.100 pcm_s16le
      Side data:
        audio service type: main