hevc_videotoolbox not working anymore

[Atarikid]

hevc_videotoolbox dos not work anymore on macOS Monterey (does work fine on older macOS versions)
Also tried very old FFmpeg builds up to the latest daily build.

Here is the FFmpeg log

Last login: Sun Nov 21 09:44:49 on ttys000
cdv@cdv-MacBook-Pro-2 ~ % /Users/cdv/Downloads/ffmpeg-5 -i /Users/cdv/Desktop/Ryl.mp4 -c:a copy -c:v hevc_videotoolbox -b:v 1295.585k -y /Users/cdv/Movies/Ryl.mp4
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers

built with Apple clang version 11.0.0 (clang-1100.0.33.16)
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/cdv/Desktop/Ryl.mp4':

Metadata:

major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.36.100

Duration: 00:02:34.55, start: 0.000000, bitrate: 1649 kb/s

Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, smpte170m/bt470bg/bt709), 720x426 [SAR 64:45 DAR 512:213], 1260 kb/s, SAR 1775:1248 DAR 125:52, 24.99 fps, 25 tbr, 90k tbn, 180k tbc (default)
Metadata:

handler_name : VideoHandler

Stream #0:1(eng): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 384 kb/s (default)
Metadata:

handler_name : SoundHandler

Side data:

audio service type: main

Stream mapping:

Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_videotoolbox))
Stream #0:1 -> #0:1 (copy)

Press [q] to stop, ? for help
[hevc_videotoolbox @ 0x7fed48810400] Error encoding frame: -12905
[hevc_videotoolbox @ 0x7fed48810400] popping: -542398533
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

[Atarikid]

EDIT:
More info. I have now tried FFmpeg 4.1.4 and with this version it does work fine.
So it seems it is a regression bug introduced after 4.1.4 (4.2 does not work anymore)

[Atarikid]

And some more info:
When you set the -pix_fmt to yuv422p it works fine.
Also for yuv420p10le (10bit) it works fine.

It's only for yuv420p it does not work.

[Atarikid]

And some more important info. :-)

It seems the older FFmpeg versions dropped a warning when YUV420P is not possible with hevc_videotoolbox and converted it automatically to NV12.

Incompatible pixel format 'yuv420p' for codec 'hevc_videotoolbox', auto-selecting format 'nv12'

This is not the case anymore with the current FFmpeg versions.
If you set the pixel format to YUV422P, the warning also pops up but the auto switching to NV12 is done correctly.

Basically the fix is easy: do let the warning popup again when the pixel format is set to YUV420P (which is btw the default pixel format when you do not set -pix_fmt )

[Atarikid]

FWW this is fixed in the latest daily build.