Encoding with hevc_videotoolbox (hardware) using Mac OS Sonoma 14.2.1 on M2 fails with error

[f.fischer]

Summary of the bug:

Encoding any input-file using hevc_videotoolbox on Mac OS Sonoma 14.2.1 using an Apple M2 MBP / i7 MBP / M1 MacStudio fails with "Error: cannot create compression session: -12908".

What I wanted to accomplish:

Create a preview file of a high res videofile (in my case AVC-I 100 in MXF container, but any other container / codec combination I tested failed as well).

What problem I encountered:

ffmpeg stoppes the encoding with named error and terminates the thread. Detail of the error encountered (excerpt of the logfile attached):

[hevc_videotoolbox @ 0x7fa551f0c440] Error: cannot create compression session: -12908
[hevc_videotoolbox @ 0x7fa551f0c440] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
[vost#0:0/hevc_videotoolbox @ 0x7fa551f0c000] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 0x7fa551f0cd00] Error sending frames to consumers: Generic error in an external library
[vf#0:0 @ 0x7fa551f0cd00] Task finished with error code: -542398533 (Generic error in an external library)
(...)
[vf#0:0 @ 0x7fa551f0cd00] Terminating thread with return code -542398533 (Generic error in an external library)

Please find attached a full terminal output with -v 9 -loglevel 99.

How to reproduce:

Use i7 / M1 / M2 Apple computer (other chips not tested) with Mac OS Sonoma (tested up to 14.2.1) and use a prompt with -c:v hevc_videotoolbox.

The exact prompt I used:

ffmpeg -i "/Users/devuser/Documents/Testclips/Testclip1.mxf" -c:a aac -c:v hevc_videotoolbox -y "/Users/devuser/Documents/Testclips/Testclip1.mp4"

fmpeg version N-113169-ge1c1dc8347-tessus (UTC compile date: 2024-01-01) as offered here: ​https://evermeet.cx/ffmpeg/
with
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay

What I did to prevent the issue / work around it:

• added profile:v main or -profile:v main10 to check if there is a problem with it - no success, same error
• added -pix_fmt yuv420p10le to see if I can provoke ffmpeg to check the value and fall back to the right colorprofile (8 bit in this case)
• used -allow_sw 1 as suggested to enable software rendering, which works, but reduces rendering speed to about 25% of hardware rendering

[f.fischer]

ffmpeg full terminal output with -v 9 -loglevel 99 on Mac OS Sonoma

[f.fischer]

Obvious error is obvious: This static version was not configured with "--enable-videotoolbox --enable-audiotoolbox" and thus it will not work.

Sorry for not having seen this. Please close or make a request to include "--enable-videotoolbox --enable-audiotoolbox" into the Mac OS static builds, if possible.

[Pablo]

Replying to f.fischer:

Obvious error is obvious: This static version was not configured with "--enable-videotoolbox --enable-audiotoolbox" and thus it will not work.

Sorry for not having seen this. Please close or make a request to include "--enable-videotoolbox --enable-audiotoolbox" into the Mac OS static builds, if possible.

No. thats not the problem, i have ffmpeg compiled with this options but after upgrade to sonoma ffmp started with that error when i try to encode videos on x265

ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers

built with Apple clang version 15.0.0 (clang-1500.3.9.4)
configuration: --prefix=/usr/local/Cellar/ffmpeg/6.1.1_6 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopenvino --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100

[Pablo]

I make a downgrade to Ventura and ffmpeg 6.1.1 work perfect with videotoolbox hw encoding, apple make some changes on OS and videotoolbox module not work anymore on recent os version

[gnattu]

You are downloading an Intel build of ffmpeg and trying to use hardware encoder through rosetta on Apple Silicon, this won't work, VideoToolbox will report the kVTCouldNotFindVideoEncoderErr if you attempted to do this. Please please please, use a native arm64 build on arm64 cpu.