tonemap_vaapi: Failed to start picture processing: 1 (operation failed).

[Ishan Jain]

Summary of the bug:
How to reproduce:

% /usr/lib/jellyfin-ffmpeg/ffmpeg -report -v verbose -init_hw_device "vaapi=va" -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -threads 1 -i file:/root/downscaled/output.mkv -threads 1 -filter_threads 1 -map 0:v:0 -vf "tonemap_vaapi=format=nv12:p=bt709:t=bt709:m=bt709" -debug_ts -f null -

ffmpeg version 5.1.2-Jellyfin Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-static --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil      57. 28.100 / 57. 28.100
libavcodec     59. 37.100 / 59. 37.100
libavformat    59. 27.100 / 59. 27.100
libavdevice    59.  7.100 / 59.  7.100
libavfilter     8. 44.100 /  8. 44.100
libswscale      6.  7.100 /  6.  7.100
libswresample   4.  7.100 /  4.  7.100
libpostproc    56.  6.100 / 56.  6.100
built on ...

1. The ffmpeg instance is running in a Ubuntu Jammy LXC container. ( {{{ Linux pvenas 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100) x86_64 GNU/Linux }}}
2. The host has i5 12400 processor.
3. A sample video is available here, ​http://dl.ishanjain.me/ffmpeg.tonemap.vaapi.sample.mkv

(I am not sure if I can upload this here? I tried trimming it but it was around ~100mb minimum)

4. ffmpeg builds are available here, ​https://github.com/jellyfin/jellyfin-ffmpeg. I also tried latest build from here, ​https://github.com/BtbN/FFmpeg-Builds/releases/tag/latest [98b1aa8] and it was failing there as well with the same error at the same location.

5. ffmpeg report on the sample has been attached and I have included other reports from runs on the full video and with different flags.

mediainfo ffmpeg.tonemap.vaapi.sample.mkv

emerald :: ~ 130 » mediainfo output.mkv
General
Unique ID                                : 188666213137960694644364664136156075615 (0x8DEFCA7AE3DE67AE7F06F5D34F38565F)
Complete name                            : output.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 153 MiB
Duration                                 : 22 s 813 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 56.4 Mb/s
Movie name                               : Mission.Impossible.Fallout.2018.2160p.BluRay.x265.10bit.HDR.DTS-HD.MA.TrueHD.7.1.Atmos-SWTYBLZ
Writing application                      : Lavf58.76.100
Writing library                          : Lavf58.76.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@High
HDR format                               : SMPTE ST 2086, HDR10 compatible
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 20 s 29 ms
Bit rate                                 : 62.2 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 026 pixels
Display aspect ratio                     : 1.895
Frame rate mode                          : Variable
Frame rate                               : 35.948 FPS
Original frame rate                      : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 2)
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.223
Stream size                              : 223 MiB
Title                                    : Mission.Impossible.Fallout.2018.2160p.BluRay.x265.10bit.HDR.DTS-HD.MA.TrueHD.7.1.Atmos-SWTYBLZ
Writing library                          : x265 2.8+74-fd517ae68f93:[Windows][GCC 8.2.1][64 bit] 10bit
Encoding settings                        : cpuid=1111039 / frame-threads=16 / numa-pools=24,24,24,24 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=3840x2026 / interlace=0 / total-frames=0 / level-idc=51 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / repeat-headers / annexb / aud / hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=1 / keyint=24 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=25 / lookahead-slices=4 / scenecut=40 / radl=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=3 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=3 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=16.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=160000 / vbv-bufsize=160000 / vbv-init=0.9 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=9 / transfer=16 / colormatrix=9 / chromaloc=1 / chromaloc-top=2 / chromaloc-bottom=2 / display-window=0 / master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,01) / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / hdr / hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0001 cd/m2, max: 1000 cd/m2

Audio
ID                                       : 2
Format                                   : MLP FBA 16-ch
Format/Info                              : Meridian Lossless Packing FBA with 16-channel presentation
Commercial name                          : Dolby TrueHD with Dolby Atmos
Codec ID                                 : A_TRUEHD
Duration                                 : 20 s 30 ms
Bit rate mode                            : Variable
Bit rate                                 : 5 506 kb/s
Maximum bit rate                         : 8 175 kb/s
Channel(s)                               : 8 channels
Channel layout                           : L R C LFE Ls Rs Lb Rb
Sampling rate                            : 48.0 kHz
Frame rate                               : 1 200.000 FPS (40 SPF)
Bit depth                                : 24 bits
Compression mode                         : Lossless
Delay relative to video                  : -941 ms
Stream size                              : 19.7 MiB (13%)
Title                                    : Mission.Impossible.Fallout.2018.2160p.BluRay.x265.10bit.HDR.DTS-HD.MA.TrueHD.7.1.Atmos-SWTYBLZ
Language                                 : English
Default                                  : Yes
Forced                                   : No
Number of dynamic objects                : 11
Bed channel count                        : 1 channel
Bed channel configuration                : LFE

Text
ID                                       : 3
Format                                   : ASS
Codec ID                                 : S_TEXT/ASS
Codec ID/Info                            : Advanced Sub Station Alpha
Duration                                 : 22 s 813 ms
Bit rate                                 : 120 b/s
Frame rate                               : 0.658 FPS
Count of elements                        : 15
Compression mode                         : Lossless
Stream size                              : 486 Bytes (0%)
Title                                    : English-SRT
Writing library                          : Lavc58.134.100 ssa
Language                                 : English
Default                                  : Yes
Forced                                   : No

Menu
00:00:00.000                             : Chapter 3

[Ishan Jain]

ffmpeg logs when I ran it on the sample video

[Ishan Jain]

Hey, I hadn't seen the bug reports section on this site. I have uploaded the video to streams.videolan.org and the file name is, ffmpeg.tonemap.vaapi.sample.mkv