CUVID decoder fails to process specific JPGs

[Alex Agranovsky]

Summary of the bug:
This seems to only happen with JPEGs of mjpeg (Progressive), yuvj444p(pc, bt470bg/unknown/unknown) type; not sure if it's the color space, or Progressive that does it.

How to reproduce:

% c:\bin\ffmpeg6.0\ffmpeg -hwaccel cuda -i u:\err.jpg foo.tif
ffmpeg version n6.0-22-g549430e14d-20230525 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (crosstool-NG 1.25.0.152_89671bf)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --disable-libmfx --enable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --disable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20230525
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, image2, from 'u:\err.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 86019 kb/s
  Stream #0:0: Video: mjpeg (Progressive), yuvj444p(pc, bt470bg/unknown/unknown), 2000x1500 [SAR 738:738 DAR 4:3], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> tiff (native))
Press [q] to stop, [?] for help
[mjpeg @ 000002d6f892cd80] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_IMAGE: device kernel image is invalid
Error while decoding stream #0:0: Generic error in an external library
[swscaler @ 000002d6fa6c0300] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to 'foo.tif':
  Metadata:
    encoder         : Lavf60.3.100
  Stream #0:0: Video: tiff, rgb24, 2000x1500 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.3.100 tiff
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A    /s speed=N/A
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Conversion failed!

This was first observed on 4.4.2, and confirmed as still occurring with 6.0.

[Alex Agranovsky]

Input causing the error

[Balling]

Also happens with #10190, assert.

[Balling]

Yep, apparently the reason is ​https://forums.developer.nvidia.com/t/nvjpeg-issues-and-inconsistencies-with-transcoding/213202/11?u=val.zapod.vz

So newest CUDA 12.1 has fixes for prpgressive decoding. But again we do not use CUDA toolkit.