Multi-GPU: NVDec cant find CUDA device when used inside a docker container and targeting specific GPU ID

[baudneo]

What I was trying to accomplish:
Multi nvidia gpu setup using a frigate tensorrt image and nvidia-container-toolkit. Forcing a specific GPU ID (that is not the default index: 0) using one of several methods results in an error:

[h264 @ 0x557b6d7f3900] decoder->cvdl->cuvidGetDecoderCaps(&caps) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

The issue here is that from what I can tell, only ffmpeg has this issue. Other libs/apps in the container that target a specific GPU work: loading ONNX models into the GPU works when targeting a specific device_id that is not the default index of 0.

If I target the default index of 0, everything works. It's only when targeting the non-default index. I've also tried specifying GPU_UUIDs with the same result. In my case, I can't physically move the GPUs around to force which device gets index 0, so I am left with forcing it via NVIDIA_VISIBLE_DEVICES env var or docker compose deploy options.

Using the global args '-hide_banner -loglevel warning -threads 2 -hwaccel_device 1' does not seem to work.

This issue has been reproduced by other frigate users and in ubuntu cuda images with ffmpeg 7, so not just localized to my setup.

Here are a couple issues:

• ​https://github.com/blakeblackshear/frigate/discussions/18018
• ​https://github.com/blakeblackshear/frigate/discussions/18722

Here is an issue I opened in the nvidia-container-toolkit repo: ​https://github.com/NVIDIA/nvidia-container-toolkit/issues/1209

Version:

ffmpeg version n7.0.2-18-g3e6cec1286-20240919 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14.2.0 (crosstool-NG 1.26.0.106_ed12fa6)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-libxml2 --enable-openssl --enable-lzma --enable-fontconfig --enable-libharfbuzz --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --enable-libdavs2 --enable-libdvdread --enable-libdvdnav --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libaribcaption --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-libzmq --enable-lv2 --enable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-libs='-ldl -lgomp' --extra-ldflags=-pthread --extra-ldexeflags=-pie --cc=x86_64-ffbuild-linux-gnu-gcc --cxx=x86_64-ffbuild-linux-gnu-g++ --ar=x86_64-ffbuild-linux-gnu-gcc-ar --ranlib=x86_64-ffbuild-linux-gnu-gcc-ranlib --nm=x86_64-ffbuild-linux-gnu-gcc-nm --extra-version=20240919
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100

Command used:

/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent "FFmpeg Frigate/0.16.0-0b7a33d" -rtsp_transport tcp -timeout 10000000 -fflags nobuffer -flags low_delay -i rtsp://127.0.0.1:8554/living_room -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac "/tmp/cache/living_room@%Y%m%d%H%M%S%z.mp4" -r 5 -vf "fps=5,scale_cuda=w=1920:h=1080,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5" -threads 2 -f rawvideo -pix_fmt yuv420p

FFREPORT log file: ​https://pastebin.com/qKedrL6W

[baudneo]

FFREPORT log file

[Ray Aguilar]

I would assume, like most people, this ticket was the starting point they hit when investigating this error.

In short, I believe this to be a bug that the nvidia driver itself has to fix. But until then, I was able to get it working by building my own library that intercepts the calls via libnvidia-encode.so and reformulates the response to match up with the correct gpu inside the container.

I put all the details in a repo, as well as the code: ​https://github.com/flexgrip/nvidia-gpu-enumeration/