Segmentaion fault using vulkan decoder on Windows with NVIDIA

[Sn0wCrack]

I'm currently experiencing a segmentation fault error when attmepting to use Vulkan as a video decoder as part of the following project: ​https://github.com/streetpea/chiaki4deck

After some debugging with the developers I've been directed here as it seems to be some upstream issue relating to ffmpeg itself. The discussion can be found here: ​https://github.com/streetpea/chiaki4deck/issues/239

For debugging I've specifically checked out the n6.1.1 tag to create my debug build.

The segmentation fault occurs after calling avcodec_send_packet with the content of the received frame.

I'm currently using an NVIDIA RTX 3080 with driver version 551.23 on the latest updates of Windows 11.

I have the following stack trace:

[avcodec-60.dll] ff_vk_exec_add_dep_frame vulkan.c:610
[avcodec-60.dll] ff_vk_decode_frame vulkan_decode.c:527
[avcodec-60.dll] vk_hevc_end_frame vulkan_hevc.c:909
[avcodec-60.dll] hevc_decode_frame hevcdec.c:3360
[avcodec-60.dll] decode_simple_internal decode.c:430
[avcodec-60.dll] decode_simple_receive_frame decode.c:609
[avcodec-60.dll] decode_receive_frame_internal decode.c:637
[avcodec-60.dll] avcodec_send_packet decode.c:734
[chiaki.exe] chiaki_ffmpeg_decoder_video_sample_cb ffmpegdecoder.c:125
[chiaki.exe] chiaki_video_receiver_flush_frame videoreceiver.c:204
[chiaki.exe] chiaki_video_receiver_av_packet videoreceiver.c:144
[chiaki.exe] stream_connection_takion_av streamconnection.c:1018
[chiaki.exe] stream_connection_takion_cb streamconnection.c:380
[chiaki.exe] takion_handle_packet_av takion.c:1265
[chiaki.exe] takion_handle_packet takion.c:922
[chiaki.exe] takion_thread_func takion.c:792
[chiaki.exe] win32_thread_func thread.c:20
[kernel32.dll] BaseThreadInitThunk 0x00007fff4f03257d
[ntdll.dll] RtlUserThreadStart 0x00007fff4fa2aa58
<unknown> 0x0000000000000000

Applying the following patch seems to resolve the segmentation fault crash without creating any additional issue in the context of the application ffmpeg is being used in:

diff --git a/libavutil/vulkan.c b/libavutil/vulkan.c
index bf8456b..2c9ff96 100644
--- a/libavutil/vulkan.c
+++ b/libavutil/vulkan.c
@@ -607,6 +607,9 @@ int ff_vk_exec_add_dep_frame(FFVulkanContext *s, FFVkExecContext *e, AVFrame *f,
     uint32_t *queue_family_dst;
     VkAccessFlagBits *access_dst;
 
+    if (!f || !f->hw_frames_ctx)
+        return 1;
+
     AVHWFramesContext *hwfc = (AVHWFramesContext *)f->hw_frames_ctx->data;
     AVVulkanFramesContext *vkfc = hwfc->hwctx;
     AVVkFrame *vkf = (AVVkFrame *)f->data[0];

I'm not particularly able to replicate this crash in other applications utilising ffmpeg, such as mpv or even the ffmpeg command line tool itself.

[Amro Emad]

Same issue on Linux, with mpv, coredump shows the following:

[Thread debugging using libthread_db enabled]                                                                                                                                                             
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/bin/mpv --player-operation-mode=pseudo-gui -- /home/amro/Videos/The.World.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ff_vk_exec_add_dep_frame.constprop.0.isra.0 (e=0x5b3ac64a0580, f=0x0, signal_stage=67108864, wait_stage=67108864, s=<optimized out>) at ./libavutil/vulkan.c:612
612	   AVVkFrame *vkf = (AVVkFrame *)f->data[0];                                                                                                                                                     
[Current thread is 1 (Thread 0x7e23ce3abf40 (LWP 177563))]

[Amro Emad]

coredump_trace

[bermond]

I can confirm that vulkan decoding is also broken in GNU/Linux currently, at least when using the latest NVIDIA proprietary driver. A segmentation fault occurs. I think that the title could be updated to reflect this.

Vulkan filters are working fine.

Same result with ffmpeg git master and 6.1.1.

Example command:

$ ffmpeg -init_hw_device vulkan=dGPU:0 -hwaccel vulkan -hwaccel_output_format vulkan -hwaccel_device dGPU -i '200 IQ Husky Climbs Ladder To Board Boat [n5jf9PpZzGY].mp4' -vf 'hwdownload,format=nv12,format=yuv420p' -an -c:v libx264 test-out.mp4
ffmpeg version N-114618-g2204ea00d6 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 13.2.1 (GCC) 20230801
  configuration: --prefix=/usr --extra-cflags='-I/opt/cuda/include -I/usr/include/tensorflow' --extra-ldflags=-L/opt/cuda/lib64 --enable-lto --disable-rpath --enable-gpl --enable-version3 --enable-nonfree --enable-shared --disable-static --disable-stripping --disable-htmlpages --enable-gray --enable-alsa --enable-avisynth --enable-bzlib --enable-chromaprint --enable-frei0r --enable-gcrypt --enable-gmp --enable-gnutls --enable-iconv --enable-ladspa --enable-lcms2 --enable-libaom --enable-libaribb24 --enable-libaribcaption --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcelt --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdavs2 --enable-libdc1394 --enable-libfdk-aac --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libglslang --enable-libgme --enable-libgsm --enable-libiec61883 --enable-libilbc --enable-libjack --enable-libjxl --enable-libklvanc --enable-libkvazaar --enable-liblensfun --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopencv --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopenvino --enable-libopus --enable-libplacebo --enable-libpulse --enable-libqrencode --enable-librabbitmq --enable-librav1e --enable-librist --enable-librsvg --enable-librubberband --enable-librtmp --disable-libshaderc --enable-libshine --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libsvthevc --enable-libsvtvp9 --enable-libtensorflow --enable-libtesseract --enable-libtheora --disable-libtls --disable-libtorch --enable-libtwolame --enable-libuavs3d --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxevd --enable-libxeve --enable-libxavs --enable-libxavs2 --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-lzma --enable-decklink --disable-mbedtls --enable-libmysofa --enable-openal --enable-opencl --enable-opengl --disable-openssl --disable-pocketsphinx --enable-sndio --enable-sdl2 --enable-vapoursynth --enable-vulkan --enable-xlib --enable-zlib --enable-amf --enable-cuda-nvcc --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-libdrm --enable-libvpl --enable-libnpp --enable-nvdec --enable-nvenc --enable-omx --enable-rkmpp --enable-v4l2-m2m --enable-vaapi --enable-vdpau
  libavutil      59. 10.100 / 59. 10.100
  libavcodec     61.  5.100 / 61.  5.100
  libavformat    61.  3.100 / 61.  3.100
  libavdevice    61.  2.100 / 61.  2.100
  libavfilter    10.  2.100 / 10.  2.100
  libswscale      8.  2.100 /  8.  2.100
  libswresample   5.  2.100 /  5.  2.100
  libpostproc    58.  2.100 / 58.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '200 IQ Husky Climbs Ladder To Board Boat [n5jf9PpZzGY].mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2023-02-17T22:54:31.000000Z
  Duration: 00:00:40.75, start: 0.000000, bitrate: 817 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 576x1024 [SAR 1:1 DAR 9:16], 684 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
      Metadata:
        creation_time   : 2023-02-17T22:54:31.000000Z
        handler_name    : ISO Media file produced by Google Inc. Created on: 02/17/2023.
        vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
      Metadata:
        creation_time   : 2023-02-17T22:54:31.000000Z
        handler_name    : ISO Media file produced by Google Inc. Created on: 02/17/2023.
        vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Press [q] to stop, [?] for help
Segmentation fault (core dumped)