Vulkan encoder race conditions and conversion failures

[Capitrium]

We're comparing the new h264_vulkan encoder against the h264_nvenc encoder we currently use with our NVIDIA T4 GPUs, and are running into inconsistent behaviour and what looks like race conditions in the vulkan encoder.

ffmpeg version 7.1.git (from https://ffmpeg.org/releases/ffmpeg-snapshot-git.tar.bz2 on 2025-01-28)

ffmpeg commands:
cuda (works):

ffmpeg -v 9 -loglevel 99 -init_hw_device "cuda=cu:0,debug=1" -hwaccel cuvid -hwaccel_output_format cuda -i /tmp/bbb.mp4 -filter_hw_device cu -c:v h264_nvenc -an -y /tmp/output-cuda.mp4

vulkan (fails):

ffmpeg -v 9 -loglevel 99 -init_hw_device "vulkan=vk:0,debug=1" -hwaccel vulkan -hwaccel_output_format vulkan -i /tmp/bbb.mp4 -filter_hw_device vk -c:v h264_vulkan -an -y /tmp/output-vulkan.mp4

input file:
Get the first 30s of Big Buck Bunny with this command:

ffmpeg -ss 00:00:00 -t 30 -i http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4 -c copy /tmp/bbb.mp4

The full debug logs are attached, but here's an excerpt from the vulkan logs:

...
[h264_vulkan @ 0xaaaaba9bacc0] Pick nothing to encode next - need more input for reference pictures.
[h264_vulkan @ 0xaaaaba9bacc0] Input frame: 1920x1080 (5).
[h264_vulkan @ 0xaaaaba9bacc0] Pick normal P-picture to encode next.
[h264_vulkan @ 0xaaaabcb904f0] Creating a buffer of 6332416 bytes, usage: 0x8000, flags: 0xa
[h264_vulkan @ 0xaaaaba9bacc0] Default RefPicList0 for fn=1/poc=3:  fn=0/poc=0
[AVHWDeviceContext @ 0xaaaab9309dd0] Validation Error: [ SYNC-HAZARD-WRITE-RACING-READ ] Object 0: handle = 0xaaaabc4311e0, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0xccb77929 | vkQueueSubmit2():  Hazard WRITE_RACING_READ for entry 0, VkCommandBuffer 0xffff440099b0[], Submitted access info (submitted_usage: SYNC_IMAGE_LAYOUT_TRANSITION, command: vkCmdPipelineBarrier2, seq_no: 1, VkImage 0x470000000047[], reset_no: 1). Access info (prior_usage: SYNC_VIDEO_DECODE_VIDEO_DECODE_READ, read_barriers: VK_PIPELINE_STAGE_2_BOTTOM_OF_PIPE_BIT|VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, queue: VkQueue 0xaaaabc27dd60[], submit: 7, batch: 0, batch_tag: 65, command: vkCmdDecodeVideoKHR, command_buffer: VkCommandBuffer 0xffff3c0cc0d0[], seq_no: 2, reset_no: 5).
[h264_vulkan @ 0xaaaaba9bacc0] Frame 0/0 encoded
[h264_vulkan @ 0xaaaaba9bacc0] Output packet: pts 2, dts 1, size 834 bytes.
[h264_vulkan @ 0xaaaaba9bacc0] Input frame: 1920x1080 (6).
[h264 @ 0xaaaabcb268b0] Decoding frame, 141 bytes, 1 slices
[AVHWDeviceContext @ 0xaaaab9309dd0] Validation Error: [ SYNC-HAZARD-READ-RACING-WRITE ] Object 0: handle = 0xaaaabc27dd60, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0x29910a35 | vkQueueSubmit2():  Hazard READ_RACING_WRITE for entry 0, VkCommandBuffer 0xffff3c0c39d0[], Submitted access info (submitted_usage: SYNC_VIDEO_DECODE_VIDEO_DECODE_READ, command: vkCmdDecodeVideoKHR, seq_no: 2, reset_no: 7). Access info (prior_usage: SYNC_VIDEO_DECODE_VIDEO_DECODE_WRITE, write_barriers: SYNC_HOST_HOST_READ|SYNC_HOST_HOST_WRITE|SYNC_VIDEO_ENCODE_VIDEO_ENCODE_READ|SYNC_VIDEO_ENCODE_VIDEO_ENCODE_WRITE|SYNC_ACCELERATION_STRUCTURE_COPY_ACCELERATION_STRUCTURE_READ|SYNC_ACCELERATION_STRUCTURE_COPY_ACCELERATION_STRUCTURE_WRITE|SYNC_ACCELERATION_STRUCTURE_COPY_TRANSFER_READ|SYNC_ACCELERATION_STRUCTURE_COPY_TRANSFER_WRITE|SYNC_COPY_TRANSFER_READ|SYNC_COPY_TRANSFER_WRITE|SYNC_RESOLVE_TRANSFER_READ|SYNC_RESOLVE_TRANSFER_WRITE|SYNC_BLIT_TRANSFER_READ|SYNC_BLIT_TRANSFER_WRITE|SYNC_CLEAR_TRANSFER_WRITE, queue: VkQueue 0xaaaabc3577a[h264 @ 0xaaaaba9af5b0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[vf#0:0 @ 0xaaaaba9ae540] Clipping frame in rate conversion by 0.019997
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaaba9ac3c0] stream 0, sample 25, dts 416333
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaaba9ac3c0] stream 1, sample 13, dts 416000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaaba9ac3c0] stream 0, sample 25, dts 416333
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaaba9ac3c0] stream 1, sample 14, dts 448000
[h264 @ 0xaaaaba9af5b0] Decoding frame, 364 bytes, 1 slices
[h264 @ 0xaaaabc6a9970] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaaba9ac3c0] stream 0, sample 26, dts 433000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaaba9ac3c0] stream 1, sample 14, dts 448000
[vf#0:0 @ 0xaaaaba9ae540] Clipping frame in rate conversion by 0.019997
[AVHWDeviceContext @ 0xaaaab9309dd0] Validation Error: [ SYNC-HAZARD-READ-RACING-WRITE ] Object 0: handle = 0xaaaabc3577a0, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0x29910a35 | vkQueueSubmit2():  Hazard READ_RACING_WRITE for entry 0, VkCommandBuffer 0xffff3c0c7940[], Submitted access info (submitted_usage: SYNC_VIDEO_DECODE_VIDEO_DECODE_READ, command: vkCmdDecodeVideoKHR, seq_no: 2, reset_no: 7). Access info (prior_usage: SYNC_VIDEO_DECODE_VIDEO_DECODE_WRITE, write_barriers: SYNC_HOST_HOST_READ|SYNC_HOST_HOST_WRITE|SYNC_VIDEO_ENCODE_VIDEO_ENCODE_READ|SYNC_VIDEO_ENCODE_VIDEO_ENCODE_WRITE|SYNC_ACCELERATION_STRUCTURE_COPY_ACCELERATION_STRUCTURE_READ|SYNC_ACCELERATION_STRUCTURE_COPY_ACCELERATION_STRUCTURE_WRITE|SYNC_ACCELERATION_STRUCTURE_COPY_TRANSFER_READ|SYNC_ACCELERATION_STRUCTURE_COPY_TRANSFER_WRITE|SYNC_COPY_TRANSFER_READ|SYNC_COPY_TRANSFER_WRITE|SYNC_RESOLVE_TRANSFER_READ|SYNC_RESOLVE_TRANSFER_WRITE|SYNC_BLIT_TRANSFER_READ|SYNC_BLIT_TRANSFER_WRITE|SYNC_CLEAR_TRANSFER_WRITE, queue: VkQueue 0xaaaabc27dd6[h264_vulkan @ 0xaaaaba9bacc0] Pick B-picture at depth 1 to encode next.
[h264_vulkan @ 0xaaaaba9bacc0] Default RefPicList0 for fn=2/poc=1:  fn=0/poc=0  fn=1/poc=3
[h264_vulkan @ 0xaaaaba9bacc0] Default RefPicList1 for fn=2/poc=1:  fn=1/poc=3  fn=0/poc=0
[h264_vulkan @ 0xaaaaba9bacc0] Frame 3/1 encoded
[h264_vulkan @ 0xaaaaba9bacc0] Output packet: pts 5, dts 2, size 16272 bytes.
[h264_vulkan @ 0xaaaaba9bacc0] Input frame: 1920x1080 (7).
...
[h264_vulkan @ 0xaaaaba9bacc0] Frame 39/37 encoded
[h264_vulkan @ 0xaaaaba9bacc0] Output packet: pts 41, dts 38, size 3549 bytes.
[h264_vulkan @ 0xaaaaba9bacc0] Input frame: 1920x1080 (43).
[h264 @ 0xaaaabc6a9970] Decoding frame, 327 bytes, 1 slices
[vf#0:0 @ 0xaaaaba9ae540] Clipping frame in rate conversion by 0.019997
[h264_vulkan @ 0xaaaaba9bacc0] Pick B-picture at depth 1 to encode next.
[h264_vulkan @ 0xaaaaba9bacc0] Default RefPicList0 for fn=14/poc=38:  fn=12/poc=36  fn=13/poc=39
[h264_vulkan @ 0xaaaaba9bacc0] Default RefPicList1 for fn=14/poc=38:  fn=13/poc=39  fn=12/poc=36
[h264_vulkan @ 0xaaaaba9bacc0] Encode failed: -1163371440.
[vost#0:0/h264_vulkan @ 0xaaaaba9bbff0] [enc:h264_vulkan @ 0xaaaaba9aeaf0] Error submitting video frame to the encoder
[vost#0:0/h264_vulkan @ 0xaaaaba9bbff0] [enc:h264_vulkan @ 0xaaaaba9aeaf0] Error encoding a frame: Error number -1163371440 occurred
[vost#0:0/h264_vulkan @ 0xaaaaba9bbff0] Task finished with error code: -1163371440 (Error number -1163371440 occurred)
[vost#0:0/h264_vulkan @ 0xaaaaba9bbff0] Terminating thread with return code -1163371440 (Error number -1163371440 occurred)
[vf#0:0 @ 0xaaaaba9b0280] All consumers returned EOF
[out#0/mp4 @ 0xaaaaba81cfa0] All streams finished
[out#0/mp4 @ 0xaaaaba81cfa0] Terminating thread with return code 0 (success)
[vf#0:0 @ 0xaaaaba9b0280] Terminating thread with return code 0 (success)
[vist#0:0/h264 @ 0xaaaaba9bba20] [dec:h264 @ 0xaaaaba9ab4b0] Decoder returned EOF, finishing
[vist#0:0/h264 @ 0xaaaaba9bba20] [dec:h264 @ 0xaaaaba9ab4b0] Terminating thread with return code 0 (success)
[vist#0:0/h264 @ 0xaaaaba9bba20] All consumers of this stream are done
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaabc611400] All consumers are done
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaabc611400] Terminating thread with return code 0 (success)
[h264 @ 0xaaaabc9935d0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[h264 @ 0xaaaabc9935d0] Decoding frame, 285 bytes, 1 slices
[AVIOContext @ 0xaaaabcb78450] Statistics: 389789 bytes written, 2 seeks, 5 writeouts
[out#0/mp4 @ 0xaaaaba81cfa0] Output file #0 (/tmp/output-vulkan.mp4):
[out#0/mp4 @ 0xaaaaba81cfa0]   Output stream #0:0 (video): 42 frames encoded; 38 packets muxed (388233 bytes); 
[out#0/mp4 @ 0xaaaaba81cfa0]   Total: 38 packets (388233 bytes) muxed
[out#0/mp4 @ 0xaaaaba81cfa0] video:379KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.399760%
frame=   38 fps=0.0 q=-0.0 Lsize=     381KiB time=00:00:00.65 bitrate=4797.4kbits/s speed=3.38x    
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaabc611400] Input file #0 (/tmp/bbb.mp4):
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaabc611400]   Input stream #0:0 (video): 72 packets read (77578 bytes); 57 frames decoded; 0 decode errors; 
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaabc611400]   Total: 72 packets (77578 bytes) demuxed
[AVIOContext @ 0xaaaab977fd70] Statistics: 192903 bytes read, 2 seeks
Conversion failed!
[AVHWDeviceContext @ 0xaaaab9309dd0] Validation Error: [ VUID-vkDestroyDevice-device-05137 ] Object 0: handle = 0xd700000000d7, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0x4872eaa0 | vkCreateDevice():  OBJ ERROR : For VkDevice 0xaaaaba9cef00[], VkBuffer 0xd700000000d7[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkDestroyDevice-device-05137)
...

To me, the SYNC-HAZARD-READ-RACING-WRITE and SYNC-HAZARD-WRITE-RACING-READ lines point to some kind of race condition. It's worth noting that I've seen the vulkan encode succeed sporadically with a shorter/smaller input video (7sec./720p/23.98fps), and I'm pretty sure I've also seen SYNC-HAZARD-WRITE-RACING-WRITE in other tests as well, but that one doesn't appear in these logs.

Interestingly, ffmpeg doesn't report any vulkan decoders in -codecs, despite being built with --enable-vulkan:

# ffmpeg -codecs | grep vulkan
ffmpeg version 7.1.git Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.2.0-23ubuntu4)
  configuration: --disable-doc --disable-ffplay --disable-stripping --disable-decoder=libopenjpeg --enable-cuda --enable-cuda-nvcc --enable-nvenc --enable-cuvid --enable-libnpp --enable-vulkan --enable-libshaderc --enable-shared --enable-nonfree --enable-frei0r --enable-gpl --enable-libfdk_aac --enable-libass --enable-libfreetype --enable-libfontconfig --enable-libfribidi --enable-libvidstab --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-openssl --enable-postproc --enable-small --enable-version3 --extra-cflags='-I/opt/ffmpeg/include -I/opt/ffmpeg/vulkan/include -I/opt/ffmpeg/include/ffnvcodec -I/usr/local/cuda/include/' --extra-ldflags='-L/opt/ffmpeg/lib -L/usr/local/cuda/lib64/' --extra-libs=-ldl --nvccflags='-gencode arch=compute_75,code=sm_75 -O2' --prefix=/opt/ffmpeg
  libavutil      59. 56.100 / 59. 56.100
  libavcodec     61. 31.101 / 61. 31.101
  libavformat    61.  9.106 / 61.  9.106
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  9.100 / 10.  9.100
  libswscale      8. 13.100 /  8. 13.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
 DEV..S ffv1                  (encoders: ffv1 ffv1_vulkan)
 DEV.LS h264                  (decoders: h264 h264_v4l2m2m h264_cuvid) (encoders: libx264 libx264rgb h264_nvenc h264_v4l2m2m h264_vulkan)
 DEV.L. hevc                  (decoders: hevc hevc_v4l2m2m hevc_cuvid) (encoders: libx265 hevc_nvenc hevc_v4l2m2m hevc_vulkan)

There are also some other open issues regarding vulkan decoding/encoding:
​https://trac.ffmpeg.org/ticket/10970
​https://trac.ffmpeg.org/ticket/11219
​https://trac.ffmpeg.org/ticket/11296

So, two questions:

1. Are the Vulkan video APIs in ffmpeg considered stable as of 7.1? Or are they still experimental?
2. Should ffmpeg be reporting any Vulkan video decoders when built with --enable-vulkan? Or are extra configure flags required to enable Vulkan video decoding?

Thanks!

[Capitrium]

Zip file containing logs-cuda.log and logs-vulkan.log, corresponding to the commands in this ticket