Opened 6 months ago

#10723 new defect

Issue in vulkan_export_to_cuda when dealing with RGBA frames

Reported by: Xaphiosis Owned by:
Priority: normal Component: avutil
Version: unspecified Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:

This comes from the ALVR project. We were trying to get a Vulkan-backed AVFrame (with vulkan pixel format VK_FORMAT_R8G8B8A8_UNORM and FFMPEG pixel format AV_PIX_FMT_BGRA) copied to a CUDA-backed AVFrame with same AV_PIX_FMT_BGRA (for encoding with nvenc).
It kept dying with CUDA_ERROR_INVALID_VALUE at cuMemcpy2D.
After re-implementing the export and doing the copy manually with CUDA, we found the following:

When requesting the external memory with
cuExternalMemoryGetMappedMipmappedArray, the arrayDesc .NumChannels is
set in vulkan_export_to_cuda as:

.NumChannels = 1 + ((planes == 2) && i)

but our vulkan frame is VK_FORMAT_R8G8B8A8_UNORM on the vulkan side,
AV_PIX_FMT_BGRA on the FFMPEG side, which means .NumChannels should be
4, and there's no way it can possibly be 4. It looks like this will work
with NV12, but cannot work in our case, as number of channels will be set to 1.

After discussing, the following change was proposed, but it's not clear to me whether this will work in all cases either:

diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
index 204b57c011..e3bd6ace9b 100644
--- a/libavutil/hwcontext_vulkan.c
+++ b/libavutil/hwcontext_vulkan.c
@@ -2859,7 +2859,7 @@ static int vulkan_export_to_cuda(AVHWFramesContext *hwfc,
                 .arrayDesc = {
                     .Depth = 0,
                     .Format = cufmt,
-                    .NumChannels = 1 + ((planes == 2) && i),
+                    .NumChannels = desc->comp[i].step,
                     .Flags = 0,
                 },
                 .numLevels = 1, 

You can see the relevant ALVR PR here for more context:
https://github.com/alvr-org/ALVR/pull/1911

Change History (0)

Note: See TracTickets for help on using tickets.