Opened 3 years ago

Last modified 8 months ago

#9188 open enhancement

Add support for Vulkan Video decode/encode extensions

Reported by: Jacob Lifshay Owned by: Lynne
Priority: wish Component: undetermined
Version: git-master Keywords: vulkan
Cc: programmerjake@gmail.com, Lynne Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Vulkan Video is aiming to be an industry-standard cross-platform interface to video decode/encode hardware (and probably also software).

I'm proposing adding support for using Vulkan Video as a new ffmpeg codec backend. (probably a better way to describe that...)

They (as of 2021-04-14) have provisional support for encode/decode for h.264 and decode for h.265, they are planning on shortly gaining support for encode for h.265, decode for vp9, and encode/decode for av1.

An Introduction to Vulkan Video - The Khronos Group Inc

Supporting Vulkan Video would allow hardware vendors (such as the Libre-SOC project, which I'm part of) to only need to implement each codec once, rather than N times for each of N different software libraries that need support.

Funding may be available from Libre-SOC, will have to check.

Change History (4)

comment:1 by Jacob Lifshay, 3 years ago

Cc: programmerjake@gmail.com added

comment:2 by Carl Eugen Hoyos, 3 years ago

Keywords: vulkan added; Vulkan Video VK_KHR_video_queue removed
Priority: normalwish
Version: unspecifiedgit-master

comment:3 by Lynne, 10 months ago

Cc: Lynne added
Owner: set to Lynne
Status: newopen

Decoding implemented and merged in git master.
I've implemented encoding, but the spec is still in beta, and there are large issues with it that have to be addressed before it can be usable and merged.

comment:4 by Noah Sherwin, 8 months ago

The Vulkan decoding in ffmpeg 6.0 is an excellent addition, albeit unfinished.
The lack of entries in the CLI's decoders list (e.g. ffmpeg -h decoder=hevc_vulkan) makes learning and troubleshooting difficult.
There seems to be a disconnect between the pixel formats that can be decoded and the pixel formats that can be passed to Vulkan-based filters in the filter graph. The VK decoder passes 'yuvj420p' to vk_libplacebo (which is referred to as 'libplacebo' in the CLI) where the Vulkan device then complains that the format is unsupported. Explicitly converting the incoming frames to a compatible format will use swscale when copying to system memory is undesirable.
Excerpts of verbose command line output: https://imgur.com/a/khx9nix

Note: See TracTickets for help on using tickets.