HW Decoded VAAPI video gets unexpected bars when mapped to vulkan

[Gregory Beauregard]

Summary of the bug:
How to reproduce: You need a system capable of VAAPI hardware decoding. I'm using an i9-9900K (coffee lake refresh).

Then, run the following commands using the test-img.jpg file from ​https://stream.gably.net/images/test-img.jpg or attached to the bug.

We generate a test video from the image, encode it in a format that vaapi can hardware decode, then encode it after doing a hwmap to a noop vulkan filter. The video output from this final encode can be compared to the intermediate to show extraneous green bars were added. This bug was discovered when trying to do fully hardware vaapi encoding with a vulkan filter which was resulting in unexpected bars in videos. So far I've only tested this on git master. I'm attaching the full debug log of the 3 encodes as well.

#!/bin/bash
shopt -s expand_aliases
alias ffmpeg="LD_LIBRARY_PATH=$HOME/apps/ffmpeg/build/lib /home/gregory/apps/ffmpeg/bin/ffmpeg"
alias ffprobe="LD_LIBRARY_PATH=$HOME/apps/ffmpeg/build/lib /home/gregory/apps/ffmpeg/bin/ffprobe"

vulkanvaapi_flags=(
  -hwaccel vaapi
  -init_hw_device vaapi=va:/dev/dri/renderD128
  -hwaccel_device va

  -hwaccel_output_format vaapi
  -init_hw_device vulkan=vulk
  -filter_hw_device vulk
)
vaapi_flags=(
  -init_hw_device vaapi=foo:/dev/dri/renderD128
  -filter_hw_device foo
)

# gen test encode
ffmpeg -loglevel debug -stream_loop 1 -i test-img.jpg -t 5 -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2,format=yuv420p" -c:v libx264 -pix_fmt yuv420p -crf 0 -y vtest.nut

# put in hw-decodable format
# no black bars
LIBVA_DRIVER_NAME=i965 ffmpeg -loglevel debug -hide_banner "${vaapi_flags[@]}" -i vtest.nut -c:v h264_vaapi -r:v 30 -profile:v high -qp:v 18 -bf:v 2 -vf format=nv12,hwupload -map 0 -y vaapiencode.mkv

# test, sw encode, hw decode, has green bars
LIBVA_DRIVER_NAME=i965 ffmpeg -loglevel debug -hide_banner "${vulkanvaapi_flags[@]}" -i vaapiencode.mkv -c:v libx264 -r:v 30 -profile:v high -preset veryfast -vf "hwmap,chromaber_vulkan=0:0,hwdownload,format=nv12" -map 0 -y vaapiouts.mkv

[Gregory Beauregard]

debug output

[Gregory Beauregard]

image needed to run reproducing testcase

[Gregory Beauregard]

Make sure to remove or adjust my ffmpeg command alias.

[Gregory Beauregard]

Make sure to remove or adjust my ffmpeg command alias; ffmpeg version information and such is in the attached log if you want to match master but it was up to date as of today.

[Xu Guangxin]

Hi aphysically,
Your steps 2 proved vaapi encoder is ok.
and your step 3 is not related to vaapi, right?

could you use the following command line to prove it's a vulkan filter's issue?

LIBVA_DRIVER_NAME=i965 ffmpeg -loglevel debug -hide_banner "${vulkanvaapi_flags[@]}" -i vaapiencode.mkv -c:v libx264 -r:v 30 -profile:v high -preset veryfast -y vaapiouts.mkv

thanks

[Gregory Beauregard]

Replying to XuGuangxin:

Hi aphysically,
Your steps 2 proved vaapi encoder is ok.
and your step 3 is not related to vaapi, right?

could you use the following command line to prove it's a vulkan filter's issue?

LIBVA_DRIVER_NAME=i965 ffmpeg -loglevel debug -hide_banner "${vulkanvaapi_flags[@]}" -i vaapiencode.mkv -c:v libx264 -r:v 30 -profile:v high -preset veryfast -y vaapiouts.mkv

thanks

This file produced is fine. I get the problem very specifically with both (1) vaapi hardware decode and (2) that decode is directly mapped into vulkan.

If I hwdownload the vaapi decode first there is no extra bars, even with vulkan.

This bug is probably most notable/visible because it blocks using any vulkan filters in a full vaapi hardware pipeline (i.e. without any hwdownload to software).

[Xu Guangxin]

Hi aphaysically,
Sorry, I never try Vulkan before, do you have guidance to build ffmpeg with Vulkan.

thanks

[Gregory Beauregard]

Replying to XuGuangxin:

Hi aphaysically,
Sorry, I never try Vulkan before, do you have guidance to build ffmpeg with Vulkan.

thanks

Hi,

I think on debian-like dists you need something like libvulkan-dev glslang-dev spirv-tools then you can --enable-vulkan and --enable-libglslang while building to get vulkan and vulkan filter support. You might have to adjust the source for your dist to get the glslang recognition working. I revert 98ea1a662ee172961feaa374fe8d26078838d250 for debian.

[Xu Guangxin]

thanks for the suggestion.
Please try:
​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210121043513.1961168-1-guangxin.xu@intel.com/
We still have one dark line at the right of your image. it because the image has an odd width (939x932). The odd resolution for uv is not well defined. you may need to find an even resolution source image.

[Lynne]

Fixed in ae97f69ce1032f96e361c0622355ac15cf7f9029