Opened 17 months ago

Last modified 7 weeks ago

#7691 new defect

GPU side format conversions working with MediaSDK and old i965 driver, fail with new iHD driver

Reported by: eero-t Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords:
Cc: fei.w.wang@intel.com, dominik@greysector.net Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug: HW upload with format conversion for the decoded data doesn't work for iHD QSV & VA-API backends, although that same conversion works fine with the MediaSDK sample application, and with older i965 driver VA-API backend.

Setup:

  • Distro: Ubuntu 18.04
  • FFmpeg: latest compiled from Git
  • MediaSDK & its deps: latest compiled from Git
  • HW: tested on KBL GT2, KBL GT3e and CFL GT2

How to reproduce:

  1. Use iHD driver:
    $ export LIBVA_DRIVER_NAME=iHD
    
  2. Decode + format conversion with MediaSDK sample app:
    $ sample_decode -hw h265 -w 1920 -h 1080 -i420 -i 1920x540_8bit_60fps.h265 -o /dev/null
    
  3. Decode + format conversion with QSV:
    $ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x540_8bit_60fps.h265 -vf scale_qsv=w=1920:h=1080,hwdownload,format=yuv420p -f null -
    
  4. Decode + format conversion with VA-API:
    $ ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_8bit_60fps.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -
    
  5. Switch to i965 driver:
    $ export LIBVA_DRIVER_NAME=i965
    
  6. Decode + format conversion with VA-API:
    $ ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_8bit_60fps.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -
    

Expected output:

  • Decoded video format conversion works fine with all the drivers (it's just somewhat slower than hwdownloading the Media driver native format)

Actual outcome:

  • Format conversion works with MSDK decode application and i965 VA-API, but fails with iHD VA-API and QSV. Those seem to support only "nv12" for 8-bit data and "p010" for 10-bit data

Note:

  • With i965 driver, following formats work without errors: "yuv420p", "yuyv422", "uyvy422", "nv12", "rgb0", "bgr0", and "p010". With i965, "p010" works only with 10-bit input, whereas MediaSDK sample application seems to support also 8-bit <=> 10-bit format conversions

(Besides transcode, doing something for the decoded data is the other important use-case for video driver. For that, the video format needs to be something that decoded data consumer can do use. If that conversion can't also be done on the GPU, especially for larger video sizes, there's much less point performance-wise in doing the decode there either.)

Attachments (1)

ffmpeg-format-support.sh (2.9 KB) - added by eero-t 7 months ago.
FFmpeg format support test script

Download all attachments as: .zip

Change History (13)

comment:1 Changed 16 months ago by cehoyos

Please provide a (actually one) command line that you tested together with the complete, uncut console output to make this a valid ticket.

comment:2 Changed 16 months ago by eero-t

iHD driver "yuv420p" format failure with QSV:

$ LIBVA_DRIVER_NAME=iHD ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x540_60_yuv420p_4800.h265 -vf scale_qsv=w=1920:h=1080,hwdownload,format=yuv420p -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
    Last message repeated 1 times
[hevc_qsv @ 0x55e66ecf6700] A decode call did not consume any data: expect more data at input (-10)
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
[hevc_qsv @ 0x55e66ecf6700] A decode call did not consume any data: expect more data at input (-10)
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
[hevc_qsv @ 0x55e66ecf6700] A decode call did not consume any data: expect more data at input (-10)
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
    Last message repeated 2 times
[hwdownload @ 0x55e66ed60a80] Invalid output format yuv420p for hwframe download.
[Parsed_hwdownload_1 @ 0x55e66f104e40] Failed to configure output pad on Parsed_hwdownload_1
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

iHD VA-API fail (everything build from git yesterday like in above case):

$ LIBVA_DRIVER_NAME=iHD ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i
 1920x540_60_yuv420p_4800.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.26.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc
    Metadata:
      encoder         : Lavc58.44.100 wrapped_avframe
[AVHWFramesContext @ 0x558ee4ad3440] Failed to sync surface 0x22: 1 (operation failed).
[hwdownload @ 0x558ee48ef180] Failed to download frame: -5.
Error while filtering: Input/output error
Failed to inject frame into filter network: Input/output error
Error while processing the decoded data for stream #0:0
Conversion failed!

i965 VA-API works fine ("i965_drv_video.so" is from Ubuntu 18.04, rest is like above):

$ LIBVA_DRIVER_NAME=i965 ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_60_yuv420p_4800.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.26.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc
    Metadata:
      encoder         : Lavc58.44.100 wrapped_avframe
frame= 4800 fps=376 q=-0.0 Lsize=N/A time=00:01:20.00 bitrate=N/A speed=6.27x    
video:2512kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

As does the iHD driver if output format is changed from yuv420p to nv12 (for 8bit input):

$ LIBVA_DRIVER_NAME=iHD ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_60_yuv420p_4800.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=nv12 -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.26.100
    Stream #0:0: Video: wrapped_avframe, nv12, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc
    Metadata:
      encoder         : Lavc58.44.100 wrapped_avframe
frame= 4800 fps=408 q=-0.0 Lsize=N/A time=00:01:20.00 bitrate=N/A speed= 6.8x    
video:2512kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

PS. test Video is converted from http://4ksamples.com/ses-astra-uhd-test-1-2160p-uhdtv/ with FFmpeg / libx265.

comment:3 Changed 16 months ago by eero-t

As a reference, at the bottom of this page is a list of platform / format combinations supported by libva iHD backend:
https://github.com/intel/media-driver/blob/master/docs/media_features.md

comment:4 Changed 15 months ago by eero-t

I just noticed that in the vaapi backend iHD driver case, instead of hwdownloading nv12 format, one could also remove the scale_vaapi stuff and it works.

I.e. either format conversion, or scaling, but not both, are supported by FFmpeg with iHD vaapi driver, although doing both works with i965 vaapi driver (and when using iHD driver with MediaSDK sample_decode application).

Removing scale_qsv doesn't help with FFmpeg QSV backend though, only nv12 hwdownload works with that.

comment:5 Changed 12 months ago by eero-t

With latest Git versions of everything, situation is same as earlier, FFmpeg VAAPI backend supports either scaling or format conversion, but not both, and QSV backend supports only scaling.

(Whereas with i965 driver or MediaSDK sample applications, both work fine.)

comment:6 Changed 8 months ago by wangfei

  • Cc fei.w.wang@intel.com added

Hi @eero-t, I used the latest ffmepg/media-driver version, your cmdline work good. Would you confirm this? Thanks.
ffmpeg commit id: 7ab4fbdebcf1c319298d7319b89fc514620a824f
media-driver commit id: 644fc6d2bb9e6271a2b91578b1bc2f63275d184a

comment:7 follow-up: Changed 8 months ago by eero-t

VA-API

Scaling and format conversions seem to be working now, except for "uyvy422" which has been segfaulting for several months now: https://github.com/intel/media-driver/issues/757

QSV

Only scaling and "nv12" format works, for all other formats I get:

$ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x1080_60_yuv420p.h265 -vf scale_qsv=w=1920:h=540,hwdownload,format=yuv420p -f null -
...
Input #0, hevc, from '1920x1080_60_yuv420p.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[hwdownload @ 0x5598aefa46c0] Invalid output format yuv420p for hwframe download.
[Parsed_hwdownload_1 @ 0x5598aefa3600] Failed to configure output pad on Parsed_hwdownload_1
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument

I'm using git versions from yesterday:

comment:8 in reply to: ↑ 7 Changed 7 months ago by wangfei

Replying to eero-t:

VA-API

Scaling and format conversions seem to be working now, except for "uyvy422" which has been segfaulting for several months now: https://github.com/intel/media-driver/issues/757

QSV

Only scaling and "nv12" format works, for all other formats I get:

$ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x1080_60_yuv420p.h265 -vf scale_qsv=w=1920:h=540,hwdownload,format=yuv420p -f null -
...
Input #0, hevc, from '1920x1080_60_yuv420p.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[hwdownload @ 0x5598aefa46c0] Invalid output format yuv420p for hwframe download.
[Parsed_hwdownload_1 @ 0x5598aefa3600] Failed to configure output pad on Parsed_hwdownload_1
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument

I'm using git versions from yesterday:

For the QSV cmdline, you can specific format inside scale_qsv, which can be like this:

ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x1080_60_yuv420p.h265 -vf scale_qsv=w=1920:h=540:format=bgra,hwdownload,format=bgra -f null -

While, msdk now doesn't support yuv420p, so if you change the format in above cmdline from bgra to yuv420p, the cmdline still return error. You can list the format you need, then we can file a requirement to MSDK to ask to support it.

comment:9 Changed 7 months ago by eero-t

You can list the format you need, then we can file a requirement to MSDK to ask to support it.

IMHO it should support the same formats as VA-API and the old i965 driver, anything else would be a regression for a user switching between the drivers.

Attached is a script testing format support for all three FFmpeg driver versions.

Here is its abbreviated output for the old i965 driver in Ubuntu, and VA-API & QSV backends for last night git build of iHD / FFmpeg:

	i965:	VA-API:	QSV:	Format:
	-	O	-	0bgr
	-	O	-	0rgb
	-	X	-	abgr
	-	X	-	argb
	X	O	-	bgr0
	-	X	X	bgra
	-	O	-	gray
	X	X	X	nv12
	X	X	X	p010le
	X	X	-	rgb0
	-	X	-	rgba
	X	O	-	uyvy422 (iHD media-driver bug)
	-	O	-	yuv411p
	X	X	-	yuv420p
	-	X	-	yuv422p
	-	X	-	yuv440p
	-	O	-	yuv444p
	X	X	-	yuyv422
  • X = format conversion before download
  • O = format conversion only after download
  • - = no support
Last edited 7 months ago by eero-t (previous) (diff)

Changed 7 months ago by eero-t

FFmpeg format support test script

comment:10 Changed 7 months ago by eero-t

Note: I didn't test whether the formats in above table are actually output correctly by FFmpeg QSV & VA-API backends, only that the names of those formats didn't cause them & FFmpeg to error exit.

As a reference, here's a list of formats supported by the popular SDL library:
https://wiki.libsdl.org/SDL_PixelFormatEnum#Pixel_Format_Values

(libSDL is one of the possible FFmpeg outputs.)

comment:11 Changed 6 months ago by eero-t

FYI: driver bug is fixed, so uyvy422 works again: https://github.com/intel/media-driver/issues/757

comment:12 Changed 7 weeks ago by rathann

  • Cc dominik@greysector.net added
Note: See TracTickets for help on using tickets.