Context Navigation

#7691 new defect

GPU side format conversions working with MediaSDK and old i965 driver, fail with new iHD driver

Reported by:	eero-t	Owned by:
Priority:	normal	Component:	undetermined
Version:	git-master	Keywords:
Cc:	fei.w.wang@intel.com, dominik@greysector.net	Blocked By:
Blocking:		Reproduced by developer:	no
Analyzed by developer:	no

Description

Summary of the bug: HW upload with format conversion for the decoded data doesn't work for iHD QSV & VA-API backends, although that same conversion works fine with the MediaSDK sample application, and with older i965 driver VA-API backend.

Setup:

Distro: Ubuntu 18.04
FFmpeg: latest compiled from Git
MediaSDK & its deps: latest compiled from Git
HW: tested on KBL GT2, KBL GT3e and CFL GT2

How to reproduce:

Use iHD driver:
```
$ export LIBVA_DRIVER_NAME=iHD
```

Decode + format conversion with MediaSDK sample app:

$ sample_decode -hw h265 -w 1920 -h 1080 -i420 -i 1920x540_8bit_60fps.h265 -o /dev/null

Decode + format conversion with QSV:

$ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x540_8bit_60fps.h265 -vf scale_qsv=w=1920:h=1080,hwdownload,format=yuv420p -f null -

Decode + format conversion with VA-API:

$ ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_8bit_60fps.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -

Switch to i965 driver:
```
$ export LIBVA_DRIVER_NAME=i965
```

Decode + format conversion with VA-API:

$ ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_8bit_60fps.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -

Expected output:

Decoded video format conversion works fine with all the drivers (it's just somewhat slower than hwdownloading the Media driver native format)

Actual outcome:

Format conversion works with MSDK decode application and i965 VA-API, but fails with iHD VA-API and QSV. Those seem to support only "nv12" for 8-bit data and "p010" for 10-bit data

Note:

With i965 driver, following formats work without errors: "yuv420p", "yuyv422", "uyvy422", "nv12", "rgb0", "bgr0", and "p010". With i965, "p010" works only with 10-bit input, whereas MediaSDK sample application seems to support also 8-bit <=> 10-bit format conversions

(Besides transcode, doing something for the decoded data is the other important use-case for video driver. For that, the video format needs to be something that decoded data consumer can do use. If that conversion can't also be done on the GPU, especially for larger video sizes, there's much less point performance-wise in doing the decode there either.)

Attachments (1)

ffmpeg-format-support.sh (2.9 KB ) - added by eero-t 5 years ago.: FFmpeg format support test script

Download all attachments as: .zip

Change History (15)

comment:1 by Carl Eugen Hoyos, 6 years ago

Please provide a (actually one) command line that you tested together with the complete, uncut console output to make this a valid ticket.

comment:2 by eero-t, 6 years ago

iHD driver "yuv420p" format failure with QSV:

$ LIBVA_DRIVER_NAME=iHD ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x540_60_yuv420p_4800.h265 -vf scale_qsv=w=1920:h=1080,hwdownload,format=yuv420p -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
    Last message repeated 1 times
[hevc_qsv @ 0x55e66ecf6700] A decode call did not consume any data: expect more data at input (-10)
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
[hevc_qsv @ 0x55e66ecf6700] A decode call did not consume any data: expect more data at input (-10)
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
[hevc_qsv @ 0x55e66ecf6700] A decode call did not consume any data: expect more data at input (-10)
[NULL @ 0x55e66ece3380] missing picture in access unit with size 1
    Last message repeated 2 times
[hwdownload @ 0x55e66ed60a80] Invalid output format yuv420p for hwframe download.
[Parsed_hwdownload_1 @ 0x55e66f104e40] Failed to configure output pad on Parsed_hwdownload_1
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

iHD VA-API fail (everything build from git yesterday like in above case):

$ LIBVA_DRIVER_NAME=iHD ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i
 1920x540_60_yuv420p_4800.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.26.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc
    Metadata:
      encoder         : Lavc58.44.100 wrapped_avframe
[AVHWFramesContext @ 0x558ee4ad3440] Failed to sync surface 0x22: 1 (operation failed).
[hwdownload @ 0x558ee48ef180] Failed to download frame: -5.
Error while filtering: Input/output error
Failed to inject frame into filter network: Input/output error
Error while processing the decoded data for stream #0:0
Conversion failed!

i965 VA-API works fine ("i965_drv_video.so" is from Ubuntu 18.04, rest is like above):

$ LIBVA_DRIVER_NAME=i965 ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_60_yuv420p_4800.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=yuv420p -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.26.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc
    Metadata:
      encoder         : Lavc58.44.100 wrapped_avframe
frame= 4800 fps=376 q=-0.0 Lsize=N/A time=00:01:20.00 bitrate=N/A speed=6.27x    
video:2512kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

As does the iHD driver if output format is changed from yuv420p to nv12 (for 8bit input):

$ LIBVA_DRIVER_NAME=iHD ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x540_60_yuv420p_4800.h265 -vf scale_vaapi=w=1920:h=1080,hwdownload,format=nv12 -f null -
ffmpeg version N-93005-gd92f06eb66 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/opt/install/Nightly_1626 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --enable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 44.100 / 58. 44.100
  libavformat    58. 26.100 / 58. 26.100
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, hevc, from '1920x540_60_yuv420p_4800.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x540 [SAR 1:2 DAR 16:9], 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.26.100
    Stream #0:0: Video: wrapped_avframe, nv12, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc
    Metadata:
      encoder         : Lavc58.44.100 wrapped_avframe
frame= 4800 fps=408 q=-0.0 Lsize=N/A time=00:01:20.00 bitrate=N/A speed= 6.8x    
video:2512kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

PS. test Video is converted from http://4ksamples.com/ses-astra-uhd-test-1-2160p-uhdtv/ with FFmpeg / libx265.

comment:3 by eero-t, 6 years ago

As a reference, at the bottom of this page is a list of platform / format combinations supported by libva iHD backend:
https://github.com/intel/media-driver/blob/master/docs/media_features.md

comment:4 by eero-t, 6 years ago

I just noticed that in the vaapi backend iHD driver case, instead of hwdownloading nv12 format, one could also remove the scale_vaapi stuff and it works.

I.e. either format conversion, or scaling, but not both, are supported by FFmpeg with iHD vaapi driver, although doing both works with i965 vaapi driver (and when using iHD driver with MediaSDK sample_decode application).

Removing scale_qsv doesn't help with FFmpeg QSV backend though, only nv12 hwdownload works with that.

comment:5 by eero-t, 5 years ago

With latest Git versions of everything, situation is same as earlier, FFmpeg VAAPI backend supports either scaling or format conversion, but not both, and QSV backend supports only scaling.

(Whereas with i965 driver or MediaSDK sample applications, both work fine.)

comment:6 by Fei Wang, 5 years ago

Cc:	fei.w.wang@intel.com added

Hi @eero-t, I used the latest ffmepg/media-driver version, your cmdline work good. Would you confirm this? Thanks.
ffmpeg commit id: 7ab4fbdebcf1c319298d7319b89fc514620a824f
media-driver commit id: 644fc6d2bb9e6271a2b91578b1bc2f63275d184a

follow-up: 8 comment:7 by eero-t, 5 years ago

VA-API

Scaling and format conversions seem to be working now, except for "uyvy422" which has been segfaulting for several months now: https://github.com/intel/media-driver/issues/757

QSV

Only scaling and "nv12" format works, for all other formats I get:

$ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x1080_60_yuv420p.h265 -vf scale_qsv=w=1920:h=540,hwdownload,format=yuv420p -f null -
...
Input #0, hevc, from '1920x1080_60_yuv420p.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[hwdownload @ 0x5598aefa46c0] Invalid output format yuv420p for hwframe download.
[Parsed_hwdownload_1 @ 0x5598aefa3600] Failed to configure output pad on Parsed_hwdownload_1
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument

I'm using git versions from yesterday:

ffmpeg https://github.com/FFmpeg/FFmpeg.git at 39d5acbd097b86a3ab8232a0156a18bf8d8a71ef 2019-10-17_15-35-28 avcodec/libdav1d: set AVFrame reordered_opaque
libva git://github.com/intel/libva at 95eb8cf469367b532b391042fa0e77ca513ac94e 2019-10-12_07-02-48 Add missed slice parameter 'slice_data_num_emu_prevn_bytes'
gmmlib git://github.com/intel/gmmlib at 77699a1c8c44884bf39fc609772b734621e120a3 2019-10-15_14-02-10 Disable ULT for 32bit
media-driver git://github.com/intel/media-driver at 4f1b23183c0ee74d9476db3dd4e68e2f987bd965 2019-10-17_09-41-08 [Decode] Fix VC1 Green Line
media-sdk git://github.com/Intel-Media-SDK/MediaSDK at f87a709646f1af1aafcf8e5fe14baf01ddb39678 2019-10-17_12-00-25 [HEVCd] Skip frames after new SPS without any subsequent slice

in reply to: 7 comment:8 by Fei Wang, 5 years ago

Replying to eero-t:

VA-API

Scaling and format conversions seem to be working now, except for "uyvy422" which has been segfaulting for several months now: https://github.com/intel/media-driver/issues/757

QSV

Only scaling and "nv12" format works, for all other formats I get:
$ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x1080_60_yuv420p.h265 -vf scale_qsv=w=1920:h=540,hwdownload,format=yuv420p -f null -
...
Input #0, hevc, from '1920x1080_60_yuv420p.h265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 60 fps, 60 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[hwdownload @ 0x5598aefa46c0] Invalid output format yuv420p for hwframe download.
[Parsed_hwdownload_1 @ 0x5598aefa3600] Failed to configure output pad on Parsed_hwdownload_1
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
I'm using git versions from yesterday:

ffmpeg https://github.com/FFmpeg/FFmpeg.git at 39d5acbd097b86a3ab8232a0156a18bf8d8a71ef 2019-10-17_15-35-28 avcodec/libdav1d: set AVFrame reordered_opaque
libva git://github.com/intel/libva at 95eb8cf469367b532b391042fa0e77ca513ac94e 2019-10-12_07-02-48 Add missed slice parameter 'slice_data_num_emu_prevn_bytes'
gmmlib git://github.com/intel/gmmlib at 77699a1c8c44884bf39fc609772b734621e120a3 2019-10-15_14-02-10 Disable ULT for 32bit
media-driver git://github.com/intel/media-driver at 4f1b23183c0ee74d9476db3dd4e68e2f987bd965 2019-10-17_09-41-08 [Decode] Fix VC1 Green Line
media-sdk git://github.com/Intel-Media-SDK/MediaSDK at f87a709646f1af1aafcf8e5fe14baf01ddb39678 2019-10-17_12-00-25 [HEVCd] Skip frames after new SPS without any subsequent slice

For the QSV cmdline, you can specific format inside scale_qsv, which can be like this:

ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v hevc_qsv -i 1920x1080_60_yuv420p.h265 -vf scale_qsv=w=1920:h=540:format=bgra,hwdownload,format=bgra -f null -

While, msdk now doesn't support yuv420p, so if you change the format in above cmdline from bgra to yuv420p, the cmdline still return error. You can list the format you need, then we can file a requirement to MSDK to ask to support it.

comment:9 by eero-t, 5 years ago

You can list the format you need, then we can file a requirement to MSDK to ask to support it.

IMHO it should support the same formats as VA-API and the old i965 driver, anything else would be a regression for a user switching between the drivers.

Attached is a script testing format support for all three FFmpeg driver versions.

Here is its abbreviated output for the old i965 driver in Ubuntu, and VA-API & QSV backends for last night git build of iHD / FFmpeg:

	i965:	VA-API:	QSV:	Format:
	-	O	-	0bgr
	-	O	-	0rgb
	-	X	-	abgr
	-	X	-	argb
	X	O	-	bgr0
	-	X	X	bgra
	-	O	-	gray
	X	X	X	nv12
	X	X	X	p010le
	X	X	-	rgb0
	-	X	-	rgba
	X	O	-	uyvy422 (iHD media-driver bug)
	-	O	-	yuv411p
	X	X	-	yuv420p
	-	X	-	yuv422p
	-	X	-	yuv440p
	-	O	-	yuv444p
	X	X	-	yuyv422
}

* X = format conversion before download
* O = format conversion only after download
* - = no support

Version 0, edited 5 years ago by eero-t (next)

by eero-t, 5 years ago

Attachment:	ffmpeg-format-support.sh added

FFmpeg format support test script

comment:10 by eero-t, 5 years ago

Note: I didn't test whether the formats in above table are actually output correctly by FFmpeg QSV & VA-API backends, only that the names of those formats didn't cause them & FFmpeg to error exit.

As a reference, here's a list of formats supported by the popular SDL library:
https://wiki.libsdl.org/SDL_PixelFormatEnum#Pixel_Format_Values

(libSDL is one of the possible FFmpeg outputs.)

comment:11 by eero-t, 5 years ago

FYI: driver bug is fixed, so uyvy422 works again: https://github.com/intel/media-driver/issues/757

comment:12 by Dominik Mierzejewski, 5 years ago

Cc:	dominik@greysector.net added

comment:13 by eero-t, 4 years ago

FYI: All the formats marked as "O" in above table, i.e. doing format conversion after hwdownload (because GPU side doesn't support those formats), broke with media-driver few days ago: https://github.com/intel/media-driver/issues/1016

comment:14 by eero-t, 4 years ago

Issue is now fixed. There appears to be one new format supported with iHD driver VA-API (x2rgb10le) and with QSV (yuyv422) since I last tested this:

	i965:	VA-API:	QSV:	Format:
	-	O	-	0bgr
	-	O	-	0rgb
	-	X	-	abgr
	-	X	-	argb
	X	O	-	bgr0
	-	X	X	bgra
	-	O	-	gray
	X	X	X	nv12
	-	X	X	p010le
	X	X	-	rgb0
	-	X	-	rgba
	X	O	-	uyvy422
	-	X	-	x2rgb10le
	-	O	-	yuv411p
	X	X	-	yuv420p
	-	X	-	yuv422p
	-	X	-	yuv440p
	-	O	-	yuv444p
	X	X	X	yuyv422

X = format conversion before download
O = format conversion only after download
- = no support

PS. After upgrading from Ubuntu 18.04 to 20.04, Ubuntu i965 driver gives now error with p010le format. 20.04 upgraded i965 VA driver version from 2.1.0 to 2.4.0, but I don't see any mention of this in: https://github.com/intel/intel-vaapi-driver/releases

Was i965 VA driver not giving error earlier for 10-bit just a bug?

Note: See TracTickets for help on using tickets.

Download in other formats: