av_hwdevice_ctx_init fails for AV_HWDEVICE_TYPE_VAAPI with some drivers

[Steve Browne]

The problem occurs in vaapi_decode_find_best_format because it uses vaQuerySurfaceAttributes to find the software format. The problem is the versions of both nvidia_drv_video.so and vdpau_drv_video.so on Ubuntu 18.04 both fail with invalid parameter when specifying a null attribute list to get the size required. This causes the following output:

Failed to query surface attributes: 18 (invalid parameter).
Failed setup for format vaapi_vld: hwaccel initialisation returned error.

In the latest release of libva-vdpau-driver (0.74b) they no longer return with invalid parameter, but they only return 2 surface attributes and neither of them are a pixel format so it will still fail. Note that this hasn't changed in head as of this ticket either. See:
​https://github.com/arekinath/libva-vdpau-driver/blob/14b32901a96e5bf135d06e8dcf3d60a3c841fc14/src/vdpau_video.c#L314

I can work up a patch when I get some time, but my proposed solution would be to ignore vaQuerySurfaceAttribute failures and if no pixel format is found from surface attributes fall back to using vaMaxNumImageFormats and vaQueryImageFormats.

[wenbin,chen]

hi swbrowne.

I think you should submit this issue to libva team.

thanks