segfault calling av_hwframe_transfer_data on qsv decoded frame if MFXVideoVPP_Init fails

[Steve Browne]

Summary of the bug:
I know this has worked before even in the same version so I think there's an Intel driver bug causing my MFXVideoVPP_Init to not be successful. I bet if I reboot it would be fine, but currently it's returning MFX_WRN_PARTIAL_ACCELERATION. This happens in hwcontext_qsv.c.

Once a machine gets into that state or if your GPU for whatever reason only has partial support it tries to fallback and do the hwframe transfer by other means. What happens in this case is it eventually gets to qsv_transfer_data_child in hwcontext_qsv.c where it creates a dummy frame, but only sets data[3] and not data[0]. After that this dummy frame works its way down to d3d11va_transfer_data in hwcontext_d3d11va.c where it expects the texture to be in data[0] and does no null check so it ends up segfaulting. Additionally it looks at data[0] for the index which won't be set either so it would always use index 0 which may be fine.

How to reproduce:
I don't really know how you would reproduce this without just simply forcing MFXVideoVPP_Init to fail when creating the session so it falls back. So just comment that line out and set the error code to simulate it.
Then it's just a matter of decoding a frame with h264_qsv and calling av_hwframe_transfer_data with the AV_HWFRAME_TRANSFER_DIRECTION_FROM flag.