av_hwframe_transfer_data fails from QSV decoded frame on Intel 4th gen hardware

[Steve Browne]

Summary of the bug:
On Intel 4th gen (probably older as well) hardware decode with an h264_qsv decoder then try to transfer the frame to system memory with av_hwframe_transfer_data.

For reference the hwdevice_ctx was created as follows:

av_dict_set(&dict, "child_device_type", "d3d11va", 0);
av_hwdevice_ctx_create(&ctx, AV_HWDEVICE_TYPE_QSV, "auto", dict, 0);

Then try to copy the frame back to system memory:

av_hwframe_transfer_data(dstFrame, frame, AV_HWFRAME_TRANSFER_DIRECTION_FROM);

This will fail with:

Error synchronizing the operation: -17

Error -17 corresponds to MFX_ERR_DEVICE_FAILED. I assume the problem is simply that MFXVideoVPP_RunFrameVPPAsync doesn't support copying to system memory on the older hardware. I updated to the latest Intel drivers etc and it still didn't work. On newer Intel hardware this works just fine.

Luckily this was a result of a recent ffmpeg upgrade of our code where I was trying to ditch our existing code that worked with qsv/mfx directly so using our existing code I could see that we were creating a staging texture and mapping/unmapping that. So the workaround is to avoid av_hwframe_transfer_data and instead use av_hwframe_map to map to an AV_PIX_FMT_D3D11 frame. Then create the D3D11 staging texture using the device I get from the ID3D11Texture2D from the mapped frame, lock the staging texture, put those buffers on an intermediate frame then av_frame_copy that to the final output frame.

I've timed the code using av_hwframe_transfer_data vs the workaround on newer Intel hardware and there's not much of a performance difference. If anything the workaround is ~0.1ms faster on average.