h264_amf and hevc_amf is extremely slow when use D3D11 hwaccel as input when frame->initial_pool_size > 0

[Reito]

Summary of the bug:
How to reproduce:

Initialize an encoding codec using h264_amf or hevc_amf, create an D3D11 device and initialize the hw_device and hw_frame, feed D3D11Texture2D to AVFrame and then avcodec_send_frame, It takes about 12ms for each frame.

        AVHWFramesContext* frame = reinterpret_cast<AVHWFramesContext*>(hwf->data);
        frame->format = AV_PIX_FMT_D3D11;
        frame->sw_format = AV_PIX_FMT_NV12;
        frame->width = ctx->width;
        frame->height = ctx->height;
        frame->initial_pool_size = 2; // 2 is max

After doing a dig into the amfenc.c source, I find out these code cost about 10ms every frame and literally did nothing useful:

Line 673

        if (hw_surface) {
            AMFBuffer *frame_ref_storage_buffer;

            // input HW surfaces can be vertically aligned by 16; tell AMF the real size
            surface->pVtbl->SetCrop(surface, 0, 0, frame->width, frame->height);

            frame_ref_storage_buffer = amf_create_buffer_with_frame_ref(frame, ctx->context);
            AMF_RETURN_IF_FALSE(ctx, frame_ref_storage_buffer != NULL, AVERROR(ENOMEM), "create_buffer_with_frame_ref() returned NULL\n");

            res = amf_set_property_buffer(surface, L"av_frame_ref", frame_ref_storage_buffer);
            AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_UNKNOWN, "SetProperty failed for \"av_frame_ref\" with error %d\n", res);
            ctx->hwsurfaces_in_queue++;
            frame_ref_storage_buffer->pVtbl->Release(frame_ref_storage_buffer);
        }

Line 733

            if (data->pVtbl->HasProperty(data, L"av_frame_ref")) {
                AMFBuffer *frame_ref_storage_buffer;
                res = amf_get_property_buffer(data, L"av_frame_ref", &frame_ref_storage_buffer);
                AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_UNKNOWN, "GetProperty failed for \"av_frame_ref\" with error %d\n", res);
                amf_release_buffer_with_frame_ref(frame_ref_storage_buffer);
                ctx->hwsurfaces_in_queue--;
            }

After remove these code the encode speed is normal (about 0.5ms per frame).

In my understanding, these code is requesting an buffer from amf, clone an AVFrame, copy AVFrame to it, pass it to amf. After frame is submitted and encoded, it retrives the buffer, copy AVFrame out and release everything. In summary, neither amf nor ffmpeg use anything from this procedure, it is totally useless code.

More interestingly, if i change frame->initial_pool_size = 1, it will also encode every frame for 12ms, so it might related to ctx->hwsurfaces_in_queue

ffmpeg version
built on master, 5.1.2, 4.4.3