VAAPI pixel format support regression between FFmpeg 3.x and 4.x

[bmegli]

Summary of the bug:

In 3.x (e.g. 3.4 in Ubuntu 18.04) it is possible to use FFmpeg libavcodec encoding with formats like bgr0 as frames context sw_format for encoding (e.g. 444 input chroma subsampling).

In 4.x (e.g. 4.2 in Ubuntu 20.04) this is no longer possible and ends with error (e.g. for H.264):

 [h264_vaapi @ 0x55af43d3ee40] No usable encoding profile found.

Both in 3.x and 4.x av_hwframe_transfer_get_formats reports those formats as uploadable for VAAPI.

The reason for this problem seems to be dynamic vaapi profile selection introduced in ​commit present since FFmpeg 4.1.

The profile auto-detection feature tries to match software format depth, number of components, log2_chroma_w and log2_chroma_h vs supported codec profile and later match it with VAAPI profile. Naturally it is out of luck, most VAAPI encoders do not support 4:4:4 chroma subsampling. In 3.4 it would accept surface in such pixel format and perform encoding (obviously not in 4:4:4 but in 4:2:0 or 4:2:2).

The relevant vaapi_encode.c fragment from ​current master:

    desc = av_pix_fmt_desc_get(ctx->input_frames->sw_format);
    if (!desc) {
        av_log(avctx, AV_LOG_ERROR, "Invalid input pixfmt (%d).\n",
               ctx->input_frames->sw_format);
        return AVERROR(EINVAL);
    }
    depth = desc->comp[0].depth;

    //some code omitted for clarity

    for (i = 0; (ctx->codec->profiles[i].av_profile !=
                 FF_PROFILE_UNKNOWN); i++) {
        profile = &ctx->codec->profiles[i];
        if (depth               != profile->depth ||
            desc->nb_components != profile->nb_components)
            continue;
        if (desc->nb_components > 1 &&
            (desc->log2_chroma_w != profile->log2_chroma_w ||
             desc->log2_chroma_h != profile->log2_chroma_h))
            continue;
        if (avctx->profile != profile->av_profile &&
            avctx->profile != FF_PROFILE_UNKNOWN)
            continue;

#if VA_CHECK_VERSION(1, 0, 0)
        profile_string = vaProfileStr(profile->va_profile);
#else
        profile_string = "(no profile names)";
#endif

        for (j = 0; j < n; j++) {
            if (va_profiles[j] == profile->va_profile)
                break;
        }
        if (j >= n) {
            av_log(avctx, AV_LOG_VERBOSE, "Compatible profile %s (%d) "
                   "is not supported by driver.\n", profile_string,
                   profile->va_profile);
            continue;
        }

        ctx->profile = profile;
        break;
    }
    if (!ctx->profile) {
        av_log(avctx, AV_LOG_ERROR, "No usable encoding profile found.\n");
        err = AVERROR(ENOSYS);
        goto fail;
    }

The problem is, there is currently no way to use pixel format like bgr0, even if encoding profile is specified manually.

How to reproduce:

The easiest way to reproduce is to use doc/examples/vaapi_encode.c
with ​different sw_format, e.g.:

//frames_ctx->sw_format = AV_PIX_FMT_NV12;
frames_ctx->sw_format = AV_PIX_FMT_BGR0;

The rest of example doesn't need to be modified (it will never be reached due to error with auto profile selection).

Note, that with 3.4 this example would require additional call to avcodec_register_all to work (deprecated and meaningless in 4.2).

If this new VAAPI behaviour is intentional what is now the correct way to work with bgr0 VAAPI surfaces keeping hardware format conversions (like in 3.x)?

[Carl Eugen Hoyos]

What kind of output file was produced for bgr0 input?

[bmegli]

Tested with:

• Ubuntu 18.04
• FFmpeg version 7:3.4.8-0ubuntu0.2 (from system package)
• Kaby Lake platfrom
• i965 driver
• ​own bgr0 test, ​own yuyv422 test
• profile is set to FF_PROFILE_H264_HIGH (similar effect with other)

When enabled with trace debugging or queried with av_hwframe_transfer_get_formats following formats are reported:

[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x32315659 -> yuv420p.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x30323449 -> unknown.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x32595559 -> yuyv422.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x59565955 -> uyvy422.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x48323234 -> yuv422p.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x58424752 -> rgb0.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x58524742 -> bgr0.
[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x30313050 -> p010le.

I have tried bgr0, rgb0 and yuyv422 (all of which I or my users use directly).

The raw avcodec output dumped to file, queried with ffprobe returns:

Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1200k tbn, 60 tbc

So looks like YUV420P regardless of sw pixel format.

Doing the same with doc/examples/vaapi_encode.c would require at least some changes when
​loading data/setting linesize (bgr0 is 32 bpp single plane, yuyv422 is 16bpp single plane).

All three formats don't work in 4.2 (Ubuntu 20.04) with both i965 and iHD with same effect and message ("No usable encoding profile found")

[wenbin,chen]

I think the "[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x32315659 ->" means the pixel format supported by libva and the pixel format that can be encoded by h264 is a subset of them, because libva can also do video processing.
In 4.x version, you may need to do the pixel format transformation manually. You can use sws_scale or "hwupload" filter to do this.

[bmegli]

Thank you.

I think the "[AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x32315659 ->" means the pixel format supported by libva and the pixel format that can be encoded by h264 is a subset of them, because libva can also do video processing.

From the user perspective, in 3.x the conversion happened automatically (probably by libva or below).

1. Do you know if libva conversions are hardware or software?

From my user perspective using this path was like "let the hardware interface decide the optimal execution path", maybe it works with this kind of surface, maybe it performs additional software conversion on the way.

In 4.x version, you may need to do the pixel format transformation manually. You can use sws_scale or "hwupload" filter to do this.

Sure, I am aware of those possibilities.

sws_scale will result in (hardcoded) software conversion on the way.

So "hwupload" may be the possibility to recreate behavior from 3.x (performing conversion through libva whatever that means in the end).

[wenbin,chen]

Replying to bmegli:

1. Do you know if libva conversions are hardware or software?

sorry, I don't know how it does the conversion in 3.x version, but libva can do conversion in hardware.

I tried the "hwupload" and found that it only upload frames to hardware, so you may append "vaapi_scale=format=nv12" filter to do the conversion.

[bmegli]

Ok! I already have some workflows with vaapi_scale before encoding so testing it should be rather simple.

If this change of behavior in FFMpeg was intentional the ticket should probably end up closed with "wontfix"

From what I remember 4.x accepts 4:2:0 chroma formats other than nv12 (e.g. yuv420p) so there may be required some user code logic when to invoke vaapi_scale with nv12 conversion.

I am going to test it (but not soon) and leave some final feedback.

[bmegli]

I have just made some tests.

I first went with vaapi_scale e.g.

"format=vaapi,scale_vaapi=w=%d:h=%d:format=nv12"

But it occured that the critical difference (scale_vaapi is not necessary) lies in

AVHWFramesContext->sw_format

I got working bgr0 with both 3.4 (Ubuntu 18.04) and 4.2 (Ubuntu 20.04) by setting

frames_ctx->sw_format = AV_PIX_FMT_NV12;

In other test with 3.4 where I use sw_format as yuyv422 I see in the log:

[h264_vaapi @ 0x558bdfa854e0] Using nv12 as format of reconstructed frames.

From what I remember setting yuyv422 sw_format is what would cause error in 4.2, at least with H.264 and my Kaby Lake.

This means that for H.264 I may set NV12 as sw_format for 8 bit data and P010LE for 10 bit data and it is ok, at least on my Kaby Lake.

When looking at larger picture, beyond Kaby Lake it begins to look more complex.

​https://github.com/intel/media-driver#decodingencoding-features

Starting from Ice Lake there is support for HEVC 422 and 444 data
I wonder if setting frames_ctx->sw_format to AV_PIX_FMT_NV12 would result in loss of data in such case.

[wenbin,chen]

yes, it will lose data.

[wenbin,chen]

Close as wontfix as bmegli said.