FFmpeg MPEG-2 encode fails to "invalid RC mode" when iHD driver is used through VA-API instead of QSV

[eero-t]

Setup:

• Ubuntu 18.04
• Any HW with MPEG-2 HW acceleration support enabled by intel-driver (Intel Skylake, Kabylake, Coffeelake...)
• Latest FFmpeg, MediaSDK, libva, intel-driver and gmmlib built from sources at Github
• drm-git kernel (e.g. v4.19 or v4.20)

Bug can be reproduced by any FFmpeg MPEG-2 encoding with VA-API. For example:

ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 720x480p_30.00_4mb_h264_cabac.264 -c:v mpeg2_vaapi -b:v 2M -compression_level 4 -y output.mpg
...
Input #0, h264, from '720x480p_30.00_4mb_h264_cabac.264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(tv, smpte170m, progressive), 720x480 [SAR 10:11 DAR 15:11], 30 fps, 30 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mpeg2video (mpeg2_vaapi))
Press [q] to stop, [?] for help
[mpeg2_vaapi @ 0x564a286e6100] Driver does not support some wanted packed headers (wanted 0x3, found 0x10).
Assertion 0 && "Invalid RC mode." failed at src/libavcodec/vaapi_encode_mpeg2.c:545

Exactly same use-case works HW accelerated when done:

• through FFmpeg QSV API, or
• using MediaSDK sample transcode application

Looking at the FFmpeg source code, it seems to accept only VA_RC_CQP mode: ​https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/vaapi_encode_mpeg2.c#L523

And mode is something set before FFmpeg vaapi_encode_mpeg2_configure() is called, as it's not set in ff_cbs_init(): ​https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/cbs.c#L74

[Linjie.Fu]

VA_RC_VBR mode is set in vaapi_encode_init_rate_control:
​https://github.com/FFmpeg/FFmpeg/blob/eb81fd792fa88f9015b5e99e4940a464de9182a5/libavcodec/vaapi_encode.c#L1362

And calls ctx->codec->configure() in ff_vaapi_encode_init to set vaapi_encode_mpeg2_configure.

You may try to use default CQP mode by remove -b:v 2M, or add support for VBR and CBR:

--- a/libavcodec/vaapi_encode_mpeg2.c
+++ b/libavcodec/vaapi_encode_mpeg2.c
@@ -541,6 +541,13 @@ static av_cold int vaapi_encode_mpeg2_configure(AVCodecContext *avctx)
                "%d / %d / %d for I- / P- / B-frames.\n",
                priv->quant_i, priv->quant_p, priv->quant_b);

+    } else if (ctx->va_rc_mode == VA_RC_CBR ||
+               ctx->va_rc_mode == VA_RC_VBR) {
+        // These still need to be set for quantiser_scale_code.
+        priv->quant_i = 10;
+        priv->quant_p = 10;
+        priv->quant_b = 10;
+
     } else {
         av_assert0(0 && "Invalid RC mode.");
     }

[eero-t]

I can verify that either of these will allow FFmpeg to work.

However, with the patch, I get much worse PSNR / SSIM with VA-API:

ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 720x480p_30.00_4mb_cabac.264 -c:v mpeg2_vaapi -b:v 2M -compression_level 4 -y output.mpg
...
ffmpeg -i 720x480p_30.00_4mb_cabac.264 -i output.mpg -filter_complex "ssim;[0:v][1:v]psnr" -frames 2400 -f null -
...
SSIM Y:0.592489 (3.898609) U:0.864354 (8.675915) V:0.905782 (10.258669) All:0.690015 (5.086599)
PSNR y:16.888131 u:27.188619 v:30.218137 average:18.499885 min:13.885003 max:36.662048

than with QSV:

ffmpeg -hwaccel qsv -c:v h264_qsv -i 720x480p_30.00_4mb_h264_cabac.264 -c:v mpeg2_qsv -b:v 2M -compression_level 4 -y output.mpg
...
ffmpeg -i 720x480p_30.00_4mb_cabac.264 -i output.mpg -filter_complex "ssim;[0:v][1:v]psnr" -frames 2400 -f null -
...
SSIM Y:0.963254 (14.347895) U:0.976697 (16.325860) V:0.976484 (16.286427) All:0.967700 (14.907913)
PSNR y:37.747504 u:44.358034 v:44.510943 average:39.066017 min:31.195431 max:52.336077

I.e. average PSNR drops to half, from 39 with QSV, to 18.5 with VA-API.

In case it matters, there are following additional warnings with VA-API:

[mpeg2_vaapi @ 0x56228b35f100] Driver does not support some wanted packed headers (wanted 0x3, found 0x10).
[mpeg2_vaapi @ 0x56228b35f100] Sample aspect ratio 10:11 is not representable, signalling square pixels instead.

However, if I drop the "-b:v 2M" option, I still see these extra warnings, but PSNR with VA-API is close to one with QSV.

In few other tests I did, where output is H264 encoded, there was no PSNR/SSIM metrics difference between using VA-API and QSV.

[Carl Eugen Hoyos]

Please provide your (simplified) command line together with the complete, uncut console output to make this a valid ticket.

[Linjie.Fu]

I think the psnr issue was caused by the mismatch between FFmpeg and iHD driver.

In FFmpeg VAAPI level, hrd_buffer_size and hrd_initial_buffer_fullness will be set in vaapi_encode_init_rate_control:
​https://github.com/FFmpeg/FFmpeg/blob/282a4718576d6928b6c5900db89b45d83556407a/libavcodec/vaapi_encode.c#L1305
with a higher priority (vbvBuffer_size set in vaapi_encode_mpeg2_init_sequence_params will be covered in driver).

The mismatch exists in the unit of the m_vbvBufferSize:
ffmpeg vaapi: in bits;
mpeg2 in driver: in 16kbits;

Another problem is the m_rateControlMethod was set to RATECONTROL_CBR if calling DdiEncodeMpeg2::ParseMiscParamVBV in driver.

One way to solve this PSNR issue in FFmpeg level is set the bufsize specially for mpeg2 as the unit of 16kb, and discard the default CBR mode in driver.

Another way is to fix this all in driver level:
​https://github.com/intel/media-driver/pull/495

[Linjie.Fu]

Try this to fix in FFmpeg level, and discard the default CBR mode in driver as well:
​https://patchwork.ffmpeg.org/patch/11770/

[eero-t]

After building latest FFmpeg from git with the above bitrate patch, and this:

​​https://patchwork.ffmpeg.org/patch/11770/

I get similar SSIM & PSNR values as with QSV, so this looks good.

Still get the same warnings, but I guess they can be ignored:

[mpeg2_vaapi @ 0x56458668d9c0] Driver does not support some wanted packed headers (wanted 0x3, found 0x10).
[mpeg2_vaapi @ 0x56458668d9c0] Sample aspect ratio 10:11 is not representable, signalling square pixels instead.
[mpeg @ 0x564586689e80] VBV buffer size not set, using default size of 230KB

[Linjie.Fu]

Replying to eero-t:

After building latest FFmpeg from git with the above bitrate patch, and this:

​​https://patchwork.ffmpeg.org/patch/11770/

I get similar SSIM & PSNR values as with QSV, so this looks good.

Still get the same warnings, but I guess they can be ignored:

[mpeg2_vaapi @ 0x56458668d9c0] Driver does not support some wanted packed headers (wanted 0x3, found 0x10).

Driver only supports VA_ENC_PACKED_HEADER_RAW_DATA for mpeg2 profile
​https://github.com/intel/media-driver/blob/2d4ae991d359389b594f6e67080909029640b690/media_driver/linux/common/ddi/media_libva_caps.cpp#L540
But desired was set to VA_ENC_PACKED_HEADER_SEQUENCE | VA_ENC_PACKED_HEADER_SEQUENCE
I think a patch can be sent to avoid this warning.

[mpeg2_vaapi @ 0x56458668d9c0] Sample aspect ratio 10:11 is not representable, signalling square pixels instead.

This is due to unprovided aspected ratio. Maybe could define a new ration if not provided like h264 does.

[mpeg @ 0x564586689e80] VBV buffer size not set, using default size of 230KB
}}}

[jkqxz]

1e0fac76639e31eea48aa315cbca89aeb4761fde

Replying to eero-t:

Still get the same warnings, but I guess they can be ignored:

[mpeg2_vaapi @ 0x56458668d9c0] Driver does not support some wanted packed headers (wanted 0x3, found 0x10).

This seems to be an iHD driver issue - it doesn't support the required sequence and picture headers, so we can't fill the extradata in a consistent way. In my testing the i965 driver does the right thing here on the same hardware, so it will only be a software problem.

[mpeg2_vaapi @ 0x56458668d9c0] Sample aspect ratio 10:11 is not representable, signalling square pixels instead.

That's a limitation of MPEG-2 - it only supports a specific set of aspect ratios (see H.262 table 6-3). If the ratio of your input stream isn't on the list then it falls back to declaring that the stream has square pixels (and gives you the warning to say it has done that).

[eero-t]

Verified, I don't anymore get the same error and everything works now using QSV & VA-API backends with iHD driver on GEN9+ Core devices.

On BXT / APL devices I get now following error:

ffmpeg version N-93252-gf948082e5f Copyright (c) 2000-2019 the FFmpeg developers
...
[AVHWDeviceContext @ 0x555a5dcbb6c0] Initialised VAAPI connection: version 1.4
[AVHWDeviceContext @ 0x555a5dcbb6c0] VAAPI driver: Intel iHD driver - 1.0.0.
[AVHWDeviceContext @ 0x555a5dcbb6c0] Driver not found in known nonstandard list, using standard behaviour.
[h264 @ 0x555a5dcf64c0] Reinit context to 720x480, pix_fmt: yuv420p
[h264 @ 0x555a5dcf54c0] max_analyze_duration 500000 reached at 500000 microseconds st:0
Input #0, h264, from '720x480p_30.00_4mb_h264_cabac_180s.264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(tv, smpte170m, progressive, left), 720x480 [SAR 10:11 DAR 15:11], 30 fps, 30 tbr, 1200k tbn, 60 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mpeg2video (mpeg2_vaapi))
Press [q] to stop, [?] for help
[h264 @ 0x555a5dcf9c80] Reinit context to 720x480, pix_fmt: vaapi_vld
[graph 0 input from stream 0:0 @ 0x555a5dd30300] w:720 h:480 pixfmt:vaapi_vld tb:1/1200000 fr:30/1 sar:10/11 sws_param:flags=2
[mpeg2_vaapi @ 0x555a5dd1ba40] Input surface format is nv12.
[mpeg2_vaapi @ 0x555a5dd1ba40] Using VAAPI profile VAProfileMPEG2Main (1).
[mpeg2_vaapi @ 0x555a5dd1ba40] No usable encoding entrypoint found for profile VAProfileMPEG2Main (1).
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

And following with the i965 VA driver on all other devices:

[mpeg2_vaapi @ 0x55613f89e700] Driver does not support any RC mode compatible with selected options (supported modes: CQP).

Should I file separate ticket(s) for these? (Where?)