corrupt output and rate control issues when using vp9_vaapi encoder in CBR or VBR mode

[armena]

Summary of the bug:

When using the vp9_vaapi encoder in CBR or VBR mode, the output video may be corrupted. At a dissolve from a full-screen logo to video from a camera, the output became garbled for several seconds. Also, in VBR mode, the output bitrate is approximately 1/2 of the requested value.

This was found using the i965 vaapi driver on an Intel Kabylake cpu. I attempted to reproduce using vp9enc (shipped with vaapi-utils) but was not able to do so, so my initial assumption is that this is an ffmpeg issue. I've been able to encode the same file successfully using other vaapi codecs (vp8, h264, hevc) with similar ffmpeg options.

How to reproduce:

/usr/local/bin/ffmpeg -vaapi_device /dev/dri/renderD128 -i /var/www/html/rpitv_test/20170121_whockey_union_p0.mp4 -vf "format=nv12,hwupload,deinterlace_vaapi,scale_vaapi=w=1920:h=1080" -c:v vp9_vaapi -b:v 4400k -bufsize 17600k -maxrate 4400k -g 150 -keyint_min 150 -bf 1 -bsf:v vp9_raw_reorder,vp9_superframe -an -f webm -dash 1 -r 29.97 20170121_whockey_union_p0-v4400.webm
ffmpeg version N-87873-g2e94490225 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18) 20170516
  configuration: --enable-libvorbis --enable-libvpx --enable-vaapi

[armena]

Also of note, I was not able to reproduce the corrupt output when encoding in constant-quality mode.

[jkqxz]

I can't reproduce this myself with the command-line given (I might be able to with the initial part of the input file?), but I can certainly believe this sort of problem will happen. The opaque rate control for VP9 in the Intel driver seems to be generally a bit flaky - e.g. see <​https://github.com/01org/intel-vaapi-driver/issues/87> (not just the hang, you can also get broken output without other failure with that setup). I suggest reporting your case there; ideally remove as many of the options as you can to still get it.

[armena]

Opened ​https://github.com/01org/intel-vaapi-driver/issues/284. There's a sample file there if you'd like to try to reproduce.

[Zhong,Li]

The CBR corruption issue is a vaapi driver issue. And a patch has been sent there.
The input clip (20170121_whockey_union_p0.mp4) to be transcoded is a special, which has many repeat frames. That why Mark cann't reproduce it without the input file.

[jkqxz]

Fixed in the i965 driver: <​https://github.com/01org/intel-vaapi-driver/commit/5a848a79e306941e4f78a0f96d6617456103fd0c>.