[regression] VAAPI: MPEG2 Encode Core Dump

[U. Artie Eoff]

Summary of the bug:

Any MPEG2 encoding (CQP with I, IP or IPB frames) on ffmpeg-vaapi (iHD and i965 drivers) fails since:

commit 29816e278f4ff22f50d9c831e3cf5f4d1b0e0214
Author: Mark Thompson <sw@jkqxz.net>
Date:   Sun Sep 23 22:52:57 2018 +0100

    vaapi_encode_mpeg2: Use common slice sizing code

How to reproduce:

% ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -v verbose \
  -f rawvideo -pix_fmt yuv420p -s:v 1280x720 \
  -i input.yuv -vf 'format=nv12,hwupload' -c:v mpeg2_vaapi -vframes 10 -y test.m2v

ffmpeg version N-92276-g29816e278f4f Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8 (GCC)
  configuration: --prefix=/home/uaeoff/Work/workspace/media/install --disable-static --enable-shared --enable-libdrm --enable-vaapi --enable-libmfx --disable-amf --disable-audiotoolbox --disable-cuda --disable-cuda-sdk --disable-cuvid --disable-d3d11va --disable-dxva2 --disable-libnpp --disable-mmal --disable-nvdec --disable-nvenc --disable-omx --disable-omx-rpi --disable-rkmpp --disable-v4l2-m2m --disable-vdpau --disable-videotoolbox --enable-nonfree --enable-gpl
  libavutil      56. 20.100 / 56. 20.100
  libavcodec     58. 34.100 / 58. 34.100
  libavformat    58. 19.102 / 58. 19.102
  libavdevice    58.  4.106 / 58.  4.106
  libavfilter     7. 38.100 /  7. 38.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
  libpostproc    55.  2.100 / 55.  2.100
[AVHWDeviceContext @ 0x83b480] Opened VA display via DRM device /dev/dri/renderD128.
[AVHWDeviceContext @ 0x83b480] libva: VA-API version 1.4.0
[AVHWDeviceContext @ 0x83b480] libva: va_getDriverName() returns 0
[AVHWDeviceContext @ 0x83b480] libva: Trying to open /home/uaeoff/Work/workspace/media/install/lib/dri/i965_drv_video.so
[AVHWDeviceContext @ 0x83b480] libva: Found init function __vaDriverInit_1_4
[AVHWDeviceContext @ 0x83b480] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x83b480] Initialised VAAPI connection: version 1.4
[AVHWDeviceContext @ 0x83b480] VAAPI driver: Intel i965 driver for Intel(R) Kaby Lake - 2.2.1.pre1 (2.2.1.pre1).
[AVHWDeviceContext @ 0x83b480] Driver not found in known nonstandard list, using standard behaviour.
[rawvideo @ 0x84c280] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from '/home/uaeoff/Work/workspace/media/src/input.yuv':
  Duration: 00:00:24.16, start: 0.000000, bitrate: 276480 kb/s
    Stream #0:0: Video: rawvideo, 1 reference frame (I420 / 0x30323449), yuv420p, 1280x720, 276480 kb/s, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg2video (mpeg2_vaapi))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0x85e9c0] w:1280 h:720 pixfmt:yuv420p tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
[auto_scaler_0 @ 0x861e40] w:iw h:ih flags:'bicubic' interl:0
[Parsed_format_0 @ 0x85dec0] auto-inserting filter 'auto_scaler_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_format_0'
[auto_scaler_0 @ 0x861e40] w:1280 h:720 fmt:yuv420p sar:0/1 -> w:1280 h:720 fmt:nv12 sar:0/1 flags:0x4
[mpeg2_vaapi @ 0x859ac0] Input surface format is nv12.
[mpeg2_vaapi @ 0x859ac0] Using VAAPI profile VAProfileMPEG2Main (1).
[mpeg2_vaapi @ 0x859ac0] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[mpeg2_vaapi @ 0x859ac0] Using VAAPI render target format YUV420 (0x1).
[mpeg2_vaapi @ 0x859ac0] Using constant-quality mode.
[mpeg2_vaapi @ 0x859ac0] Using intra, P- and B-frames (supported references: 1 / 1).
[mpeg2_vaapi @ 0x859ac0] All wanted packed headers available (wanted 0x3, found 0xb).
Output #0, mpeg2video, to 'test.m2v':
  Metadata:
    encoder         : Lavf58.19.102
    Stream #0:0: Video: mpeg2video (mpeg2_vaapi) (Main), 1 reference frame, vaapi_vld, 1280x720, q=-1--1, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.34.100 mpeg2_vaapi
ffmpeg: gen8_mfc.c:2284: gen8_mfc_mpeg2_pipeline_slice_group: Assertion `h_start_pos + slice_param->num_macroblocks <= width_in_mbs' failed.
Aborted (core dumped)

Software stack:

libva (master) heads/master-0-g285267586a3d  https://github.com/01org/libva
gmmlib (master) heads/master-0-gaedd82e2da01  https://github.com/intel/gmmlib
intel-media-driver (master) heads/master-0-g9d258e13eba1  https://github.com/intel/media-driver
intel-vaapi-driver (master) heads/master-0-gdc20b6405a53  https://github.com/01org/intel-vaapi-driver
msdk (master) heads/master-0-gc6b3d1cd4569  https://github.com/Intel-Media-SDK/MediaSDK
ffmpeg (master) heads/master-0-g29816e278f4f  https://git.ffmpeg.org/ffmpeg

[jkqxz]

c0692cb2bb3b51660b501f14bd344005c68df465

Thank you for the report!