ffmpeg-qsv: 6.0 regression - Error running VPP: undefined behavior (-16)

[nyanmisaka]

Summary of the bug:

Hybrid VA-API->QSV and software->VPP transcoding fails with VPP: undefined behavior (-16).

Worked fine in 5.1.2 so it should be a 6.0 regression caused by the recent {scale,deinterlace,overlay,vpp}_qsv refactor. The issue happens in both libmfx and libvpl builds.

Device context:

GPU: Arc A380 and A770 (DG2)
libva: 2.17.0
gmmlib: 22.3.4
media-driver: 23.1.3
onevpl-gpu: 23.1.3
mediasdk: 23.1.3

How to reproduce:

ffmpeg -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device qsv=qs@va -filter_hw_device qs \
-hwaccel vaapi -hwaccel_output_format vaapi -i test-clip.mkv -an -sn \
-vf hwmap=derive_device=qsv,format=qsv,scale_qsv=format=nv12 -f null -

Also fails with software decoding + VPP.

ffmpeg -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device qsv=qs@va -filter_hw_device qs \
-i test-clip.mkv -an -sn \
-vf hwupload=extra_hw_frames=64,format=qsv,scale_qsv=format=nv12 -f null -

Error messages:

MFXVideoVPP_RunFrameVPPAsync or MFXVideoCORE_SyncOperation failed in ​https://github.com/FFmpeg/FFmpeg/blob/ea3d24bbe3c58b171e55fe2151fc7ffaca3ab3d2/libavfilter/qsvvpp.c#L932

[Parsed_scale_qsv_1 @ 0x55d2b97f7480] Error running VPP: undefined behavior (-16)0x
Error while filtering: Internal bug, should not have happened
Failed to inject frame into filter network: Internal bug, should not have happened

Video stream info:

MPEG2 progressive 720x480 yuv420p

Stream #0:0(eng): Video: mpeg2video (Main), 1 reference frame, yuv420p(tv, progressive, left), 720x480 [SAR 32:27 DAR 16:9], SAR 853:720 DAR 853:480, 29.97 fps, 29.97 tbr, 1k tbn
    Metadata:
      BPS-eng         : 4353639
      DURATION-eng    : 00:43:03.914666666
      NUMBER_OF_FRAMES-eng: 61957
      NUMBER_OF_BYTES-eng: 1406178722
      SOURCE_ID-eng   : 0100E0
      _STATISTICS_WRITING_APP-eng: MakeMKV v1.17.3 darwin(arm64-release)
      _STATISTICS_WRITING_DATE_UTC-eng: 2023-03-19 04:25:37
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES SOURCE_ID
      DURATION        : 00:02:00.070000000
    Side data:
      cpb: bitrate max/min/avg: 9800000/0/0 buffer size: 1835008 vbv_delay: N/A

Video clip:

​https://f.losno.co/v/test-clip.mkv