VA-API + QSV H264 => MPEG2 transcode PSNR dropped by 25%

[eero-t]

Between following FFmpeg commits:

• 2021-09-22 a487635b85: avcodec/dynamic_hdr10_plus: check size before using it
• 2021-09-23 6e26015a6b: avformat/astenc: Simplify writing padding

PSNR for following transcode operation:

$ ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 720x480p_30.00_4mb_h264_cabac_180s.264 -c:v mpeg2_vaapi -b:v 2000K -compression_level 4 -an -vframes 4800 -y 0039_SD03MP2_1.0.mpg

Has dropped by 25%, as measured by:

$ ffmpeg -i 720x480p_30.00_4mb_h264_cabac_180s.264 -i 0039_SD03MP2_1.0.mpg -an -vframes 2399 -filter_complex psnr -f null -

(I'm using fewer frames in PSNR check than for transcode, because there were odd results if all frames are used for PSNR calculation.)

Between same commits, PSNR for similar transcode done using QSV:
ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i 720x480p_30.00_4mb_h264_cabac_180s.264 -c:v mpeg2_qsv -b:v 2000K -compression_level 4 -an -vframes 4800 -y 0039_SD03MP2_1.0.mpg

also dropped to same value, but the drop for QSV transcode PSNR was smaller because it had dropped from the original VA-API PSNR value by 7% already earlier, between these FFmpeg commits:

• 2021-03-14 1af4885014: avcodec: add a mention about get_encode_buffer in the old encode API doxy
• 2021-03-15 63344337f9: libavformat/hls: Reset options after open_url_keepalive() fails

(I did not pay attention to that earlier/smaller QSV PSNR drop because there was at the same time marginal (<1%) perf increase, and updates in underlying drm-tip kernel & MediaSDK components.)

See also #9377.

Now results of both VA-API and QSV transcode operations give same (low) PSNR as using MediaSDK tool to do same transcode has been giving:
sample_multi_transcode -i::h264 720x480p_30.00_4mb_h264_cabac_180s.264 -o::mpeg2 0039_SD03MP2_1.0.mpg -b 2000 -u 4 -n 4800 -async 4 -hw

[eero-t]

These H264 -> MPEG2 transcode results PSNR drops are visible in the data from all the machines I have which support encoding to MPEG2 (SKL, KBL, CML, TGL).

Here's FFmpeg output for the VA-API transcode:

ffmpeg version N-103794-g6e26015a6b Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/opt/Nightly_2618 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      57.  6.100 / 57.  6.100
  libavcodec     59.  9.100 / 59.  9.100
  libavformat    59.  5.100 / 59.  5.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8.  9.100 /  8.  9.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
[AVHWDeviceContext @ 0x55f1f31ca480] libva: VA-API version 1.13.0
[AVHWDeviceContext @ 0x55f1f31ca480] libva: User environment variable requested driver 'iHD'
[AVHWDeviceContext @ 0x55f1f31ca480] libva: Trying to open /opt/Nightly_2618/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x55f1f31ca480] libva: Found init function __vaDriverInit_1_13
[AVHWDeviceContext @ 0x55f1f31ca480] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55f1f31ca480] Initialised VAAPI connection: version 1.13
[AVHWDeviceContext @ 0x55f1f31ca480] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.4 (926db969).
[AVHWDeviceContext @ 0x55f1f31ca480] Driver not found in known nonstandard list, using standard behaviour.
[h264 @ 0x55f1f320ba80] Reinit context to 720x480, pix_fmt: yuv420p
[h264 @ 0x55f1f3209e80] max_analyze_duration 500000 reached at 500000 microseconds st:0
Input #0, h264, from '720x480p_30.00_4mb_h264_cabac_180s.264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(tv, smpte170m, progressive, left), 720x480 [SAR 10:11 DAR 15:11], 30 fps, 30 tbr, 1200k tbn
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mpeg2video (mpeg2_vaapi))
Press [q] to stop, [?] for help
[h264 @ 0x55f1f3212940] Reinit context to 720x480, pix_fmt: vaapi
[graph 0 input from stream 0:0 @ 0x55f1f3240800] w:720 h:480 pixfmt:vaapi tb:1/1200000 fr:30/1 sar:10/11
[mpeg2_vaapi @ 0x55f1f3211340] Using input frames context (format vaapi) with mpeg2_vaapi encoder.
[mpeg2_vaapi @ 0x55f1f3211340] Input surface format is nv12.
[mpeg2_vaapi @ 0x55f1f3211340] Using VAAPI profile VAProfileMPEG2Main (1).
[mpeg2_vaapi @ 0x55f1f3211340] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[mpeg2_vaapi @ 0x55f1f3211340] Using VAAPI render target format YUV420 (0x1).
[mpeg2_vaapi @ 0x55f1f3211340] RC mode: VBR.
[mpeg2_vaapi @ 0x55f1f3211340] RC target: 50% of 4000000 bps over 500 ms.
[mpeg2_vaapi @ 0x55f1f3211340] RC buffer: 2000000 bits, initial fullness 1500000 bits.
[mpeg2_vaapi @ 0x55f1f3211340] RC framerate: 30/1 (30.00 fps).
[mpeg2_vaapi @ 0x55f1f3211340] Using intra, P- and B-frames (supported references: 1 / 1).
[mpeg2_vaapi @ 0x55f1f3211340] Driver does not support some wanted packed headers (wanted 0x3, found 0x10).
[mpeg2_vaapi @ 0x55f1f3211340] Sample aspect ratio 10:11 is not representable, signalling square pixels instead.
[mpeg @ 0x55f1f322bbc0] VBV buffer size not set, using default size of 230KB
If you want the mpeg file to be compliant to some specification
Like DVD, VCD or others, make sure you set the correct buffer size
Output #0, mpeg, to '0039_SD03MP2_1.0.mpg':
  Metadata:
    encoder         : Lavf59.5.100
  Stream #0:0: Video: mpeg2video (Main), 1 reference frame, vaapi(tv, smpte170m, progressive, left), 720x480 (0x0) [SAR 10:11 DAR 15:11], q=2-31, 2000 kb/s, 30 fps, 90k tbn
    Metadata:
      encoder         : Lavc59.9.100 mpeg2_vaapi
frame=    1 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
frame=  684 fps=0.0 q=-0.0 size=    5376kB time=00:00:22.70 bitrate=1940.1kbits/s speed=44.6x    
frame= 1408 fps=1396 q=-0.0 size=   11264kB time=00:00:46.83 bitrate=1970.3kbits/s speed=46.4x    
frame= 2132 fps=1413 q=-0.0 size=   17152kB time=00:01:10.96 bitrate=1979.9kbits/s speed=  47x    
frame= 2846 fps=1416 q=-0.0 size=   23040kB time=00:01:34.76 bitrate=1991.7kbits/s speed=47.2x    
frame= 3574 fps=1424 q=-0.0 size=   28928kB time=00:01:59.03 bitrate=1990.9kbits/s speed=47.4x    
frame= 4301 fps=1429 q=-0.0 size=   34816kB time=00:02:23.26 bitrate=1990.8kbits/s speed=47.6x    
No more output streams to write to, finishing.
frame= 4800 fps=1429 q=-0.0 Lsize=   39106kB time=00:02:39.93 bitrate=2003.1kbits/s speed=47.6x    
video:38934kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.442416%
Input file #0 (input/720x480p_30.00_4mb_h264_cabac_180s.264):
  Input stream #0:0 (video): 4813 packets read (83249568 bytes); 4801 frames decoded; 
  Total: 4813 packets (83249568 bytes) demuxed
Output file #0 (output/0039_SD03MP2_1.0.mpg):
  Output stream #0:0 (video): 4800 frames encoded; 4800 packets muxed (39868161 bytes); 
  Total: 4800 packets (39868161 bytes) muxed
[AVIOContext @ 0x55f1f320fdc0] Statistics: 0 seeks, 153 writeouts
[AVIOContext @ 0x55f1f3213500] Statistics: 83263488 bytes read, 0 seeks

[wenbin,chen]

I found the problem is not about VA-API or QSV. Between two commits, the encoded stream is identical. It is PSNR calculation command that returns different results. If I run PSNR calculation command on the same output video on two commits, I get different result.
The decoded yuv data of build-in 264 decoder on two commit have same md5 value, so seems it is an issue of PSNR filter.

[Balling]

so seems it is an issue of PSNR filter.

Or a bugfix. After all MS SSIM was fixed in fcc0424c933742c8fc852371e985d16b6eb4bfe9 and it did change the output.

[eero-t]

FYI: Quality reported by SSIM for that transcode operation dropped also between these commits. SSIM drop (9%) was smaller than with PSNR (25%).

[eero-t]

Note: using "-vsync passthrough" is not a workaround for this regression, like it is for #9377. Neither with QSV nor with VA-API.

[Balling]

You need to set all (at least 3) parameters for PSNR for it to work on the same frames. Please also rename an issue.

I will quote Paul on this:

"Use "setpts=PTS-STARTPTS,split=2[a][b],[a][1:v]ssim;[b][1:v]psnr".
Your first input pts does not start from 0.

You also need to use settb to set both inputs to same timebase, and do that before using setpts filters."