Opened 3 years ago

Last modified 17 months ago

#9377 new defect

QSV MPEG2 => H264 transcode PSNR dropped by 32%

Reported by: eero-t Owned by:
Priority: important Component: undetermined
Version: git-master Keywords: qsv regression
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Between following FFmpeg commits:

  • 2021-06-19 2cf95f2dd9: lavfi/dnn_backend_openvino.c: fix crash when target is not specified
  • 2021-06-20 efc22e63e5: lavfi/vf_vpp_qsv: fix the time_base for outlink

PSNR for following transcode operation:

$ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v mpeg2_qsv -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -c:v h264_qsv -b:v 6000K -compression_level 7 -an -vframes 2400 -y 0024_HD17i7_1.0.h264

Has dropped by 32%, as measured by:

$ ffmpeg -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -i 0024_HD17i7_1.0.h264 -an -vframes 1199 -filter_complex psnr -f null -

(I'm using fewer frames in PSNR check than for transcode, because there were odd results if all frames are used for PSNR calcuation.)

Change History (11)

comment:1 by eero-t, 3 years ago

Full FFmpeg output for the transcode (from last night Git with the same bad PSNR):

ffmpeg version N-103273-g51a80aacce Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/opt/install/Nightly_2581 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      57.  4.100 / 57.  4.100
  libavcodec     59.  4.101 / 59.  4.101
  libavformat    59.  4.101 / 59.  4.101
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8.  2.100 /  8.  2.100
  libswscale      6.  0.100 /  6.  0.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
[AVHWDeviceContext @ 0x5608f04a3b40] libva: VA-API version 1.13.0
[AVHWDeviceContext @ 0x5608f04a3b40] libva: User requested driver 'iHD'
[AVHWDeviceContext @ 0x5608f04a3b40] libva: Trying to open /opt/install/Nightly_2581/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x5608f04a3b40] libva: Found init function __vaDriverInit_1_13
[AVHWDeviceContext @ 0x5608f04a3b40] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5608f04a3b40] Initialised VAAPI connection: version 1.13
[AVHWDeviceContext @ 0x5608f04a3b40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.2 (088b052c).
[AVHWDeviceContext @ 0x5608f04a3b40] Driver not found in known nonstandard list, using standard behaviour.
[AVHWDeviceContext @ 0x5608f04a3640] Initialize MFX session: API version is 1.35, implementation version is 1.35
[mpegvideo @ 0x5608f04f4800] max_analyze_duration 500000 reached at 500500 microseconds st:0
[mpegvideo @ 0x5608f04f4800] Estimating duration from bitrate, this may be inaccurate
WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv".
Input #0, mpegvideo, from '1920x1080i_29.97_20mb_mpeg2_high.mpv':
  Duration: 00:01:59.91, bitrate: 20000 kb/s
  Stream #0:0: Video: mpeg2video (Main), 1 reference frame, yuv420p(tv, bottom first, left), 1920x1080 [SAR 1:1 DAR 16:9], 20000 kb/s, 29.97 fps, 29.97 tbr, 1200k tbn
    Side data:
      cpb: bitrate max/min/avg: 20000000/0/0 buffer size: 9781248 vbv_delay: N/A
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video (mpeg2_qsv) -> h264 (h264_qsv))
Press [q] to stop, [?] for help
[AVHWDeviceContext @ 0x5608f0539a40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.2 (088b052c).
[AVHWDeviceContext @ 0x5608f0539a40] Driver not found in known nonstandard list, using standard behaviour.
[mpeg2_qsv @ 0x5608f04fbb40] Decoder: output is video memory surface
[AVHWDeviceContext @ 0x5608f0706800] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.2 (088b052c).
[AVHWDeviceContext @ 0x5608f0706800] Driver not found in known nonstandard list, using standard behaviour.
[mpeg2_qsv @ 0x5608f04fbb40] Decoder: output is video memory surface
[graph 0 input from stream 0:0 @ 0x5608f05801c0] w:1920 h:1080 pixfmt:qsv tb:1/1200000 fr:30000/1001 sar:1/1
[h264_qsv @ 0x5608f0539e00] Using input frames context (format qsv) with h264_qsv encoder.
[h264_qsv @ 0x5608f0539e00] Encoder: input is video memory surface
[h264_qsv @ 0x5608f0539e00] Using the variable bitrate (VBR) ratecontrol method
[h264_qsv @ 0x5608f0539e00] MFMode:2
[h264_qsv @ 0x5608f0539e00] profile: high; level: 40
[h264_qsv @ 0x5608f0539e00] GopPicSize: 250; GopRefDist: 4; GopOptFlag: closed ; IdrInterval: 0
[h264_qsv @ 0x5608f0539e00] TargetUsage: 7; RateControlMethod: VBR
[h264_qsv @ 0x5608f0539e00] BufferSizeInKB: 2250; InitialDelayInKB: 1125; TargetKbps: 6000; MaxKbps: 9000; BRCParamMultiplier: 1
[h264_qsv @ 0x5608f0539e00] NumSlice: 1; NumRefFrame: 3
[h264_qsv @ 0x5608f0539e00] RateDistortionOpt: OFF
[h264_qsv @ 0x5608f0539e00] RecoveryPointSEI: OFF IntRefType: 0; IntRefCycleSize: 0; IntRefQPDelta: 0
[h264_qsv @ 0x5608f0539e00] MaxFrameSize: 783360; MaxSliceSize: 0; 
[h264_qsv @ 0x5608f0539e00] BitrateLimit: ON; MBBRC: OFF; ExtBRC: OFF
[h264_qsv @ 0x5608f0539e00] Trellis: auto
[h264_qsv @ 0x5608f0539e00] VDENC: OFF
[h264_qsv @ 0x5608f0539e00] RepeatPPS: OFF; NumMbPerSlice: 0; LookAheadDS: 2x
[h264_qsv @ 0x5608f0539e00] AdaptiveI: OFF; AdaptiveB: OFF; BRefType: pyramid
[h264_qsv @ 0x5608f0539e00] MinQPI: 0; MaxQPI: 0; MinQPP: 0; MaxQPP: 0; MinQPB: 0; MaxQPB: 0
[h264_qsv @ 0x5608f0539e00] Entropy coding: CABAC; MaxDecFrameBuffering: 3
[h264_qsv @ 0x5608f0539e00] NalHrdConformance: ON; SingleSeiNalUnit: ON; VuiVclHrdParameters: OFF VuiNalHrdParameters: ON
[h264_qsv @ 0x5608f0539e00] FrameRateExtD: 1001; FrameRateExtN: 30000 
Output #0, h264, to '0024_HD17i7_1.0.h264':
  Metadata:
    encoder         : Lavf59.4.101
  Stream #0:0: Video: h264, 1 reference frame, qsv(tv, bottom coded first (swapped), left), 1920x1080 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 6000 kb/s, 29.97 fps, 29.97 tbn
    Metadata:
      encoder         : Lavc59.4.101 h264_qsv
    Side data:
      cpb: bitrate max/min/avg: 0/0/6000000 buffer size: 0 vbv_delay: N/A
frame=    1 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
*** 1 dup!
frame=  164 fps=0.0 q=30.0 size=    4096kB time=00:00:05.17 bitrate=6487.9kbits/s dup=1 drop=0 speed=10.1x    
frame=  324 fps=320 q=24.0 size=    7680kB time=00:00:10.51 bitrate=5985.9kbits/s dup=1 drop=0 speed=10.4x    
[mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10)
frame=  489 fps=323 q=24.0 size=   11264kB time=00:00:16.01 bitrate=5761.4kbits/s dup=1 drop=0 speed=10.6x    
[mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10)
    Last message repeated 1 times
frame=  654 fps=324 q=29.0 size=   15360kB time=00:00:21.52 bitrate=5846.7kbits/s dup=1 drop=0 speed=10.7x    
[mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10)
    Last message repeated 1 times
frame=  820 fps=326 q=29.0 size=   19968kB time=00:00:27.06 bitrate=6044.9kbits/s dup=1 drop=0 speed=10.7x    
[mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10)
    Last message repeated 1 times
frame=  986 fps=327 q=27.0 size=   23552kB time=00:00:32.59 bitrate=5918.5kbits/s dup=1 drop=0 speed=10.8x    
[mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10)
frame= 1149 fps=326 q=26.0 size=   27904kB time=00:00:38.03 bitrate=6009.5kbits/s dup=1 drop=0 speed=10.8x    
frame= 1311 fps=326 q=25.0 size=   32256kB time=00:00:43.44 bitrate=6082.4kbits/s dup=1 drop=0 speed=10.8x    
frame= 1472 fps=325 q=29.0 size=   36096kB time=00:00:48.81 bitrate=6057.5kbits/s dup=1 drop=0 speed=10.8x    
[mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10)
frame= 1635 fps=325 q=30.0 size=   40448kB time=00:00:54.25 bitrate=6107.4kbits/s dup=1 drop=0 speed=10.8x    
[mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10)
    Last message repeated 1 times
frame= 1799 fps=325 q=29.0 size=   44544kB time=00:00:59.72 bitrate=6109.6kbits/s dup=1 drop=0 speed=10.8x    
frame= 1960 fps=325 q=31.0 size=   48640kB time=00:01:05.09 bitrate=6120.9kbits/s dup=1 drop=0 speed=10.8x    
frame= 2123 fps=325 q=32.0 size=   52736kB time=00:01:10.53 bitrate=6124.6kbits/s dup=1 drop=0 speed=10.8x    
frame= 2289 fps=325 q=26.0 size=   56320kB time=00:01:16.07 bitrate=6064.6kbits/s dup=1 drop=0 speed=10.8x    
No more output streams to write to, finishing.
frame= 2400 fps=324 q=22.0 Lsize=   59139kB time=00:01:20.01 bitrate=6054.8kbits/s dup=1 drop=0 speed=10.8x    
video:59139kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Input file #0 (input/1920x1080i_29.97_20mb_mpeg2_high.mpv):
  Input stream #0:0 (video): 2404 packets read (200287518 bytes); 2400 frames decoded; 
  Total: 2404 packets (200287518 bytes) demuxed
Output file #0 (output/0024_HD17i7_1.0.h264):
  Output stream #0:0 (video): 2400 frames encoded; 2400 packets muxed (60558220 bytes); 
  Total: 2400 packets (60558220 bytes) muxed
[AVIOContext @ 0x5608f04fcf40] Statistics: 0 seeks, 232 writeouts
[AVIOContext @ 0x5608f04fd780] Statistics: 200310784 bytes read, 0 seeks

comment:2 by eero-t, 3 years ago

Full FFmpeg output for the PSNR calculation:

ffmpeg version N-103273-g51a80aacce Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/opt/install/Nightly_2581 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      57.  4.100 / 57.  4.100
  libavcodec     59.  4.101 / 59.  4.101
  libavformat    59.  4.101 / 59.  4.101
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8.  2.100 /  8.  2.100
  libswscale      6.  0.100 /  6.  0.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
[mpegvideo @ 0x55f44e597180] Estimating duration from bitrate, this may be inaccurate
Input #0, mpegvideo, from '1920x1080i_29.97_20mb_mpeg2_high.mpv':
  Duration: 00:01:59.91, bitrate: 20000 kb/s
  Stream #0:0: Video: mpeg2video (Main), yuv420p(tv, bottom first), 1920x1080 [SAR 1:1 DAR 16:9], 20000 kb/s, 29.97 fps, 29.97 tbr, 1200k tbn
    Side data:
      cpb: bitrate max/min/avg: 20000000/0/0 buffer size: 9781248 vbv_delay: N/A
Input #1, h264, from '0024_HD17i7_1.0.h264':
  Duration: N/A, bitrate: N/A
  Stream #1:0: Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1200k tbn
Stream mapping:
  Stream #0:0 (mpeg2video) -> psnr:main
  Stream #1:0 (h264) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
[Parsed_psnr_0 @ 0x55f4509aeb00] not matching timebases found between first input: 1/1200000 and second input 1/1200000, results may be incorrect!
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf59.4.101
  Stream #0:0: Video: wrapped_avframe, yuv420p(tv, bottom coded first (swapped)), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn (default)
    Metadata:
      encoder         : Lavc59.4.101 wrapped_avframe
frame=    1 fps=0.0 q=-0.0 size=N/A time=00:00:00.06 bitrate=N/A speed=2.09x    
frame=  218 fps=0.0 q=-0.0 size=N/A time=00:00:07.30 bitrate=N/A speed=13.7x    
frame=  441 fps=427 q=-0.0 size=N/A time=00:00:14.74 bitrate=N/A speed=14.3x    
frame=  662 fps=432 q=-0.0 size=N/A time=00:00:22.12 bitrate=N/A speed=14.4x    
frame=  886 fps=436 q=-0.0 size=N/A time=00:00:29.59 bitrate=N/A speed=14.5x    
frame= 1110 fps=438 q=-0.0 size=N/A time=00:00:37.07 bitrate=N/A speed=14.6x    
frame= 1199 fps=436 q=-0.0 Lsize=N/A time=00:00:40.04 bitrate=N/A speed=14.6x    
video:515kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x55f4509aeb00] PSNR y:27.630109 u:42.264918 v:39.133647 average:29.278348 min:15.279748 max:49.989809

comment:3 by wenbin,chen, 2 years ago

On my side:
2021-06-19 2cf95f2dd9: lavfi/dnn_backend_openvino.c: fix crash when target is not specified
PSNR y:27.821483 u:44.197586 v:43.288310 average:29.526907 min:12.768913 max:77.215654

2021-06-20 efc22e63e5: lavfi/vf_vpp_qsv: fix the time_base for outlink
PSNR y:27.821483 u:44.197586 v:43.288310 average:29.526907 min:12.768913 max:77.215654

2021-10-23 069f7831a2: avfilter/vf_owdenoise: relicense my code
PSNR y:47.045426 u:52.790706 v:52.699500 average:48.257933 min:35.995547 max:77.215654

On the latest commit, the PSNR increases. Seems someone fix the issue.
I cannot reproduce the PSNR drop between the two commits provided by you. Is it a stream specified problem?

comment:4 by eero-t, 2 years ago

I cannot reproduce the PSNR drop between the two commits provided by you. Is it a stream specified problem?

That's possible. Another possibility is that your test environment did not have new enough version of rest of the stack.

In my case I was using these versions:

I have data for those components from that day until yesterday, and there has been no improvement since the regression. This drop is still visible on all machines from which I have data (BXT, SKL, KBL, CML and TGL).

=> i.e. you can pick any later version of above components.

("drm-tip" kernel repo gets rebased to upstream regularly, so the above kernel commit ID is not available any more, but you can just use any any later commit.)

PSNR remains still at the earlier higher level when transcode is done using VA-API:
ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -c:v h264_vaapi -b:v 6000K -compression_level 7 -an -vframes 2400 -y 0024_HD17i7_1.0.h264

Or using MediaSDK tool for that:
sample_multi_transcode -i::mpeg2 1920x1080i_29.97_20mb_mpeg2_high.mpv -o::h264 output/0024_HD17i7_1.0.h264 -b 6000 -u 7 -n 2400 -async 4 -hw

=> i.e. this drop is really specific to how FFmpeg uses/configures MediaSDK (QSV).

(Between the listed FFmpeg commit IDs, there was also a minor update to drm-tip kernel, but I do not see how kernel change could affect video quality, and *only* for FFmpeg with QSV, but not FFmpeg with VA-API or MediaSDK.)

comment:5 by eero-t, 2 years ago

Note: the "A decode call did not consume any data: expect more data at input" messages in FFmpeg output are there also before PSNR regression, they are not new.

FYI: When asking FFmpeg to calculate SSIM between the original and transcoded file instead of PSNR, the regression is ~5%. Performance is not impacted.

comment:6 by wenbin,chen, 2 years ago

I use the same video as you. Now I can reproduce the issue.

comment:7 by wenbin,chen, 2 years ago

I add "-vsync passthrough" to command

ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v mpeg2_qsv -vsync passthrough -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -c:v h264_qsv -b:v 6000K -compression_level 7 -an -vframes 2400 -y 0024_HD17i7_1.0.h264

The psnr drop is unseen.

comment:8 by wenbin,chen, 2 years ago

Commit 971b4ac7336 adds code to map pts between AV_NOPTS_VALUE and MFX_TIMESTAMP_UNKNOWN. Before this commit, ffmpeg-qsv just pass pts directly to MediaSDK, but MediaSDK doesn't recognize AV_NOPTS_VALUE, so the output pts is a mess. Now ffmpeg-qsv pass MFX_TIMESTAMP_UNKNOWN to MediaSDK, and it recognize this value and try to correct output pts.
The MediaSDK set first pts to 0 and set second pts to 8008, but for this stream pts_duration should be 4004 so ffmpeg add a duplication frame between them.

comment:9 by eero-t, 2 years ago

According FFmpeg documentation, default for "-vsync" is "auto" which selects either "cfr" or "vfr". Because I specified bitrate, not FPS, I assume FFmpeg to have selected VFR, which seems to drop timestamp only if consecutive frames would have duplicate timestamp.

Are you saying that this is MediaSDK timestamp handling bug exposed by the FFmpeg change (to actually tell MediaSDK that there was no timestamp on some frame), one that can be only fixed on MediaSDK side? (will you file bug?)

As PSNR / SSIM calculations do frame-by-frame comparison, any change in frame timestamps during transcoding would make things go wrong as it affect frame content => should I always use "-vsync passtrough" (and "-an") on transcode operation intended for PSNR / SSIM calculations? And for performance comparisons?

PS. Is the cause for my other (more recent) PSNR bug similar: https://trac.ffmpeg.org/ticket/9481 ?

in reply to:  9 comment:10 by wenbin,chen, 2 years ago

Replying to eero-t:

According FFmpeg documentation, default for "-vsync" is "auto" which selects either "cfr" or "vfr". Because I specified bitrate, not FPS, I assume FFmpeg to have selected VFR, which seems to drop timestamp only if consecutive frames would have duplicate timestamp.

According to my debug, it chooses VSCFR which is a subset of CFR

Are you saying that this is MediaSDK timestamp handling bug exposed by the FFmpeg change (to actually tell MediaSDK that there was no timestamp on some frame), one that can be only fixed on MediaSDK side? (will you file bug?)

Not sure if it is bug, because all timestamp are correct except the first frame which is set to 0. I track the process how MediaSDK decode. For this video, the timestamp that ffmpeg passes to MSDK are all NOPTS, and MediaSDK can only get timestamp from B frame and it guess timestamp for other frames (I and P). The first two frames are I and P frame, so MSDK guess them wrong.

As PSNR / SSIM calculations do frame-by-frame comparison, any change in frame timestamps during transcoding would make things go wrong as it affect frame content => should I always use "-vsync passtrough" (and "-an") on transcode operation intended for PSNR / SSIM calculations? And for performance comparisons?

You'd better add this option in case ffmpeg inserts or removes frames and you don't know.

PS. Is the cause for my other (more recent) PSNR bug similar: https://trac.ffmpeg.org/ticket/9481 ?

I will check this issue.

comment:11 by Carl Eugen Hoyos, 17 months ago

Keywords: qsv regression added
Priority: normalimportant
Note: See TracTickets for help on using tickets.