Opened 3 years ago
Last modified 18 months ago
#9377 new defect
QSV MPEG2 => H264 transcode PSNR dropped by 32%
| Reported by: | eero-t | Owned by: | |
|---|---|---|---|
| Priority: | important | Component: | undetermined |
| Version: | git-master | Keywords: | qsv regression |
| Cc: | Blocked By: | ||
| Blocking: | Reproduced by developer: | no | |
| Analyzed by developer: | no |
Description
Between following FFmpeg commits:
- 2021-06-19 2cf95f2dd9: lavfi/dnn_backend_openvino.c: fix crash when target is not specified
- 2021-06-20 efc22e63e5: lavfi/vf_vpp_qsv: fix the time_base for outlink
PSNR for following transcode operation:
$ ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v mpeg2_qsv -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -c:v h264_qsv -b:v 6000K -compression_level 7 -an -vframes 2400 -y 0024_HD17i7_1.0.h264
Has dropped by 32%, as measured by:
$ ffmpeg -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -i 0024_HD17i7_1.0.h264 -an -vframes 1199 -filter_complex psnr -f null -
(I'm using fewer frames in PSNR check than for transcode, because there were odd results if all frames are used for PSNR calcuation.)
Change History (11)
comment:1 by , 3 years ago
comment:2 by , 3 years ago
Full FFmpeg output for the PSNR calculation:
ffmpeg version N-103273-g51a80aacce Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
configuration: --prefix=/opt/install/Nightly_2581 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
libavutil 57. 4.100 / 57. 4.100
libavcodec 59. 4.101 / 59. 4.101
libavformat 59. 4.101 / 59. 4.101
libavdevice 59. 0.101 / 59. 0.101
libavfilter 8. 2.100 / 8. 2.100
libswscale 6. 0.100 / 6. 0.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100
[mpegvideo @ 0x55f44e597180] Estimating duration from bitrate, this may be inaccurate
Input #0, mpegvideo, from '1920x1080i_29.97_20mb_mpeg2_high.mpv':
Duration: 00:01:59.91, bitrate: 20000 kb/s
Stream #0:0: Video: mpeg2video (Main), yuv420p(tv, bottom first), 1920x1080 [SAR 1:1 DAR 16:9], 20000 kb/s, 29.97 fps, 29.97 tbr, 1200k tbn
Side data:
cpb: bitrate max/min/avg: 20000000/0/0 buffer size: 9781248 vbv_delay: N/A
Input #1, h264, from '0024_HD17i7_1.0.h264':
Duration: N/A, bitrate: N/A
Stream #1:0: Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1200k tbn
Stream mapping:
Stream #0:0 (mpeg2video) -> psnr:main
Stream #1:0 (h264) -> psnr:reference
psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
[Parsed_psnr_0 @ 0x55f4509aeb00] not matching timebases found between first input: 1/1200000 and second input 1/1200000, results may be incorrect!
Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf59.4.101
Stream #0:0: Video: wrapped_avframe, yuv420p(tv, bottom coded first (swapped)), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn (default)
Metadata:
encoder : Lavc59.4.101 wrapped_avframe
frame= 1 fps=0.0 q=-0.0 size=N/A time=00:00:00.06 bitrate=N/A speed=2.09x
frame= 218 fps=0.0 q=-0.0 size=N/A time=00:00:07.30 bitrate=N/A speed=13.7x
frame= 441 fps=427 q=-0.0 size=N/A time=00:00:14.74 bitrate=N/A speed=14.3x
frame= 662 fps=432 q=-0.0 size=N/A time=00:00:22.12 bitrate=N/A speed=14.4x
frame= 886 fps=436 q=-0.0 size=N/A time=00:00:29.59 bitrate=N/A speed=14.5x
frame= 1110 fps=438 q=-0.0 size=N/A time=00:00:37.07 bitrate=N/A speed=14.6x
frame= 1199 fps=436 q=-0.0 Lsize=N/A time=00:00:40.04 bitrate=N/A speed=14.6x
video:515kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x55f4509aeb00] PSNR y:27.630109 u:42.264918 v:39.133647 average:29.278348 min:15.279748 max:49.989809
comment:3 by , 2 years ago
On my side:
2021-06-19 2cf95f2dd9: lavfi/dnn_backend_openvino.c: fix crash when target is not specified
PSNR y:27.821483 u:44.197586 v:43.288310 average:29.526907 min:12.768913 max:77.215654
2021-06-20 efc22e63e5: lavfi/vf_vpp_qsv: fix the time_base for outlink
PSNR y:27.821483 u:44.197586 v:43.288310 average:29.526907 min:12.768913 max:77.215654
2021-10-23 069f7831a2: avfilter/vf_owdenoise: relicense my code
PSNR y:47.045426 u:52.790706 v:52.699500 average:48.257933 min:35.995547 max:77.215654
On the latest commit, the PSNR increases. Seems someone fix the issue.
I cannot reproduce the PSNR drop between the two commits provided by you. Is it a stream specified problem?
comment:4 by , 2 years ago
I cannot reproduce the PSNR drop between the two commits provided by you. Is it a stream specified problem?
That's possible. Another possibility is that your test environment did not have new enough version of rest of the stack.
In my case I was using these versions:
- git://anongit.freedesktop.org/drm-tip at 940478dea0bccfe1ef632c6cbfca66c5d3fc345e 2021-06-19 drm-tip: 2021y-06m-19d-21h-09m-09s UTC integration manifest
- git://github.com/intel/gmmlib at 9253eb8b949515d2a5030f99163a35567fe81838 2021-06-17 Updating the Licensening details
- git://github.com/intel/libva at c7c404135a2dee09af4d55412e797d9904802eb9 2021-06-17 update NEWS for 2.12.0
- git://github.com/intel/media-driver at 08cbe4c102d1fe8ba1f5b699b1a0c1c831aef8f7 2021-06-18 set the picture flag to be invalid frame if the ref frame is invalid
- git://github.com/Intel-Media-SDK/MediaSDK at 883142f1f10186c84cadc9d5bf63d96934270567 2021-06-11 [VP9e] Add WebRTC mode
I have data for those components from that day until yesterday, and there has been no improvement since the regression. This drop is still visible on all machines from which I have data (BXT, SKL, KBL, CML and TGL).
=> i.e. you can pick any later version of above components.
("drm-tip" kernel repo gets rebased to upstream regularly, so the above kernel commit ID is not available any more, but you can just use any any later commit.)
PSNR remains still at the earlier higher level when transcode is done using VA-API:
ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -c:v h264_vaapi -b:v 6000K -compression_level 7 -an -vframes 2400 -y 0024_HD17i7_1.0.h264
Or using MediaSDK tool for that:
sample_multi_transcode -i::mpeg2 1920x1080i_29.97_20mb_mpeg2_high.mpv -o::h264 output/0024_HD17i7_1.0.h264 -b 6000 -u 7 -n 2400 -async 4 -hw
=> i.e. this drop is really specific to how FFmpeg uses/configures MediaSDK (QSV).
(Between the listed FFmpeg commit IDs, there was also a minor update to drm-tip kernel, but I do not see how kernel change could affect video quality, and *only* for FFmpeg with QSV, but not FFmpeg with VA-API or MediaSDK.)
comment:5 by , 2 years ago
Note: the "A decode call did not consume any data: expect more data at input" messages in FFmpeg output are there also before PSNR regression, they are not new.
FYI: When asking FFmpeg to calculate SSIM between the original and transcoded file instead of PSNR, the regression is ~5%. Performance is not impacted.
comment:7 by , 2 years ago
I add "-vsync passthrough" to command
ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v mpeg2_qsv -vsync passthrough -i 1920x1080i_29.97_20mb_mpeg2_high.mpv -c:v h264_qsv -b:v 6000K -compression_level 7 -an -vframes 2400 -y 0024_HD17i7_1.0.h264
The psnr drop is unseen.
comment:8 by , 2 years ago
Commit 971b4ac7336 adds code to map pts between AV_NOPTS_VALUE and MFX_TIMESTAMP_UNKNOWN. Before this commit, ffmpeg-qsv just pass pts directly to MediaSDK, but MediaSDK doesn't recognize AV_NOPTS_VALUE, so the output pts is a mess. Now ffmpeg-qsv pass MFX_TIMESTAMP_UNKNOWN to MediaSDK, and it recognize this value and try to correct output pts.
The MediaSDK set first pts to 0 and set second pts to 8008, but for this stream pts_duration should be 4004 so ffmpeg add a duplication frame between them.
follow-up: 10 comment:9 by , 2 years ago
According FFmpeg documentation, default for "-vsync" is "auto" which selects either "cfr" or "vfr". Because I specified bitrate, not FPS, I assume FFmpeg to have selected VFR, which seems to drop timestamp only if consecutive frames would have duplicate timestamp.
Are you saying that this is MediaSDK timestamp handling bug exposed by the FFmpeg change (to actually tell MediaSDK that there was no timestamp on some frame), one that can be only fixed on MediaSDK side? (will you file bug?)
As PSNR / SSIM calculations do frame-by-frame comparison, any change in frame timestamps during transcoding would make things go wrong as it affect frame content => should I always use "-vsync passtrough" (and "-an") on transcode operation intended for PSNR / SSIM calculations? And for performance comparisons?
PS. Is the cause for my other (more recent) PSNR bug similar: https://trac.ffmpeg.org/ticket/9481 ?
comment:10 by , 2 years ago
Replying to eero-t:
According FFmpeg documentation, default for "-vsync" is "auto" which selects either "cfr" or "vfr". Because I specified bitrate, not FPS, I assume FFmpeg to have selected VFR, which seems to drop timestamp only if consecutive frames would have duplicate timestamp.
According to my debug, it chooses VSCFR which is a subset of CFR
Are you saying that this is MediaSDK timestamp handling bug exposed by the FFmpeg change (to actually tell MediaSDK that there was no timestamp on some frame), one that can be only fixed on MediaSDK side? (will you file bug?)
Not sure if it is bug, because all timestamp are correct except the first frame which is set to 0. I track the process how MediaSDK decode. For this video, the timestamp that ffmpeg passes to MSDK are all NOPTS, and MediaSDK can only get timestamp from B frame and it guess timestamp for other frames (I and P). The first two frames are I and P frame, so MSDK guess them wrong.
As PSNR / SSIM calculations do frame-by-frame comparison, any change in frame timestamps during transcoding would make things go wrong as it affect frame content => should I always use "-vsync passtrough" (and "-an") on transcode operation intended for PSNR / SSIM calculations? And for performance comparisons?
You'd better add this option in case ffmpeg inserts or removes frames and you don't know.
PS. Is the cause for my other (more recent) PSNR bug similar: https://trac.ffmpeg.org/ticket/9481 ?
I will check this issue.
comment:11 by , 18 months ago
| Keywords: | qsv regression added |
|---|---|
| Priority: | normal → important |
Full FFmpeg output for the transcode (from last night Git with the same bad PSNR):
ffmpeg version N-103273-g51a80aacce Copyright (c) 2000-2021 the FFmpeg developers built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04) configuration: --prefix=/opt/install/Nightly_2581 --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc libavutil 57. 4.100 / 57. 4.100 libavcodec 59. 4.101 / 59. 4.101 libavformat 59. 4.101 / 59. 4.101 libavdevice 59. 0.101 / 59. 0.101 libavfilter 8. 2.100 / 8. 2.100 libswscale 6. 0.100 / 6. 0.100 libswresample 4. 0.100 / 4. 0.100 libpostproc 56. 0.100 / 56. 0.100 [AVHWDeviceContext @ 0x5608f04a3b40] libva: VA-API version 1.13.0 [AVHWDeviceContext @ 0x5608f04a3b40] libva: User requested driver 'iHD' [AVHWDeviceContext @ 0x5608f04a3b40] libva: Trying to open /opt/install/Nightly_2581/lib/dri/iHD_drv_video.so [AVHWDeviceContext @ 0x5608f04a3b40] libva: Found init function __vaDriverInit_1_13 [AVHWDeviceContext @ 0x5608f04a3b40] libva: va_openDriver() returns 0 [AVHWDeviceContext @ 0x5608f04a3b40] Initialised VAAPI connection: version 1.13 [AVHWDeviceContext @ 0x5608f04a3b40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.2 (088b052c). [AVHWDeviceContext @ 0x5608f04a3b40] Driver not found in known nonstandard list, using standard behaviour. [AVHWDeviceContext @ 0x5608f04a3640] Initialize MFX session: API version is 1.35, implementation version is 1.35 [mpegvideo @ 0x5608f04f4800] max_analyze_duration 500000 reached at 500500 microseconds st:0 [mpegvideo @ 0x5608f04f4800] Estimating duration from bitrate, this may be inaccurate WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv". Input #0, mpegvideo, from '1920x1080i_29.97_20mb_mpeg2_high.mpv': Duration: 00:01:59.91, bitrate: 20000 kb/s Stream #0:0: Video: mpeg2video (Main), 1 reference frame, yuv420p(tv, bottom first, left), 1920x1080 [SAR 1:1 DAR 16:9], 20000 kb/s, 29.97 fps, 29.97 tbr, 1200k tbn Side data: cpb: bitrate max/min/avg: 20000000/0/0 buffer size: 9781248 vbv_delay: N/A Stream mapping: Stream #0:0 -> #0:0 (mpeg2video (mpeg2_qsv) -> h264 (h264_qsv)) Press [q] to stop, [?] for help [AVHWDeviceContext @ 0x5608f0539a40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.2 (088b052c). [AVHWDeviceContext @ 0x5608f0539a40] Driver not found in known nonstandard list, using standard behaviour. [mpeg2_qsv @ 0x5608f04fbb40] Decoder: output is video memory surface [AVHWDeviceContext @ 0x5608f0706800] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.2 (088b052c). [AVHWDeviceContext @ 0x5608f0706800] Driver not found in known nonstandard list, using standard behaviour. [mpeg2_qsv @ 0x5608f04fbb40] Decoder: output is video memory surface [graph 0 input from stream 0:0 @ 0x5608f05801c0] w:1920 h:1080 pixfmt:qsv tb:1/1200000 fr:30000/1001 sar:1/1 [h264_qsv @ 0x5608f0539e00] Using input frames context (format qsv) with h264_qsv encoder. [h264_qsv @ 0x5608f0539e00] Encoder: input is video memory surface [h264_qsv @ 0x5608f0539e00] Using the variable bitrate (VBR) ratecontrol method [h264_qsv @ 0x5608f0539e00] MFMode:2 [h264_qsv @ 0x5608f0539e00] profile: high; level: 40 [h264_qsv @ 0x5608f0539e00] GopPicSize: 250; GopRefDist: 4; GopOptFlag: closed ; IdrInterval: 0 [h264_qsv @ 0x5608f0539e00] TargetUsage: 7; RateControlMethod: VBR [h264_qsv @ 0x5608f0539e00] BufferSizeInKB: 2250; InitialDelayInKB: 1125; TargetKbps: 6000; MaxKbps: 9000; BRCParamMultiplier: 1 [h264_qsv @ 0x5608f0539e00] NumSlice: 1; NumRefFrame: 3 [h264_qsv @ 0x5608f0539e00] RateDistortionOpt: OFF [h264_qsv @ 0x5608f0539e00] RecoveryPointSEI: OFF IntRefType: 0; IntRefCycleSize: 0; IntRefQPDelta: 0 [h264_qsv @ 0x5608f0539e00] MaxFrameSize: 783360; MaxSliceSize: 0; [h264_qsv @ 0x5608f0539e00] BitrateLimit: ON; MBBRC: OFF; ExtBRC: OFF [h264_qsv @ 0x5608f0539e00] Trellis: auto [h264_qsv @ 0x5608f0539e00] VDENC: OFF [h264_qsv @ 0x5608f0539e00] RepeatPPS: OFF; NumMbPerSlice: 0; LookAheadDS: 2x [h264_qsv @ 0x5608f0539e00] AdaptiveI: OFF; AdaptiveB: OFF; BRefType: pyramid [h264_qsv @ 0x5608f0539e00] MinQPI: 0; MaxQPI: 0; MinQPP: 0; MaxQPP: 0; MinQPB: 0; MaxQPB: 0 [h264_qsv @ 0x5608f0539e00] Entropy coding: CABAC; MaxDecFrameBuffering: 3 [h264_qsv @ 0x5608f0539e00] NalHrdConformance: ON; SingleSeiNalUnit: ON; VuiVclHrdParameters: OFF VuiNalHrdParameters: ON [h264_qsv @ 0x5608f0539e00] FrameRateExtD: 1001; FrameRateExtN: 30000 Output #0, h264, to '0024_HD17i7_1.0.h264': Metadata: encoder : Lavf59.4.101 Stream #0:0: Video: h264, 1 reference frame, qsv(tv, bottom coded first (swapped), left), 1920x1080 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 6000 kb/s, 29.97 fps, 29.97 tbn Metadata: encoder : Lavc59.4.101 h264_qsv Side data: cpb: bitrate max/min/avg: 0/0/6000000 buffer size: 0 vbv_delay: N/A frame= 1 fps=0.0 q=0.0 size= 0kB time=00:00:00.00 bitrate=N/A speed= 0x *** 1 dup! frame= 164 fps=0.0 q=30.0 size= 4096kB time=00:00:05.17 bitrate=6487.9kbits/s dup=1 drop=0 speed=10.1x frame= 324 fps=320 q=24.0 size= 7680kB time=00:00:10.51 bitrate=5985.9kbits/s dup=1 drop=0 speed=10.4x [mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10) frame= 489 fps=323 q=24.0 size= 11264kB time=00:00:16.01 bitrate=5761.4kbits/s dup=1 drop=0 speed=10.6x [mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10) Last message repeated 1 times frame= 654 fps=324 q=29.0 size= 15360kB time=00:00:21.52 bitrate=5846.7kbits/s dup=1 drop=0 speed=10.7x [mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10) Last message repeated 1 times frame= 820 fps=326 q=29.0 size= 19968kB time=00:00:27.06 bitrate=6044.9kbits/s dup=1 drop=0 speed=10.7x [mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10) Last message repeated 1 times frame= 986 fps=327 q=27.0 size= 23552kB time=00:00:32.59 bitrate=5918.5kbits/s dup=1 drop=0 speed=10.8x [mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10) frame= 1149 fps=326 q=26.0 size= 27904kB time=00:00:38.03 bitrate=6009.5kbits/s dup=1 drop=0 speed=10.8x frame= 1311 fps=326 q=25.0 size= 32256kB time=00:00:43.44 bitrate=6082.4kbits/s dup=1 drop=0 speed=10.8x frame= 1472 fps=325 q=29.0 size= 36096kB time=00:00:48.81 bitrate=6057.5kbits/s dup=1 drop=0 speed=10.8x [mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10) frame= 1635 fps=325 q=30.0 size= 40448kB time=00:00:54.25 bitrate=6107.4kbits/s dup=1 drop=0 speed=10.8x [mpeg2_qsv @ 0x5608f04fbb40] A decode call did not consume any data: expect more data at input (-10) Last message repeated 1 times frame= 1799 fps=325 q=29.0 size= 44544kB time=00:00:59.72 bitrate=6109.6kbits/s dup=1 drop=0 speed=10.8x frame= 1960 fps=325 q=31.0 size= 48640kB time=00:01:05.09 bitrate=6120.9kbits/s dup=1 drop=0 speed=10.8x frame= 2123 fps=325 q=32.0 size= 52736kB time=00:01:10.53 bitrate=6124.6kbits/s dup=1 drop=0 speed=10.8x frame= 2289 fps=325 q=26.0 size= 56320kB time=00:01:16.07 bitrate=6064.6kbits/s dup=1 drop=0 speed=10.8x No more output streams to write to, finishing. frame= 2400 fps=324 q=22.0 Lsize= 59139kB time=00:01:20.01 bitrate=6054.8kbits/s dup=1 drop=0 speed=10.8x video:59139kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000% Input file #0 (input/1920x1080i_29.97_20mb_mpeg2_high.mpv): Input stream #0:0 (video): 2404 packets read (200287518 bytes); 2400 frames decoded; Total: 2404 packets (200287518 bytes) demuxed Output file #0 (output/0024_HD17i7_1.0.h264): Output stream #0:0 (video): 2400 frames encoded; 2400 packets muxed (60558220 bytes); Total: 2400 packets (60558220 bytes) muxed [AVIOContext @ 0x5608f04fcf40] Statistics: 0 seeks, 232 writeouts [AVIOContext @ 0x5608f04fd780] Statistics: 200310784 bytes read, 0 seeks