Opened 21 months ago

Last modified 12 months ago

#7095 open defect

New framesync API makes psnr generate wrong results

Reported by: rbultje Owned by:
Priority: normal Component: avfilter
Version: git-master Keywords: framesync psnr
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug: when using the PSNR lavfilter to measure PSNR, the output is not consistent between releases. In some cases, the result is fundamentally broken.

How to reproduce:

% ffmpeg -i x.ivf -v error -vsync 0 -vframes 3 /tmp/compressed.yuv
% ffmpeg -i ~/Movies/nflx360p/Aerial.ffv1.mkv -v error -vsync 0 -vframes 3 /tmp/aerial.yuv
% ffmpeg -video_size 640x360 -i /tmp/compressed.yuv -video_size 640x360 -i /tmp/aerial.yuv -lavfi '[0:v][1:v]psnr' -v info -nostats -f null -
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.38)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-jack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[rawvideo @ 0x7f8205800000] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from '/tmp/compressed.yuv':
  Duration: 00:00:00.12, start: 0.000000, bitrate: 69120 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x360, 69120 kb/s, 25 tbr, 25 tbn, 25 tbc
[rawvideo @ 0x7f8205811c00] Estimating duration from bitrate, this may be inaccurate
Input #1, rawvideo, from '/tmp/aerial.yuv':
  Duration: 00:00:00.12, start: 0.000000, bitrate: 69120 kb/s
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x360, 69120 kb/s, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 (rawvideo) -> psnr:main
  Stream #1:0 (rawvideo) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 640x360, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 wrapped_avframe
frame=    3 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.12 bitrate=N/A speed=70.2x    
video:2kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x7f8204f03980] PSNR y:40.022550 u:42.672859 v:44.018973 average:40.865346 min:40.316870 max:41.634475

I'm assuming this is the correct PSNR value. Now let's do the files directly:

% ffmpeg -i x.ivf -i ~/Movies/nflx360p/Aerial.ffv1.mkv -lavfi '[0:v][1:v]psnr' -v info -nostats -f null -
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.38)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-jack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, ivf, from 'x.ivf':
  Duration: 00:00:00.17, start: 0.000000, bitrate: 3007 kb/s
    Stream #0:0: Video: vp9 (Profile 0) (VP90 / 0x30395056), yuv420p(tv), 640x360, 30 tbr, 30 tbn, 30 tbc
Input #1, matroska,webm, from '/Users/ronaldbultje/Movies/nflx360p/Aerial.ffv1.mkv':
  Metadata:
    ENCODER         : Lavf57.23.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 40448 kb/s
    Stream #1:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 640x360, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc57.22.100 ffv1
      DURATION        : 00:00:20.000000000
Stream mapping:
  Stream #0:0 (vp9) -> psnr:main
  Stream #1:0 (ffv1) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 640x360, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 wrapped_avframe
frame=  600 fps=297 q=-0.0 Lsize=N/A time=00:00:20.00 bitrate=N/A speed=9.89x    
video:309kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x7ff4bbd03220] PSNR y:15.673549 u:28.453958 v:31.570819 average:17.350123 min:14.220084 max:41.634475

That PSNR value is not correct at all. -vframes 3 changes the result, but does not resolve the issue entirely:

% ffmpeg -i x.ivf -i ~/Movies/nflx360p/Aerial.ffv1.mkv -lavfi '[0:v][1:v]psnr' -v info -nostats -f null -vframes 3 -
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.38)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-jack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, ivf, from 'x.ivf':
  Duration: 00:00:00.17, start: 0.000000, bitrate: 3007 kb/s
    Stream #0:0: Video: vp9 (Profile 0) (VP90 / 0x30395056), yuv420p(tv), 640x360, 30 tbr, 30 tbn, 30 tbc
Input #1, matroska,webm, from '/Users/ronaldbultje/Movies/nflx360p/Aerial.ffv1.mkv':
  Metadata:
    ENCODER         : Lavf57.23.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 40448 kb/s
    Stream #1:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 640x360, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc57.22.100 ffv1
      DURATION        : 00:00:20.000000000
Stream mapping:
  Stream #0:0 (vp9) -> psnr:main
  Stream #1:0 (ffv1) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 640x360, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 wrapped_avframe
frame=    3 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.10 bitrate=N/A speed=2.45x    
video:2kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x7ff69c100aa0] PSNR y:39.805490 u:42.630894 v:43.989575 average:40.682090 min:40.079826 max:41.634475

This is reproduceable with git master as well as 3.4.2. It does not occur with 3.2.2. Using the setpts filter with 'N*TB' as argument resolves a small subset of the issues but not completely.

Attachments (2)

x.ivf (61.2 KB) - added by rbultje 21 months ago.
Compressed file (3 frames)
aerial.y4m (1012.6 KB) - added by rbultje 21 months ago.
Reference file (3 frames)

Download all attachments as: .zip

Change History (8)

Changed 21 months ago by rbultje

Compressed file (3 frames)

Changed 21 months ago by rbultje

Reference file (3 frames)

comment:1 Changed 21 months ago by jamrial

  • Component changed from undetermined to avfilter
  • Keywords framesync added
  • Status changed from new to open
  • Version changed from unspecified to git-master

comment:2 Changed 21 months ago by jamrial

  • Summary changed from -lavfi psnr gives wrong result to New framesync API makes psnr generate wrong results

comment:3 Changed 21 months ago by Cigaes

The videos have different lengths, the standard behaviour is to extend the shortest to the length of the longest, hence the result you are observing. -vframes will not help since it acts after the filters. The shortest option to psnr can help, though.

comment:4 Changed 21 months ago by cehoyos

  • Keywords psnr regression added

comment:5 Changed 21 months ago by Cigaes

  • Keywords regression removed

comment:6 Changed 12 months ago by richardpl

So what should be done? Change default for shortest to 1? It makes sense but I'm afraid that there is limitation to do it nice way.

Note: See TracTickets for help on using tickets.