New framesync API makes psnr generate wrong results

[Ronald S. Bultje]

Summary of the bug: when using the PSNR lavfilter to measure PSNR, the output is not consistent between releases. In some cases, the result is fundamentally broken.

How to reproduce:

% ffmpeg -i x.ivf -v error -vsync 0 -vframes 3 /tmp/compressed.yuv
% ffmpeg -i ~/Movies/nflx360p/Aerial.ffv1.mkv -v error -vsync 0 -vframes 3 /tmp/aerial.yuv
% ffmpeg -video_size 640x360 -i /tmp/compressed.yuv -video_size 640x360 -i /tmp/aerial.yuv -lavfi '[0:v][1:v]psnr' -v info -nostats -f null -
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.38)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-jack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[rawvideo @ 0x7f8205800000] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from '/tmp/compressed.yuv':
  Duration: 00:00:00.12, start: 0.000000, bitrate: 69120 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x360, 69120 kb/s, 25 tbr, 25 tbn, 25 tbc
[rawvideo @ 0x7f8205811c00] Estimating duration from bitrate, this may be inaccurate
Input #1, rawvideo, from '/tmp/aerial.yuv':
  Duration: 00:00:00.12, start: 0.000000, bitrate: 69120 kb/s
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x360, 69120 kb/s, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 (rawvideo) -> psnr:main
  Stream #1:0 (rawvideo) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 640x360, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 wrapped_avframe
frame=    3 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.12 bitrate=N/A speed=70.2x    
video:2kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x7f8204f03980] PSNR y:40.022550 u:42.672859 v:44.018973 average:40.865346 min:40.316870 max:41.634475

I'm assuming this is the correct PSNR value. Now let's do the files directly:

% ffmpeg -i x.ivf -i ~/Movies/nflx360p/Aerial.ffv1.mkv -lavfi '[0:v][1:v]psnr' -v info -nostats -f null -
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.38)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-jack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, ivf, from 'x.ivf':
  Duration: 00:00:00.17, start: 0.000000, bitrate: 3007 kb/s
    Stream #0:0: Video: vp9 (Profile 0) (VP90 / 0x30395056), yuv420p(tv), 640x360, 30 tbr, 30 tbn, 30 tbc
Input #1, matroska,webm, from '/Users/ronaldbultje/Movies/nflx360p/Aerial.ffv1.mkv':
  Metadata:
    ENCODER         : Lavf57.23.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 40448 kb/s
    Stream #1:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 640x360, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc57.22.100 ffv1
      DURATION        : 00:00:20.000000000
Stream mapping:
  Stream #0:0 (vp9) -> psnr:main
  Stream #1:0 (ffv1) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 640x360, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 wrapped_avframe
frame=  600 fps=297 q=-0.0 Lsize=N/A time=00:00:20.00 bitrate=N/A speed=9.89x    
video:309kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x7ff4bbd03220] PSNR y:15.673549 u:28.453958 v:31.570819 average:17.350123 min:14.220084 max:41.634475

That PSNR value is not correct at all. -vframes 3 changes the result, but does not resolve the issue entirely:

% ffmpeg -i x.ivf -i ~/Movies/nflx360p/Aerial.ffv1.mkv -lavfi '[0:v][1:v]psnr' -v info -nostats -f null -vframes 3 -
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.38)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-jack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, ivf, from 'x.ivf':
  Duration: 00:00:00.17, start: 0.000000, bitrate: 3007 kb/s
    Stream #0:0: Video: vp9 (Profile 0) (VP90 / 0x30395056), yuv420p(tv), 640x360, 30 tbr, 30 tbn, 30 tbc
Input #1, matroska,webm, from '/Users/ronaldbultje/Movies/nflx360p/Aerial.ffv1.mkv':
  Metadata:
    ENCODER         : Lavf57.23.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 40448 kb/s
    Stream #1:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 640x360, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc57.22.100 ffv1
      DURATION        : 00:00:20.000000000
Stream mapping:
  Stream #0:0 (vp9) -> psnr:main
  Stream #1:0 (ffv1) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 640x360, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 wrapped_avframe
frame=    3 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.10 bitrate=N/A speed=2.45x    
video:2kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x7ff69c100aa0] PSNR y:39.805490 u:42.630894 v:43.989575 average:40.682090 min:40.079826 max:41.634475

This is reproduceable with git master as well as 3.4.2. It does not occur with 3.2.2. Using the setpts filter with 'N*TB' as argument resolves a small subset of the issues but not completely.

[Ronald S. Bultje]

Compressed file (3 frames)

[Ronald S. Bultje]

Reference file (3 frames)

[James]

Regression since 3bd11df459866d7303c1ad6779473528c2cfb76d

[Cigaes]

The videos have different lengths, the standard behaviour is to extend the shortest to the length of the longest, hence the result you are observing. -vframes will not help since it acts after the filters. The shortest option to psnr can help, though.

[Elon Musk]

So what should be done? Change default for shortest to 1? It makes sense but I'm afraid that there is limitation to do it nice way.