SSIM result will be lower than expected when use .mkv

[rxf91]

Hello.

Summary of the bug:

When using ssim filter, If I use .mkv file for main, other format (eg mp4, m2ts) for reference, SSIM will be lower than expected.

How to reproduce:

Results when using mkv container

ffmpeg -i origin.m2ts -c:v copy copied.mkv
ffmpeg -i copied.mkv -i origin.m2ts -an -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -
[Parsed_ssim_4 @ 00000148654eedc0] SSIM Y:0.922225 (11.091597) U:0.985489 (18.383000) V:0.989469 (19.775481) All:0.943976 (12.516286)

ffmpeg -i origin.m2ts -an -c:v libx264 -ssim 1 x264.mkv
SSIM Mean Y:0.9828054 (17.646db)
ffmpeg -i x264.mkv -i origin.m2ts -an -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -
[Parsed_ssim_4 @ 0000017f8d025740] SSIM Y:0.915586 (10.735878) U:0.982084 (17.467682) V:0.986489 (18.693172) All:0.938487 (12.110299)

Results when not using mkv container
Note the x264 ssim, It's same as mkv.

ffmpeg -i origin.m2ts -an -c:v copy copied.mp4
ffmpeg -i copied.mp4 -i origin.m2ts -an -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -
[Parsed_ssim_4 @ 000002bb515c7800] SSIM Y:1.000000 (inf) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (inf)

ffmpeg -i origin.m2ts -an -c:v libx264 -ssim 1 x264.mp4
[libx264 @ 000002c587d34600] SSIM Mean Y:0.9828054 (17.646db)
ffmpeg -i x264.mp4 -i origin.m2ts -an -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -
[Parsed_ssim_4 @ 0000019be0469840] SSIM Y:0.983138 (17.731021) U:0.992124 (21.037030) V:0.993634 (21.961672) All:0.986385 (18.659951)

full log

D:\bin\ffmpeg>ffmpeg -i x264.mkv -i origin.m2ts -an -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -
ffmpeg version N-104906-gdd94912479-ffmpeg-windows-build-helpers Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10.2.0 (GCC)
  configuration: --pkg-config=pkg-config --pkg-config-flags=--static --extra-version=ffmpeg-windows-build-helpers --enable-version3 --disable-debug --disable-w32threads --arch=x86_64 --target-os=mingw32 --cross-prefix=/ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32- --enable-libcaca --enable-gray --enable-libtesseract --enable-fontconfig --enable-gmp --enable-gnutls --enable-libass --enable-libbluray --enable-libbs2b --enable-libflite --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libvorbis --enable-libwebp --enable-libzimg --enable-libzvbi --enable-libmysofa --enable-libopenjpeg --enable-libopenh264 --enable-liblensfun --enable-libvmaf --enable-libsrt --enable-libxml2 --enable-opengl --enable-libdav1d --enable-cuda-llvm --enable-libsvthevc --enable-libsvtav1 --enable-libaom --enable-libvpx --enable-nvenc --enable-nvdec --extra-libs=-lharfbuzz --extra-libs=-lm --extra-libs=-lpthread --extra-cflags=-DLIBTWOLAME_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DCACA_STATIC --enable-amf --enable-libmfx --enable-gpl --enable-frei0r --enable-librubberband --enable-libvidstab --enable-libx264 --enable-libx265 --enable-avisynth --enable-libaribb24 --enable-libxvid --enable-libdavs2 --enable-libxavs2 --enable-libxavs --extra-cflags='-mtune=generic' --extra-cflags=-O3 --enable-static --disable-shared --prefix=/ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/x86_64-w64-mingw32 --enable-nonfree --enable-libfdk-aac --enable-decklink
  libavutil      57. 11.100 / 57. 11.100
  libavcodec     59. 14.100 / 59. 14.100
  libavformat    59. 10.100 / 59. 10.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8. 20.100 /  8. 20.100
  libswscale      6.  1.101 /  6.  1.101
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
Input #0, matroska,webm, from 'x264.mkv':
  Metadata:
    ENCODER         : Lavf59.10.100
  Duration: 00:01:31.09, start: 0.000000, bitrate: 4479 kb/s
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn
    Metadata:
      ENCODER         : Lavc59.14.100 libx264
      DURATION        : 00:01:31.091000000
Input #1, mpegts, from 'origin.m2ts':
  Duration: 00:01:31.10, start: 6120.000000, bitrate: 40073 kb/s
  Program 1
  Stream #1:0[0x1011]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn
  Stream #1:1[0x1100]: Audio: pcm_bluray (HDMV / 0x564D4448), 48000 Hz, stereo, s32 (24 bit), 2304 kb/s
Stream mapping:
  Stream #0:0 (h264) -> settb:default
  Stream #1:0 (h264) -> settb:default
  ssim:default -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf59.10.100
  Stream #0:0: Video: wrapped_avframe, yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn
    Metadata:
      encoder         : Lavc59.14.100 wrapped_avframe
[null @ 0000017f8c205c80] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2183 >= 2183
frame= 2185 fps=209 q=-0.0 Lsize=N/A time=00:01:31.09 bitrate=N/A speed=8.72x
video:956kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_ssim_4 @ 0000017f8d025740] SSIM Y:0.915586 (10.735878) U:0.982084 (17.467682) V:0.986489 (18.693172) All:0.938487 (12.110299)

[Balling]

You cannot -c copy stuff from m2ts or mp4 to mkv and expect perfect results, in most cases files become vfr (just -c copy back into mp4 and use mediainfo, you will see it is VFR now). Always use mp4.

[rxf91]

Thank you. I understand "-c: v copy" to mkv.
But what about encoding in x264?
Both mp4 and mkv have fixed frame rate of 23.976 (24000/1001) fps(by mediainfo), and the SSIM calculated by x264 is the same, but the SSIM calculated by ffmpeg (after encoding) is different.

[pdr0]

The timebase of mkv container is 1ms, so the timestamps will be different and you will get "jitter" , thus different results

The examples in the documentation say differently - but it should be settb=1/AVTB ; not settb=AVTB for PSNR, SSIM calculations . Works here for normalizing the timebase

[Balling]

Replying to pdr0:

The timebase of mkv container is 1ms, so the timestamps will be different and you will get "jitter" , thus different results

I am still not conviced this is not just a bug. 1 ms should be enough for 24/1.001.

[rxf91]

Replying to pdr0:

The timebase of mkv container is 1ms, so the timestamps will be different and you will get "jitter" , thus different results

The examples in the documentation say differently - but it should be settb=1/AVTB ; not settb=AVTB for PSNR, SSIM calculations . Works here for normalizing the timebase

Great thanks!
"settb=1/AVTB" option works fine for SSIM curation.

D:\bin\ffmpeg>ffmpeg -i x264.mkv -i origin.m2ts -an -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -
[Parsed_ssim_4 @ 00000203074e1140] SSIM Y:0.983138 (17.731021) U:0.992124 (21.037030) V:0.993634 (21.961672) All:0.986385 (18.659951)

I hope that this information will be added to the examples in the documentation.