Mismatched frames when using WebM containers and MP4 references

[Cosmin Stejerean]

WebM container forces timestamps to be rounded to the nearest ms to the forced timebase. This then results in mismatched frames if attempting to compare such a video stored in WebM against a reference in say MP4 for quality metric comparisons like SSIM or VMAF.

To illustrate the video I'm attaching the exact same video in both WebM and MP4, both encoded from a 30fps source, with no frame rate conversions. Attempting to compare the two videos should result in identical SSIM of 1.0, but due to frame mismatches the end result ends up being ~0.9

When looking at frame by frame data we can see that every 3rd frame is mismatched (in this example of 30fps video, the results would be slightly different at other frame rates, but anything that can't be expressed precisely in ms would show this issues).

n:1 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
n:2 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
n:3 Y:0.788922 U:0.951481 V:0.981816 All:0.848164 (8.186263)
n:4 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
n:5 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
n:6 Y:0.724944 U:0.878058 V:0.951394 All:0.788205 (6.740836)

essentially what happens with the timestamps (shown in seconds to illustrate the problem) is that in every group of 3 the first frame matches exactly, the second frame rounds down so is slightly earlier, and the 3rd frame rounds up.

0.0000 => 0.0000
0.0333 => 0.0330
0.0666 => 0.0670

Then when the filter is trying to frame sync the video the 3rd frame (from the group on the left with proper 30 fps timestamps) ends up being compared to the second one from the right because the 3rd frame is just slightly in the future. What would be desired is for framesync to instead find the nearest frame, so that 0.0666 is compared against 0.0670 instead of to 0.0330

That's a departure from how framesync works today. Fortunately in recent versions of ffmpeg a warning is shown in such cases, which at least highlights that this will be a problem.

[Parsed_ssim_0 @ 0x152209740] not matching timebases found between first input: 1/15360 and second input 1/1000, results may be incorrect!

Workarounds include decoding to YUV first (at which point frames become compared pairwise without regards to the odd timestamp rounding but this requires more disk space and IO), not using WebM in the first place to preserve accurate frame timestamps, or re-applying an fps filter to get both files onto the same timebase,

"[0] fps=fps=30 [dst]; [1] fps=fps=30 [ref]; [dst][ref] ssim

But in practice I've found this to be a very common mistake when looking at quality metrics of videos in WebM containers, so it would be great if there was an easier way to get the expected behavior directly from framesync.

[Balling]

That works as intended, webm is just mkv. So far everyone uses millisecond precision for mkv, webm actually mandated such precision.

MP4 uses PTS and DTS instead of ONLY durations in mkv and also uses microsecond precision.

[Cosmin Stejerean]

The millisecond precision in WebM is fine, it's mandated. The request is for better framesync options that can deal with such rounding and actually sync frames.

[James]

Addressed in 0c3e3fd1b478e14692f3b02bb8bf42262ee18af0. To get the new behavior you need to use the new ts_sync_mode framesync option, e.g. ssim=ts_sync_mode=nearest from the command line.