Incorrect VMAF score computation

[Ilya87]

Summary of the bug:
How to reproduce:

ffmpeg -i main_x264_2pass_5000.mkv -i reference.mkv -lavfi libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log" -f null -
VMAF score: 60.514456
ffmpeg version 92603-g8440835dbe x86_64 AntergOS Linux
VMAF 1.3.9

reference video - x264 lossless high 4:4:4 profile 2560x1090 yuv420p
main_x264_2pass_5000.mkv - high profile yuv420p

ffmpeg2vmaf.py score:
python2 ffmpeg2vmaf.py 2560 1090 reference.mkv main_x264_2pass_5000.mkv --model /usr/share/model/vmaf_v0.6.1.pkl --ref-width 2560 --ref-height 1090 --dis-width 2560 --dis-height 1090
Aggregate (mean): VMAF_feature_adm2_score:0.958728, VMAF_feature_motion2_score:9.141027, VMAF_feature_vif_scale0_score:0.664811, VMAF_feature_vif_scale1_score:0.909114, VMAF_feature_vif_scale2_score:0.951961, VMAF_feature_vif_scale3_score:0.970957, VMAF_score:94.276637

Conclusion - ffmpeg often (316 times in 1191 video frames file) gives zero VMAF score (see it in attached file). Scores for other frames are also different than ffmpeg2vmaf computes.

[Carl Eugen Hoyos]

Isn’t the used input file needed to reproduce?

[Ilya87]

Reference
​https://mega.nz/#F!23Y0UCzZ!xawr1sqKenlY2WwClls14w
Sintel1090.mkv
Encoded
​https://mega.nz/#F!Xm5QzC5D!KHy1uasIuN_y3iElS259DA!OqoUHKjS
Sintel_x264_2pass_5000.mkv

[Carl Eugen Hoyos]

This issue is not reproducible with smaller files?

[Ilya87]

Reproducible ​https://mega.nz/#F!H4dB0Q6b!ZKWRMQjFOnNL2WhAI8fNbA

[Ilya87]

If I remux Sintel_x264_2pass_5000_1.mkv using mkvmerge (Sintel_x264_2pass_5000_1_mkvmerge in folder above), ffmpeg shows right results, so the question is - it is a bug of
Haali Matroska Writer b0 (that is used by x264) or matroska demuxer in ffmpeg?
MKValidator shows 2 warnings for Haali Writer muxed file:
WRN801: The segment has no SeekHead section
WRN800: The segment has Clusters but no Cues section (bad for seeking)
Can this be the reason for bad computation and is it possible (needed) to adopt ffmpeg demuxer for this case?

[Ilya87]

Part 2 of this bug:
​https://mega.nz/#F!61VEFQwB!wUEggqchsrYqDDgogCb2Ng

ffmpeg -i klk.mkv -i klk_reference.mkv -lavfi libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log" -f null -

VMAF score nan, 0 score for frames also appears. klk16.mkv - yuv420p10le, klk16_reference.mkv - yuv420p.

2 files are absolutely valid matroska files.

[Elon Musk]

But pts are far from perfect, they have gaps. If you want same results as vmaf use setpts=N to reset timestamps prior to using libvmaf filter.

[Ilya87]

Replying to richardpl:

But pts are far from perfect, they have gaps. If you want same results as vmaf use setpts=N to reset timestamps prior to using libvmaf filter.

Provide right commandline please prior to close the issue. Thank you.

Because when I use -filter_complex "[0:v]setpts=N/((24000/1001)*TB)[v];[1:v]setpts=N/((24000/1001)*TB)[v];libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log"[v]"
I have the following message "Cannot find a matching stream for unlabeled input pad 0 on filter Parsed_libvmaf_2"

[Elon Musk]

[0:v]setpts=N/((24000/1001)*TB)[first];[1:v]setpts=N/((24000/1001)*TB)[second];[first][second]libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log