Opened 5 months ago

Closed 4 months ago

Last modified 4 months ago

#7588 closed defect (invalid)

Incorrect VMAF score computation

Reported by: Ilya87 Owned by:
Priority: normal Component: avfilter
Version: git-master Keywords: libvmaf
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
How to reproduce:

ffmpeg -i main_x264_2pass_5000.mkv -i reference.mkv -lavfi libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log" -f null -
VMAF score: 60.514456
ffmpeg version 92603-g8440835dbe x86_64 AntergOS Linux
VMAF 1.3.9

reference video - x264 lossless high 4:4:4 profile 2560x1090 yuv420p
main_x264_2pass_5000.mkv - high profile yuv420p

ffmpeg2vmaf.py score:
python2 ffmpeg2vmaf.py 2560 1090 reference.mkv main_x264_2pass_5000.mkv --model /usr/share/model/vmaf_v0.6.1.pkl --ref-width 2560 --ref-height 1090 --dis-width 2560 --dis-height 1090
Aggregate (mean): VMAF_feature_adm2_score:0.958728, VMAF_feature_motion2_score:9.141027, VMAF_feature_vif_scale0_score:0.664811, VMAF_feature_vif_scale1_score:0.909114, VMAF_feature_vif_scale2_score:0.951961, VMAF_feature_vif_scale3_score:0.970957, VMAF_score:94.276637

Conclusion - ffmpeg often (316 times in 1191 video frames file) gives zero VMAF score (see it in attached file). Scores for other frames are also different than ffmpeg2vmaf computes.

Attachments (2)

vmaf.log (183.6 KB) - added by Ilya87 5 months ago.
vmaf_ffmpeg2vmaf.log (306.8 KB) - added by Ilya87 5 months ago.

Download all attachments as: .zip

Change History (14)

Changed 5 months ago by Ilya87

Changed 5 months ago by Ilya87

comment:1 Changed 5 months ago by cehoyos

  • Keywords libvmaf added; vmaf removed

Isn’t the used input file needed to reproduce?

comment:3 Changed 5 months ago by cehoyos

This issue is not reproducible with smaller files?

comment:5 Changed 5 months ago by Ilya87

If I remux Sintel_x264_2pass_5000_1.mkv using mkvmerge (Sintel_x264_2pass_5000_1_mkvmerge in folder above), ffmpeg shows right results, so the question is - it is a bug of
Haali Matroska Writer b0 (that is used by x264) or matroska demuxer in ffmpeg?
MKValidator shows 2 warnings for Haali Writer muxed file:
WRN801: The segment has no SeekHead? section
WRN800: The segment has Clusters but no Cues section (bad for seeking)
Can this be the reason for bad computation and is it possible (needed) to adopt ffmpeg demuxer for this case?

comment:6 Changed 5 months ago by Ilya87

Part 2 of this bug:
https://mega.nz/#F!61VEFQwB!wUEggqchsrYqDDgogCb2Ng

ffmpeg -i klk.mkv -i klk_reference.mkv -lavfi libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log" -f null -

VMAF score nan, 0 score for frames also appears. klk16.mkv - yuv420p10le, klk16_reference.mkv - yuv420p.

2 files are absolutely valid matroska files.

comment:7 follow-up: Changed 4 months ago by richardpl

  • Resolution set to invalid
  • Status changed from new to closed

But pts are far from perfect, they have gaps. If you want same results as vmaf use setpts=N to reset timestamps prior to using libvmaf filter.

comment:8 in reply to: ↑ 7 Changed 4 months ago by Ilya87

Replying to richardpl:

But pts are far from perfect, they have gaps. If you want same results as vmaf use setpts=N to reset timestamps prior to using libvmaf filter.

Provide right commandline please prior to close the issue. Thank you.

Because when I use -filter_complex "[0:v]setpts=N/((24000/1001)*TB)[v];[1:v]setpts=N/((24000/1001)*TB)[v];libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log"[v]"
I have the following message "Cannot find a matching stream for unlabeled input pad 0 on filter Parsed_libvmaf_2"

comment:9 Changed 4 months ago by Ilya87

  • Resolution invalid deleted
  • Status changed from closed to reopened

comment:10 Changed 4 months ago by richardpl

  • Resolution set to wontfix
  • Status changed from reopened to closed

comment:11 Changed 4 months ago by richardpl

[0:v]setpts=N/((24000/1001)*TB)[first];[1:v]setpts=N/((24000/1001)*TB)[second];[first][second]libvmaf="model_path=/usr/share/model/vmaf_v0.6.1.pkl:pool=mean:log_path=vmaf.log"
Last edited 4 months ago by richardpl (previous) (diff)

comment:12 Changed 4 months ago by cehoyos

  • Resolution changed from wontfix to invalid
Note: See TracTickets for help on using tickets.