Opened 2 years ago

Last modified 9 months ago

#8820 open defect

skip_frame nokey skips everything but IDR frames (H.264)

Reported by: skippableframe Owned by:
Priority: minor Component: avcodec
Version: git-master Keywords: h264
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

When using ffmpeg and ffprobe with -skip_frame nokey on H.264 files, only IDR frames are retained. Other I frames are skipped.

This is doubly confusing because ffprobe -show_frames marks these (non-IDR) I frames with key_frame=1.

Some retail Blu-ray disks are encoded with non-IDR I frames throughout. Some encoders seem to emit no IDR frames when configured for open GOP.

This ticket for H.265 is similar.
https://trac.ffmpeg.org/ticket/7227

ffmpeg 4.3.1 built on FreeBSD, macOS.

Run ffprobe with -skip_frames nokey and most key frames are skipped.

% ffprobe -v error -skip_frame nokey -show_frames test.mkv
[h264 @ 0x80668b400] illegal short term buffer state detected
[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=10000
pkt_pts_time=10.000000
pkt_dts=9920
pkt_dts_time=9.920000
best_effort_timestamp=10000
best_effort_timestamp_time=10.000000
pkt_duration=40
pkt_duration_time=0.040000
pkt_pos=3318507
pkt_size=43205
width=720
height=576
pix_fmt=yuv420p
sample_aspect_ratio=157:144
pict_type=I
coded_picture_number=1
display_picture_number=0
interlaced_frame=1
top_field_first=1
repeat_pict=0
color_range=tv
color_space=bt470bg
color_primaries=bt470bg
color_transfer=bt470bg
chroma_location=left
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=0
pkt_pts_time=0.000000
pkt_dts=N/A
pkt_dts_time=N/A
best_effort_timestamp=0
best_effort_timestamp_time=0.000000
pkt_duration=40
pkt_duration_time=0.040000
pkt_pos=635
pkt_size=2862
width=720
height=576
pix_fmt=yuv420p
sample_aspect_ratio=157:144
pict_type=I
coded_picture_number=0
display_picture_number=0
interlaced_frame=1
top_field_first=1
repeat_pict=0
color_range=tv
color_space=bt470bg
color_primaries=bt470bg
color_transfer=bt470bg
chroma_location=left
[/FRAME]

That's it, just the two frames.

Run ffprobe without -skip_frames nokey and all I frames are recognized.

% ffprobe -v error -show_frames test.mkv | grep key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1

Attachments (8)

ffprobe-20200726-203017-nokey.log (311.6 KB ) - added by skippableframe 2 years ago.
ffprobe report with skip frames nokey
ffprobe-20200726-203054.log (364.3 KB ) - added by skippableframe 2 years ago.
ffprobe report without skip frames
unknown.opengop.mp4 (2.1 MB ) - added by skippableframe 2 years ago.
Minimal sample showing key_frame differences
ffprobe-show_frames.log (663.6 KB ) - added by skippableframe 2 years ago.
report output from ffprobe -show_frames, unknown.opengop.mp4
ffprobe-show_frames_nokey.log (668.9 KB ) - added by skippableframe 2 years ago.
report output from ffprobe -show_frames with nokey, unknown.opengop.mp4
ffprobe-show_frames-out.log (1.9 MB ) - added by skippableframe 2 years ago.
stdout from ffprobe -show_frames, unknown.opengop.mp4
ffprobe-show_frames_nokey-out.log (2.2 KB ) - added by skippableframe 2 years ago.
stdout from ffprobe -show_frames with nokey, unknown.opengop.mp4
ffmpeg-nokey.log (684.3 KB ) - added by skippableframe 2 years ago.
report output from ffmpeg with nokey, unknown.opengop.mp4

Change History (19)

by skippableframe, 2 years ago

ffprobe report with skip frames nokey

by skippableframe, 2 years ago

Attachment: ffprobe-20200726-203054.log added

ffprobe report without skip frames

comment:1 by skippableframe, 2 years ago

comment:2 by skippableframe, 2 years ago

I didn't specify a version because it isn't git-master, and 4.3.1 isn't in the pull-down menu.

I don't see any recent commits that would appear to affect this, but I'm in over my head at that point. :-)

comment:3 by Carl Eugen Hoyos, 2 years ago

Keywords: h264 added; avc removed
Version: unspecifiedgit-master

On this bug tracker, only current FFmpeg git head is supported.

Behaviour changed in 7eae8cd87054713cbef51b22b93f3a0eb5036c90, related to ticket #3475.

comment:4 by skippableframe, 2 years ago

Version: git-masterunspecified

Thank you. I'll build from git-master.

What is the expected behavior, please - should -skip_frame nokey skip everything but IDR frames?

It appears that ffprobe -show_frames identifies all I or IDR frames as key_frame=1.
I assumed that -skip_frame nokey would skip those same frames.
It appears that -skip_frame nokey skips all but IDR.

I expected to get all I frames, AND the difference in behavior surprised me.

Last edited 2 years ago by skippableframe (previous) (diff)

comment:5 by Carl Eugen Hoyos, 2 years ago

Version: unspecifiedgit-master

comment:6 by skippableframe, 2 years ago

Built from git-master.

I will attach another sample that was created with current x264. This behavior can be replicated easily with --open-gop. This issue was identified because some commercial Blu-ray releases are encoded in this way.

% x264 --vf=resize:360,150 --open-gop -o unknown.opengop.mp4 unknown-3.mp4

This file has 37 key frames, per ffprobe:

% ffprobe -show_frames -i unknown.opengop.mp4 | grep -c key_frame=1
ffprobe version N-98575-g16c2ed4362 Copyright (c) 2007-2020 the FFmpeg developers
  built with FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
  configuration: --cc=cc --cxx=c++ --pkgconfigdir=/usr/local/libdata/pkgconfig --prefix=/usr/local --datadir=/usr/local/share/ffmpeg --mandir=/usr/local/man --enable-shared --enable-version3 --enable-nonfree --enable-gpl --enable-pic --enable-runtime-cpudetect --enable-fontconfig --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --enable-libfdk-aac --enable-libfreetype --enable-libopus --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxml2
  libavutil      56. 57.100 / 56. 57.100
  libavcodec     58. 97.102 / 58. 97.102
  libavformat    58. 49.100 / 58. 49.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'unknown.opengop.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41isom
    creation_time   : 2020-07-28T05:57:08.000000Z
  Duration: 00:02:30.36, start: 0.000000, bitrate: 114 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv), 360x150 [SAR 1:1 DAR 12:5], 112 kb/s, 23.94 fps, 23.98 tbr, 11988 tbn, 23976 tbc (default)
    Metadata:
      creation_time   : 2020-07-28T05:57:08.000000Z
      handler_name    : L-SMASH Video Media Handler
      encoder         : AVC Coding
37

(And it has 45 pict_type=I frames.)

But when used with -skip_frame nokey, only 4 key frames remain:

% ffprobe -v0 -skip_frame nokey -show_frames -i unknown.opengop.mp4 | grep -c key_frame=1
4

When using ffmpeg with -skip_frame nokey to extract keyframes, only 3 files are produced:

% ffmpeg -v0 -skip_frame nokey -i unknown.opengop.mp4 -vsync 0 img-%06d.jpg; ls *jpg | wc -l
       3

When using ffmpeg with a video filter to extract I frames, 45 are produced (as expected):

% ffmpeg -v0 -i unknown.opengop.mp4 -vf "select=eq(pict_type\,PICT_TYPE_I)" -vsync 0 img-%06d.jpg ; ls *jpg | wc -l
      45

I understand that "What's a key frame, anyway?" isn't exactly defined.

I think that ffprobe -show_frames is producing useful output. Treating I frames (and other recovery frames?) as key frames is useful and I think is the expected behavior. I'd like -skip_frame nokey to behave the same way.

It would be understandable if -skip_frame nokey only used IDR frames, but less useful. It doesn't match the ffprobe -show_frames output, and it doesn't match the behavior of ffmpeg with H.265.

by skippableframe, 2 years ago

Attachment: unknown.opengop.mp4 added

Minimal sample showing key_frame differences

by skippableframe, 2 years ago

Attachment: ffprobe-show_frames.log added

report output from ffprobe -show_frames, unknown.opengop.mp4

by skippableframe, 2 years ago

report output from ffprobe -show_frames with nokey, unknown.opengop.mp4

by skippableframe, 2 years ago

Attachment: ffprobe-show_frames-out.log added

stdout from ffprobe -show_frames, unknown.opengop.mp4

by skippableframe, 2 years ago

stdout from ffprobe -show_frames with nokey, unknown.opengop.mp4

by skippableframe, 2 years ago

Attachment: ffmpeg-nokey.log added

report output from ffmpeg with nokey, unknown.opengop.mp4

comment:7 by Balling, 22 months ago

Status: newopen

How did this happen that issue for HEVC was fixed and this was not?

comment:8 by Elon Musk, 22 months ago

Because you sent patch.

comment:9 by skippableframe, 22 months ago

I notice that the first sample file link has expired. I would be happy to provide another if that's helpful.

comment:10 by skippableframe, 22 months ago

Last edited 22 months ago by skippableframe (previous) (diff)

comment:11 by Balling, 9 months ago

10bc4c3a7df7bb26303067b97311b7eeedfd453e is like that, no? Also please reattach the first sample. As for open gop in hevc may be fixed after this (AVC may need the same is what I mean): https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220218232001.345826-5-u@pkh.me/

Some retail Blu-ray disks are encoded with non-IDR I frames throughout

First of all this is not non-IDR, because no I frame cannot be non-keyframe in this case. Second of all, is it even allowed by Blu-ray spec? Third of all give an example and a sample.

When using ffmpeg and ffprobe with -skip_frame nokey on H.264 files, only IDR frames are retained.

This is correct behaviour. Not all I-frame are keyframes. This is not like with CLA or BLA, YOU CANNOT decode cleanly from such an I frame.

Last edited 9 months ago by Balling (previous) (diff)
Note: See TracTickets for help on using tickets.