skip_frame nokey skips everything but IDR frames (H.264)

[skippableframe]

When using ffmpeg and ffprobe with -skip_frame nokey on H.264 files, only IDR frames are retained. Other I frames are skipped.

This is doubly confusing because ffprobe -show_frames marks these (non-IDR) I frames with key_frame=1.

Some retail Blu-ray disks are encoded with non-IDR I frames throughout. Some encoders seem to emit no IDR frames when configured for open GOP.

This ticket for H.265 is similar.
​https://trac.ffmpeg.org/ticket/7227

ffmpeg 4.3.1 built on FreeBSD, macOS.

Run ffprobe with -skip_frames nokey and most key frames are skipped.

% ffprobe -v error -skip_frame nokey -show_frames test.mkv
[h264 @ 0x80668b400] illegal short term buffer state detected
[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=10000
pkt_pts_time=10.000000
pkt_dts=9920
pkt_dts_time=9.920000
best_effort_timestamp=10000
best_effort_timestamp_time=10.000000
pkt_duration=40
pkt_duration_time=0.040000
pkt_pos=3318507
pkt_size=43205
width=720
height=576
pix_fmt=yuv420p
sample_aspect_ratio=157:144
pict_type=I
coded_picture_number=1
display_picture_number=0
interlaced_frame=1
top_field_first=1
repeat_pict=0
color_range=tv
color_space=bt470bg
color_primaries=bt470bg
color_transfer=bt470bg
chroma_location=left
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=0
pkt_pts_time=0.000000
pkt_dts=N/A
pkt_dts_time=N/A
best_effort_timestamp=0
best_effort_timestamp_time=0.000000
pkt_duration=40
pkt_duration_time=0.040000
pkt_pos=635
pkt_size=2862
width=720
height=576
pix_fmt=yuv420p
sample_aspect_ratio=157:144
pict_type=I
coded_picture_number=0
display_picture_number=0
interlaced_frame=1
top_field_first=1
repeat_pict=0
color_range=tv
color_space=bt470bg
color_primaries=bt470bg
color_transfer=bt470bg
chroma_location=left
[/FRAME]

That's it, just the two frames.

Run ffprobe without -skip_frames nokey and all I frames are recognized.

% ffprobe -v error -show_frames test.mkv | grep key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1

[skippableframe]

ffprobe report with skip frames nokey

[skippableframe]

ffprobe report without skip frames

[skippableframe]

Sample file ​https://ufile.io/go3k570z.

[skippableframe]

I didn't specify a version because it isn't git-master, and 4.3.1 isn't in the pull-down menu.

I don't see any recent commits that would appear to affect this, but I'm in over my head at that point. :-)

[Carl Eugen Hoyos]

On this bug tracker, only current FFmpeg git head is supported.

Behaviour changed in 7eae8cd87054713cbef51b22b93f3a0eb5036c90, related to ticket #3475.

[skippableframe]

Thank you. I'll build from git-master.

What is the expected behavior, please - should -skip_frame nokey skip everything but IDR frames?

It appears that ffprobe -show_frames identifies all I or IDR frames as key_frame=1.
I assumed that -skip_frame nokey would skip those same frames.
It appears that -skip_frame nokey skips all but IDR.

I expected to get all I frames, AND the difference in behavior surprised me.

[skippableframe]

Built from git-master.

I will attach another sample that was created with current x264. This behavior can be replicated easily with --open-gop. This issue was identified because some commercial Blu-ray releases are encoded in this way.

% x264 --vf=resize:360,150 --open-gop -o unknown.opengop.mp4 unknown-3.mp4

This file has 37 key frames, per ffprobe:

% ffprobe -show_frames -i unknown.opengop.mp4 | grep -c key_frame=1
ffprobe version N-98575-g16c2ed4362 Copyright (c) 2007-2020 the FFmpeg developers
  built with FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
  configuration: --cc=cc --cxx=c++ --pkgconfigdir=/usr/local/libdata/pkgconfig --prefix=/usr/local --datadir=/usr/local/share/ffmpeg --mandir=/usr/local/man --enable-shared --enable-version3 --enable-nonfree --enable-gpl --enable-pic --enable-runtime-cpudetect --enable-fontconfig --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --enable-libfdk-aac --enable-libfreetype --enable-libopus --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxml2
  libavutil      56. 57.100 / 56. 57.100
  libavcodec     58. 97.102 / 58. 97.102
  libavformat    58. 49.100 / 58. 49.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'unknown.opengop.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41isom
    creation_time   : 2020-07-28T05:57:08.000000Z
  Duration: 00:02:30.36, start: 0.000000, bitrate: 114 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv), 360x150 [SAR 1:1 DAR 12:5], 112 kb/s, 23.94 fps, 23.98 tbr, 11988 tbn, 23976 tbc (default)
    Metadata:
      creation_time   : 2020-07-28T05:57:08.000000Z
      handler_name    : L-SMASH Video Media Handler
      encoder         : AVC Coding
37

(And it has 45 pict_type=I frames.)

But when used with -skip_frame nokey, only 4 key frames remain:

% ffprobe -v0 -skip_frame nokey -show_frames -i unknown.opengop.mp4 | grep -c key_frame=1
4

When using ffmpeg with -skip_frame nokey to extract keyframes, only 3 files are produced:

% ffmpeg -v0 -skip_frame nokey -i unknown.opengop.mp4 -vsync 0 img-%06d.jpg; ls *jpg | wc -l
       3

When using ffmpeg with a video filter to extract I frames, 45 are produced (as expected):

% ffmpeg -v0 -i unknown.opengop.mp4 -vf "select=eq(pict_type\,PICT_TYPE_I)" -vsync 0 img-%06d.jpg ; ls *jpg | wc -l
      45

I understand that "What's a key frame, anyway?" isn't exactly defined.

I think that ffprobe -show_frames is producing useful output. Treating I frames (and other recovery frames?) as key frames is useful and I think is the expected behavior. I'd like -skip_frame nokey to behave the same way.

It would be understandable if -skip_frame nokey only used IDR frames, but less useful. It doesn't match the ffprobe -show_frames output, and it doesn't match the behavior of ffmpeg with H.265.

[skippableframe]

Minimal sample showing key_frame differences

[skippableframe]

report output from ffprobe -show_frames, unknown.opengop.mp4

[skippableframe]

report output from ffprobe -show_frames with nokey, unknown.opengop.mp4

[skippableframe]

stdout from ffprobe -show_frames, unknown.opengop.mp4

[skippableframe]

stdout from ffprobe -show_frames with nokey, unknown.opengop.mp4

[skippableframe]

report output from ffmpeg with nokey, unknown.opengop.mp4

[Balling]

How did this happen that issue for HEVC was fixed and this was not?

[Elon Musk]

Because you sent patch.

[skippableframe]

I notice that the first sample file link has expired. I would be happy to provide another if that's helpful.

[Balling]

10bc4c3a7df7bb26303067b97311b7eeedfd453e is like that, no? Also please reattach the first sample. As for open gop in hevc may be fixed after this (AVC may need the same is what I mean): ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220218232001.345826-5-u@pkh.me/

Some retail Blu-ray disks are encoded with non-IDR I frames throughout

First of all this is not non-IDR, because no I frame cannot be non-keyframe in this case. Second of all, is it even allowed by Blu-ray spec? Third of all give an example and a sample.

When using ffmpeg and ffprobe with -skip_frame nokey on H.264 files, only IDR frames are retained.

This is correct behaviour. Not all I-frame are keyframes. This is not like with CLA or BLA, YOU CANNOT decode cleanly from such an I frame.

[Balling]

Because ffprobe -flags2 +showall -skip_frame nokey -show_frames -i unknown.opengop.mp4

prints

illegal short term buffer state detected

I decided to reencode the file with new x264 encoder. Neither mp4 nor x264 outputs do not help. Nor does x264.exe --vf=resize:1920,1080 --open-gop -o unknown.opengopnew.mp4 unknown.opengopold.mp4