Opened 18 months ago

Last modified 11 months ago

#8820 open defect

skip_frame nokey skips everything but IDR frames (H.264)

Reported by: skippableframe Owned by:
Priority: minor Component: avcodec
Version: git-master Keywords: h264
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

When using ffmpeg and ffprobe with -skip_frame nokey on H.264 files, only IDR frames are retained. Other I frames are skipped.

This is doubly confusing because ffprobe -show_frames marks these (non-IDR) I frames with key_frame=1.

Some retail Blu-ray disks are encoded with non-IDR I frames throughout. Some encoders seem to emit no IDR frames when configured for open GOP.

This ticket for H.265 is similar.
https://trac.ffmpeg.org/ticket/7227

ffmpeg 4.3.1 built on FreeBSD, macOS.

Run ffprobe with -skip_frames nokey and most key frames are skipped.

% ffprobe -v error -skip_frame nokey -show_frames test.mkv
[h264 @ 0x80668b400] illegal short term buffer state detected
[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=10000
pkt_pts_time=10.000000
pkt_dts=9920
pkt_dts_time=9.920000
best_effort_timestamp=10000
best_effort_timestamp_time=10.000000
pkt_duration=40
pkt_duration_time=0.040000
pkt_pos=3318507
pkt_size=43205
width=720
height=576
pix_fmt=yuv420p
sample_aspect_ratio=157:144
pict_type=I
coded_picture_number=1
display_picture_number=0
interlaced_frame=1
top_field_first=1
repeat_pict=0
color_range=tv
color_space=bt470bg
color_primaries=bt470bg
color_transfer=bt470bg
chroma_location=left
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=0
pkt_pts_time=0.000000
pkt_dts=N/A
pkt_dts_time=N/A
best_effort_timestamp=0
best_effort_timestamp_time=0.000000
pkt_duration=40
pkt_duration_time=0.040000
pkt_pos=635
pkt_size=2862
width=720
height=576
pix_fmt=yuv420p
sample_aspect_ratio=157:144
pict_type=I
coded_picture_number=0
display_picture_number=0
interlaced_frame=1
top_field_first=1
repeat_pict=0
color_range=tv
color_space=bt470bg
color_primaries=bt470bg
color_transfer=bt470bg
chroma_location=left
[/FRAME]

That's it, just the two frames.

Run ffprobe without -skip_frames nokey and all I frames are recognized.

% ffprobe -v error -show_frames test.mkv | grep key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1
key_frame=1

Attachments (8)

ffprobe-20200726-203017-nokey.log (311.6 KB ) - added by skippableframe 18 months ago.
ffprobe report with skip frames nokey
ffprobe-20200726-203054.log (364.3 KB ) - added by skippableframe 18 months ago.
ffprobe report without skip frames
unknown.opengop.mp4 (2.1 MB ) - added by skippableframe 18 months ago.
Minimal sample showing key_frame differences
ffprobe-show_frames.log (663.6 KB ) - added by skippableframe 18 months ago.
report output from ffprobe -show_frames, unknown.opengop.mp4
ffprobe-show_frames_nokey.log (668.9 KB ) - added by skippableframe 18 months ago.
report output from ffprobe -show_frames with nokey, unknown.opengop.mp4
ffprobe-show_frames-out.log (1.9 MB ) - added by skippableframe 18 months ago.
stdout from ffprobe -show_frames, unknown.opengop.mp4
ffprobe-show_frames_nokey-out.log (2.2 KB ) - added by skippableframe 18 months ago.
stdout from ffprobe -show_frames with nokey, unknown.opengop.mp4
ffmpeg-nokey.log (684.3 KB ) - added by skippableframe 18 months ago.
report output from ffmpeg with nokey, unknown.opengop.mp4

Change History (18)

by skippableframe, 18 months ago

ffprobe report with skip frames nokey

by skippableframe, 18 months ago

Attachment: ffprobe-20200726-203054.log added

ffprobe report without skip frames

comment:1 by skippableframe, 18 months ago

comment:2 by skippableframe, 18 months ago

I didn't specify a version because it isn't git-master, and 4.3.1 isn't in the pull-down menu.

I don't see any recent commits that would appear to affect this, but I'm in over my head at that point. :-)

comment:3 by Carl Eugen Hoyos, 18 months ago

Keywords: h264 added; avc removed
Version: unspecifiedgit-master

On this bug tracker, only current FFmpeg git head is supported.

Behaviour changed in 7eae8cd87054713cbef51b22b93f3a0eb5036c90, related to ticket #3475.

comment:4 by skippableframe, 18 months ago

Version: git-masterunspecified

Thank you. I'll build from git-master.

What is the expected behavior, please - should -skip_frame nokey skip everything but IDR frames?

It appears that ffprobe -show_frames identifies all I or IDR frames as key_frame=1.
I assumed that -skip_frame nokey would skip those same frames.
It appears that -skip_frame nokey skips all but IDR.

I expected to get all I frames, AND the difference in behavior surprised me.

Last edited 18 months ago by skippableframe (previous) (diff)

comment:5 by Carl Eugen Hoyos, 18 months ago

Version: unspecifiedgit-master

comment:6 by skippableframe, 18 months ago

Built from git-master.

I will attach another sample that was created with current x264. This behavior can be replicated easily with --open-gop. This issue was identified because some commercial Blu-ray releases are encoded in this way.

% x264 --vf=resize:360,150 --open-gop -o unknown.opengop.mp4 unknown-3.mp4

This file has 37 key frames, per ffprobe:

% ffprobe -show_frames -i unknown.opengop.mp4 | grep -c key_frame=1
ffprobe version N-98575-g16c2ed4362 Copyright (c) 2007-2020 the FFmpeg developers
  built with FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
  configuration: --cc=cc --cxx=c++ --pkgconfigdir=/usr/local/libdata/pkgconfig --prefix=/usr/local --datadir=/usr/local/share/ffmpeg --mandir=/usr/local/man --enable-shared --enable-version3 --enable-nonfree --enable-gpl --enable-pic --enable-runtime-cpudetect --enable-fontconfig --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --enable-libfdk-aac --enable-libfreetype --enable-libopus --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxml2
  libavutil      56. 57.100 / 56. 57.100
  libavcodec     58. 97.102 / 58. 97.102
  libavformat    58. 49.100 / 58. 49.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'unknown.opengop.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41isom
    creation_time   : 2020-07-28T05:57:08.000000Z
  Duration: 00:02:30.36, start: 0.000000, bitrate: 114 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv), 360x150 [SAR 1:1 DAR 12:5], 112 kb/s, 23.94 fps, 23.98 tbr, 11988 tbn, 23976 tbc (default)
    Metadata:
      creation_time   : 2020-07-28T05:57:08.000000Z
      handler_name    : L-SMASH Video Media Handler
      encoder         : AVC Coding
37

(And it has 45 pict_type=I frames.)

But when used with -skip_frame nokey, only 4 key frames remain:

% ffprobe -v0 -skip_frame nokey -show_frames -i unknown.opengop.mp4 | grep -c key_frame=1
4

When using ffmpeg with -skip_frame nokey to extract keyframes, only 3 files are produced:

% ffmpeg -v0 -skip_frame nokey -i unknown.opengop.mp4 -vsync 0 img-%06d.jpg; ls *jpg | wc -l
       3

When using ffmpeg with a video filter to extract I frames, 45 are produced (as expected):

% ffmpeg -v0 -i unknown.opengop.mp4 -vf "select=eq(pict_type\,PICT_TYPE_I)" -vsync 0 img-%06d.jpg ; ls *jpg | wc -l
      45

I understand that "What's a key frame, anyway?" isn't exactly defined.

I think that ffprobe -show_frames is producing useful output. Treating I frames (and other recovery frames?) as key frames is useful and I think is the expected behavior. I'd like -skip_frame nokey to behave the same way.

It would be understandable if -skip_frame nokey only used IDR frames, but less useful. It doesn't match the ffprobe -show_frames output, and it doesn't match the behavior of ffmpeg with H.265.

by skippableframe, 18 months ago

Attachment: unknown.opengop.mp4 added

Minimal sample showing key_frame differences

by skippableframe, 18 months ago

Attachment: ffprobe-show_frames.log added

report output from ffprobe -show_frames, unknown.opengop.mp4

by skippableframe, 18 months ago

report output from ffprobe -show_frames with nokey, unknown.opengop.mp4

by skippableframe, 18 months ago

Attachment: ffprobe-show_frames-out.log added

stdout from ffprobe -show_frames, unknown.opengop.mp4

by skippableframe, 18 months ago

stdout from ffprobe -show_frames with nokey, unknown.opengop.mp4

by skippableframe, 18 months ago

Attachment: ffmpeg-nokey.log added

report output from ffmpeg with nokey, unknown.opengop.mp4

comment:7 by Balling, 11 months ago

Status: newopen

How did this happen that issue for HEVC was fixed and this was not?

comment:8 by Elon Musk, 11 months ago

Because you sent patch.

comment:9 by skippableframe, 11 months ago

I notice that the first sample file link has expired. I would be happy to provide another if that's helpful.

comment:10 by skippableframe, 11 months ago

Last edited 11 months ago by skippableframe (previous) (diff)
Note: See TracTickets for help on using tickets.