Opened 9 months ago

Last modified 6 months ago

#9523 open defect

segment_times splits at I-frames that are not keyframes

Reported by: TurboLed Owned by:
Priority: normal Component: avcodec
Version: git-master Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by TurboLed)

Summary of the bug:

When splitting segments on an MP4 video that contains I-frames that are not keyframes (not IDR frames), the splitting can occur on those I-frames and make the next segment unreadable.

How to reproduce:

% ffmpeg -i sample.mp4 -f segment -segment_times 3 -c copy output%03d.mp4
% ffprobe -i output001.mp4 -show_frames

[FRAME]
media_type=video
stream_index=0
key_frame=0
pts=150144
pts_time=3.128000
pkt_dts=151145
pkt_dts_time=3.148854
best_effort_timestamp=150144
best_effort_timestamp_time=3.128000
pkt_duration=1001
pkt_duration_time=0.020854
pkt_pos=48
pkt_size=599189
width=2720
height=1530
pix_fmt=yuv420p
sample_aspect_ratio=1:1
pict_type=I
coded_picture_number=1
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
color_range=tv
color_space=bt709
color_primaries=bt709
color_transfer=bt709
chroma_location=left
[/FRAME]

ffmpeg version 2021-11-18-git-85a6b7f7b7-essentials_build-www.gyan.dev

Sample: https://www.filehosting.org/file/details/969740/sample_48fps_iframes.mp4

Change History (15)

comment:1 by Balling, 9 months ago

You mean this: #8820?

comment:2 by TurboLed, 9 months ago

Description: modified (diff)

in reply to:  1 comment:3 by TurboLed, 9 months ago

Replying to Balling:

You mean this: #8820?

Not quite. #8820 is about ffprobe skipping I-frames when printing, which might be the correct behavior, depending on the definition of what is a keyframe.

My sample video has I-frames that are not IDR. When splitting at those I-frames, the second part of the split sometimes cannot be used properly (the first few frames are either not playing or broken). To fix this I have to manually specify a segment_times that is close to an actual keyframe (IDR), and ffprobe reports those same IDR as keyframe=1.

This is more visible on a longer video on which I trim several parts, when joining the trimmed parts together, the playback is broken at the merge points unless the segments are all split on an IDR frame.

Last edited 9 months ago by TurboLed (previous) (diff)

comment:4 by TurboLed, 9 months ago

Description: modified (diff)

Added sample

comment:5 by Balling, 9 months ago

I think this is #8820. And I quote from there: "This is doubly confusing because ffprobe -show_frames marks these (non-IDR) I frames with key_frame=1." This is a duplicate. Also please note that SEI recovery point are keyframes too.

See also https://github.com/FFmpeg/FFmpeg/blob/98aec8c1b864e44636c0fabdaff3494d2051fda3/libavcodec/h264_slice.c#L2254

Last edited 9 months ago by Balling (previous) (diff)

comment:6 by Balling, 9 months ago

Okay, first of all I cannot reproduce it. Second of all what I can reproduce is that it does not do 3 segments but only two here!

Also, on your file (remember to use https://temp-mail.org/ to download files from that site)

ffprobe.exe -skip_frame nokey -i sample_48fps_iframes.mp4 -show_frames

prints only two frames, which I suppose is BS:

media_type=video
stream_index=0
key_frame=1
pkt_pts=0
pkt_pts_time=0.000000
pkt_dts=120120
pkt_dts_time=2.502500

I will aslo point out that in my case the files work. Both of them. :)

P.S. Does not look like #8820 after all since grep key_frame=1 on bare ffprobe -show_frames does not count more frames.

Last edited 9 months ago by Balling (previous) (diff)

in reply to:  6 comment:7 by TurboLed, 9 months ago

Replying to Balling:

Okay, first of all I cannot reproduce it. Second of all what I can reproduce is that it does not do 3 segments but only two here!

The video is 5 seconds long, the command line (-segment_times 3) asks to split the video at 3 seconds to create 2 segments (first segment: 3 seconds, second segment: 2 seconds)

Also, on your file (remember to use https://temp-mail.org/ to download files from that site)

ffprobe.exe -skip_frame nokey -i sample_48fps_iframes.mp4 -show_frames

prints only two frames, which I suppose is BS:

media_type=video
stream_index=0
key_frame=1
pkt_pts=0
pkt_pts_time=0.000000
pkt_dts=120120
pkt_dts_time=2.502500

The video is only 5 seconds long, so it contains only 2 IDR keyframes, but contains also several intermediate I-frames every 0.625 seconds or so.

I will aslo point out that in my case the files work. Both of them. :)

Yes, this short sample will work okay in your player. But the problem is that the second segment starts with an I-frame that is not an IDR frame (i.e. the first frame of output001.mp4 is not reported as a key_frame=1 by ffprobe). The consequence is that for another, longer sample, that means that the first frame might be referring to a frame in the previous part of the video which doesn't exist anymore, because only IDR keyframes are complete and fully independent.

I can find and upload a much longer sample that illustrates the problem better, but we are talking gigs of video. Believe me this is an issue when splitting and joining segments, where some segments cannot start properly because of missing information from the previous frames. In my case the video can lose track and recover after several seconds (because each IDR is 2.5 seconds apart).

I still believe this is independent from #8820 because it is about the -f segment feature, not about what ffprobe reports.

For the sake of this issue, ffprobe reports the correct thing (i.e. there is no keyframe at the start of the second segment, so it should not have accepted to split there.) The segment feature (-f segment) should split only at IDR frames because that's what a real keyframe is, not I-frames.

Last edited 9 months ago by TurboLed (previous) (diff)

comment:8 by Balling, 9 months ago

Status: newopen

Yeah, #8820 is not related, I am not even sure it is still relevant (main sample file there is gone and gop sample just corrupts itself when you skip all non-keyframes, which is not surprising)

But I will open, indeed second segment starts with non-keyframe and thus even if starting I frame will be decoded, P frame after it will depend on frames before that I frame. https://en.wikipedia.org/wiki/Video_compression_picture_types#Predicted_(P)_frames/slices

Last edited 9 months ago by Balling (previous) (diff)

comment:9 by TurboLed, 9 months ago

Component: avfilteravcodec

Seems related to how AV_PKT_FLAG_KEY is set in libx264.c.

Changed component : to avcodec

Last edited 9 months ago by TurboLed (previous) (diff)

comment:10 by TurboLed, 9 months ago

Please disregard my previous comment, libx264 is not used with -c copy.

I have managed to fix locally by commenting this section in h264_parser.c:

    // heuristic to detect non marked keyframes
    if (p->ps.sps->ref_frame_count <= 1 && p->ps.pps->ref_count[0] <= 1 && s->pict_type == AV_PICTURE_TYPE_I)
        s->key_frame = 1;
Last edited 9 months ago by TurboLed (previous) (diff)

comment:11 by TurboLed, 9 months ago

See #3083

Last edited 9 months ago by TurboLed (previous) (diff)

comment:12 by Balling, 9 months ago

So s->key_frame = 1; there does not propogate to ffprobe and more keyframes are present in the stream?

in reply to:  12 comment:13 by TurboLed, 8 months ago

Replying to Balling:

So s->key_frame = 1; there does not propogate to ffprobe and more keyframes are present in the stream?

I believe s->key_frame = 1 (AVCodecParserContext::key_frame -> h264_parse.c) is not what ffprobe displays as key_frame but rather the frame decoding (AVFrame::key_frame -> h264dec.c) which is computed differently.

I also noted that -f segment is acting as an output filter; so that when re-encoding the video (i.e. without -c copy) the splitting occurs on the newly generated keyframes from the output. But when splitting a video, you usually don't want to re-encode.

comment:14 by TurboLed, 8 months ago

The sad part is that the above heuristic has been made for a specific stream coming from Panasonic Lumix cameras (#3083)

comment:15 by Balling, 6 months ago

I believe

Do you believe or is that really the case? Because that would be quite crazy design.

splitting occurs on the newly generated keyframes from the output

Well, of course, no workaround is needed then.

But when splitting a video, you usually don't want to re-encode.

Yeah, of course.

has been made for a specific stream coming from Panasonic Lumix cameras

Well, obviously you still need to check that the samples in -f segment are valid bitstream.

Note: See TracTickets for help on using tickets.