segment_times splits at I-frames that are not keyframes

[TurboLed]

Summary of the bug:

When splitting segments on an MP4 video that contains I-frames that are not keyframes (not IDR frames), the splitting can occur on those I-frames and make the next segment unreadable.

How to reproduce:

% ffmpeg -i sample.mp4 -f segment -segment_times 3 -c copy output%03d.mp4
% ffprobe -i output001.mp4 -show_frames

[FRAME]
media_type=video
stream_index=0
key_frame=0
pts=150144
pts_time=3.128000
pkt_dts=151145
pkt_dts_time=3.148854
best_effort_timestamp=150144
best_effort_timestamp_time=3.128000
pkt_duration=1001
pkt_duration_time=0.020854
pkt_pos=48
pkt_size=599189
width=2720
height=1530
pix_fmt=yuv420p
sample_aspect_ratio=1:1
pict_type=I
coded_picture_number=1
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
color_range=tv
color_space=bt709
color_primaries=bt709
color_transfer=bt709
chroma_location=left
[/FRAME]

ffmpeg version 2021-11-18-git-85a6b7f7b7-essentials_build-www.gyan.dev

Sample: ​https://www.filehosting.org/file/details/969740/sample_48fps_iframes.mp4

[Balling]

You mean this: #8820?

[TurboLed]

Replying to Balling:

You mean this: #8820?

Not quite. #8820 is about ffprobe skipping I-frames when printing, which might be the correct behavior, depending on the definition of what is a keyframe.

My sample video has I-frames that are not IDR. When splitting at those I-frames, the second part of the split sometimes cannot be used properly (the first few frames are either not playing or broken). To fix this I have to manually specify a segment_times that is close to an actual keyframe (IDR), and ffprobe reports those same IDR as keyframe=1.

This is more visible on a longer video on which I trim several parts, when joining the trimmed parts together, the playback is broken at the merge points unless the segments are all split on an IDR frame.

[TurboLed]

Added sample

[Balling]

I think this is #8820. And I quote from there: "This is doubly confusing because ffprobe -show_frames marks these (non-IDR) I frames with key_frame=1." This is a duplicate. Also please note that SEI recovery point are keyframes too.

See also ​https://github.com/FFmpeg/FFmpeg/blob/98aec8c1b864e44636c0fabdaff3494d2051fda3/libavcodec/h264_slice.c#L2254

[Balling]

Okay, first of all I cannot reproduce it. Second of all what I can reproduce is that it does not do 3 segments but only two here!

Also, on your file (remember to use ​https://temp-mail.org/ to download files from that site)

ffprobe.exe -skip_frame nokey -i sample_48fps_iframes.mp4 -show_frames

prints only two frames, which I suppose is BS:

media_type=video
stream_index=0
key_frame=1
pkt_pts=0
pkt_pts_time=0.000000
pkt_dts=120120
pkt_dts_time=2.502500

I will aslo point out that in my case the files work. Both of them. :)

P.S. Does not look like #8820 after all since grep key_frame=1 on bare ffprobe -show_frames does not count more frames.

[TurboLed]

Replying to Balling:

Okay, first of all I cannot reproduce it. Second of all what I can reproduce is that it does not do 3 segments but only two here!

The video is 5 seconds long, the command line (-segment_times 3) asks to split the video at 3 seconds to create 2 segments (first segment: 3 seconds, second segment: 2 seconds)

Also, on your file (remember to use ​https://temp-mail.org/ to download files from that site)

ffprobe.exe -skip_frame nokey -i sample_48fps_iframes.mp4 -show_frames

prints only two frames, which I suppose is BS:

media_type=video
stream_index=0
key_frame=1
pkt_pts=0
pkt_pts_time=0.000000
pkt_dts=120120
pkt_dts_time=2.502500

The video is only 5 seconds long, so it contains only 2 IDR keyframes, but contains also several intermediate I-frames every 0.625 seconds or so.

I will aslo point out that in my case the files work. Both of them. :)

Yes, this short sample will work okay in your player. But the problem is that the second segment starts with an I-frame that is not an IDR frame (i.e. the first frame of output001.mp4 is not reported as a key_frame=1 by ffprobe). The consequence is that for another, longer sample, that means that the first frame might be referring to a frame in the previous part of the video which doesn't exist anymore, because only IDR keyframes are complete and fully independent.

I can find and upload a much longer sample that illustrates the problem better, but we are talking gigs of video. Believe me this is an issue when splitting and joining segments, where some segments cannot start properly because of missing information from the previous frames. In my case the video can lose track and recover after several seconds (because each IDR is 2.5 seconds apart).

I still believe this is independent from #8820 because it is about the -f segment feature, not about what ffprobe reports.

For the sake of this issue, ffprobe reports the correct thing (i.e. there is no keyframe at the start of the second segment, so it should not have accepted to split there.) The segment feature (-f segment) should split only at IDR frames because that's what a real keyframe is, not I-frames.

[Balling]

Yeah, #8820 is not related, I am not even sure it is still relevant (main sample file there is gone and gop sample just corrupts itself when you skip all non-keyframes, which is not surprising)

But I will open, indeed second segment starts with non-keyframe and thus even if starting I frame will be decoded, P frame after it will depend on frames before that I frame. ​https://en.wikipedia.org/wiki/Video_compression_picture_types#Predicted_(P)_frames/slices

[TurboLed]

Seems related to how AV_PKT_FLAG_KEY is set in libx264.c.

Changed component : to avcodec

[TurboLed]

Please disregard my previous comment, libx264 is not used with -c copy.

I have managed to fix locally by commenting this section in h264_parser.c:

    // heuristic to detect non marked keyframes
    if (p->ps.sps->ref_frame_count <= 1 && p->ps.pps->ref_count[0] <= 1 && s->pict_type == AV_PICTURE_TYPE_I)
        s->key_frame = 1;

[TurboLed]

See #3083

[Balling]

So s->key_frame = 1; there does not propogate to ffprobe and more keyframes are present in the stream?

[TurboLed]

Replying to Balling:

So s->key_frame = 1; there does not propogate to ffprobe and more keyframes are present in the stream?

I believe s->key_frame = 1 (AVCodecParserContext::key_frame -> h264_parse.c) is not what ffprobe displays as key_frame but rather the frame decoding (AVFrame::key_frame -> h264dec.c) which is computed differently.

I also noted that -f segment is acting as an output filter; so that when re-encoding the video (i.e. without -c copy) the splitting occurs on the newly generated keyframes from the output. But when splitting a video, you usually don't want to re-encode.

[TurboLed]

The sad part is that the above heuristic has been made for a specific stream coming from Panasonic Lumix cameras (#3083)

[Balling]

I believe

Do you believe or is that really the case? Because that would be quite crazy design.

splitting occurs on the newly generated keyframes from the output

Well, of course, no workaround is needed then.

But when splitting a video, you usually don't want to re-encode.

Yeah, of course.

has been made for a specific stream coming from Panasonic Lumix cameras

Well, obviously you still need to check that the samples in -f segment are valid bitstream.

[TurboLed]

Replying to Balling:
To summarize in simpler terms, there was a fix in the past (​https://trac.ffmpeg.org/ticket/3083) that was made for a Panasonic Lumix video. This fix is detecting and adding new key frames which should not be part of my (and maybe other) videos. I removed this fix, recompiled ffmpeg and now I can split my video correctly. The false key frames are coming from that fix.

[Carl Eugen Hoyos]

Where can we find the sample?

[Balling]

Replying to Carl Eugen Hoyos:

Where can we find the sample?

Here you go ​https://0x0.st/ofKK.mp4

[TurboLed]

Some users from stackexchange reported similar issues:
​https://video.stackexchange.com/questions/25415/ffmpeg-ffprobe-marks-non-idr-keyframes-as-keyframes
​https://video.stackexchange.com/questions/28358/ffmpeg-identify-non-idr-frame-i-frame-as-keyframe

[Tsuyoshi]

I am also facing the same issue. Is this issue will not be fixed?
It seems that if stream does not have B-frame, all of I-frame will be marked as Keyframe in container.
Here is the procedure to reproduce this issue.

1. encode BBB file first 30sec.

ffmpeg -t 30 -i bbb.mp4 -preset ultrafast -profile:v baseline -sc_threshold 50 encoded.mp4

dump all I-frames.
frame, key_frame, pts, pict_type, coded_picture_number

frame,1,0.000000,I,0
frame,0,0.100000,I,3
frame,0,0.133333,I,4
frame,0,0.166667,I,5
frame,0,0.200000,I,6
frame,0,0.233333,I,7
frame,1,4.066667,I,122
frame,0,4.200000,I,126
frame,0,4.233333,I,127
frame,0,4.466667,I,134
frame,1,12.400000,I,372
frame,1,16.300000,I,489
frame,1,20.166667,I,605
frame,1,27.466667,I,824

This is stss Box.

[stss] Size=40 Version=0 Flags=0x000000 EntryCount=6 SampleNumber=[1, 123, 373, 490, 606, 825]

2. remux encoded.mp4 file

ffmpeg -i encoded.mp4 -c copy remuxed.mp4

dump all I-frames. This is same as encoded.mp4

frame, key_frame, pts, pict_type, coded_picture_number

frame,1,0.000000,I,0
frame,0,0.100000,I,3
frame,0,0.133333,I,4
frame,0,0.166667,I,5
frame,0,0.200000,I,6
frame,0,0.233333,I,7
frame,1,4.066667,I,122
frame,0,4.200000,I,126
frame,0,4.233333,I,127
frame,0,4.466667,I,134
frame,1,12.400000,I,372
frame,1,16.300000,I,489
frame,1,20.166667,I,605
frame,1,27.466667,I,824

All of I-frames are marked as keyframe in stss box.

[stss] Size=72 Version=0 Flags=0x000000 EntryCount=14 SampleNumber=[1, 4, 5, 6, 7, 8, 123, 127, 128, 135, 373, 490, 606, 825]

I think that h264_parser.c should not include following code. Because It is just for the specific camera... If we necessary following code, it should be more restricted code.

    // heuristic to detect non marked keyframes
    if (p->ps.sps->ref_frame_count <= 1 && p->ps.pps->ref_count[0] <= 1 && s->pict_type == AV_PICTURE_TYPE_I)
        s->key_frame = 1;