`ffprobe` reported MPEG-2 GOP header timecodes 1 frame early

[MaxEliaserAWS]

Hi maintainers! Reaching out with an issue in the mpeg2video decoder and corresponding ffprobe output, but I also want to extend the gratitude of AWS Elemental for this awesome analysis tool!

Versions Tested

I have reproduced this in the following FFMPEG/ffprobe versions:

• 3.4.7 (packaged by RedHat)
• 5.1.1 (static build from johnvansickle.com)
• 7.0.2 (static build from johnvansickle.com)
• Git commit 6cf4186d1b, compiled myself on AmazonLinux 2. This was the latest master as of this writing.

Input File Description

Just to give everyone their due, I used as test content the short film _Big Buck Bunny_, which is copyright 2008 by the Blender Foundation under a CC-BY 3.0 license, see ​https://peach.blender.org/about/.

This test file is an MPEG2 essence (e.g. ISO/IEC 13818-2 without using a transport stream wrapper.) It has a fixed 90 frame GOP size and GOP timecodes are enabled. There are no B-frames and no frame reordering. Therefore, there should be embedded timecodes on each GOP header, at frames 0, 90, 180, 270, etc, and these frame indices should be the same in decode order and presentation order. I have confirmed that this is the case using multiple other analyzer tools. You can see in these screenshots that Telestream Switch shows a timecode 00:00:03;00 on the I-frame at frame index 90 and indicates that the GOP header is present on that frame, whereas Switch does not show a GOP header present the P-frame at frame index 89 (although it can still extrapolate a timecode for that frame.)

Steps To Reproduce

ffprobe -v quiet -print_format json -show_frames out_MPEG.m2v

Expected Result

The GOP timecodes should be reported on frames 0, 90, 180, 270, etc. The results should be consistent with the coded_picture_number field as well as the actual index of the frame within the JSON array.

Since coded_picture_number appears to have been removed in latest Git, I wrote this short Python script to identify which frames have GOP timecodes in a way that will be compatible with the latest ffprobe's output format.

#! /usr/bin/env python3

import sys, json

if len(sys.argv) > 1:
    infile = open(sys.argv[1], "r")
else:
    infile = sys.stdin

content = json.load(infile)

for i, frame in enumerate(content["frames"]):
    if "side_data_list" in frame:
        for side_data in frame["side_data_list"]:
            if side_data["side_data_type"] == "GOP timecode":
                print("Found GOP timecode %s on frame %d" % (side_data["timecode"], i))
                break

Actual Result

Although the timecode is reported on frame 0 as expected, subsequently they are reported a frame too early; on frames 89, 179, 269, etc.

~/FFmpeg$ ./ffprobe -v quiet -print_format json -show_frames /tmp/out_MPEG.m2v | /tmp/analyze_ffprobe_output.py
Found GOP timecode 00:00:00;00 on frame 0
Found GOP timecode 00:00:03;00 on frame 89
Found GOP timecode 00:00:06;00 on frame 179
Found GOP timecode 00:00:09;00 on frame 269
Found GOP timecode 00:00:12;00 on frame 359
Found GOP timecode 00:00:15;00 on frame 449

I am confident that this does not match the actual structure of the file.

Tentative Analysis

I did some debugging of the issue in ffprobe 3.4.7; although I have reproed in latest Git, I did not repeat the debugging exercise there. This is the best diagnosis I can give but I am not an expert on this codebase so take with a grain of salt.

From what I can tell, the code in mpeg12dec.c can sometimes buffer a frame internally before emitting it. I believe this is the "latency of 1 frame" documented here:
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L1752

So when decode_chunks in that file encounters the GOP header, it stashes the GOP timecode in Mpeg1Context.timecode_frame_start:
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L2331
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L2149

It then encounters the first frame in the new GOP and stores it in MpegEncContext.cur_pic:
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L2446
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L1358

And, when decode_chunks is about to return, it calls into slice_end to flush the decoded frame:
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L2198
However, slice_end can (for I-frames and P-frames) instead emit MpegEncContext.last_pic, which goes into the AVFrame which we are working on:
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L1752
After slice_end and decode_chunks have returned, mpeg_decode_frame sees the timecode stashed in Mpeg1Context.timecode_frame_start and always attaches it to the same AVFrame, regardless of whether that timecode was actually destined for cur_pic instead of for last_pic:
​https://github.com/FFmpeg/FFmpeg/blob/afb06aef7ebeaecd843d0af62dd32546245475c2/libavcodec/mpeg12dec.c#L2560

Since the MPEG2 format only ever puts timecodes at the start of a new GOP (definitionally always an I frame,) then aside from frame 0, I believe this code will _always_ assign the timecode to the wrong frame.

[MaxEliaserAWS]

the input file in question

[Balling]

"Since the MPEG2 format only ever puts timecodes at the start of a new GOP (definitionally always an I frame)"

In decoding order I frame is always first, not in presentation order.

[MasterQuestionable]

͏    Supposedly the interpreted presentation timestamps shall be still alright..?
͏    (so the media playback experience shouldn't differ)

͏    If so, then this might be essentially no issue.

[MaxEliaserAWS]

Correct, no issue with playback, and no issue with PTS/DTS, only with the timecodes. The only user-visible symptom would be in the ffprobe output as I have described.

We are using ffprobe in our test automation to ensure that the GOP timecodes in our files are correct. If electing not to fix the issue, that is fine, I can just configure the test to look on frame 89 instead of 90-- I just needed to file a ticket I can reference as to why the test is written that way.

[MasterQuestionable]

͏    I wonder why is the GOP timecodes thing needed after all..?
͏    Feels much like much useless metadata I've seen before.

[Balling]

MPEG2 has timecode on first and last frame as part of bitstream.

[MaxEliaserAWS]

I wonder why is the GOP timecodes thing needed after all..?

Embedded timecodes are typically used in the industry for synchronization with sidecar assets such as audio or captions which may be in external files.

There is some information on the audio usecase here, from an audio equipment vendor:
​https://rode.com/en/about/news-info/what-is-timecode-and-why-do-you-need-it

[MasterQuestionable]

͏    Thanks for the info.
͏    But isn't that overlapping with presentation timestamps?
͏    And just derivable if the strange SMPTE format is preferred. (given CFR premise)

͏    My doubt is primarily:
͏    Why should the timestamps of GOP be additionally stored, when directly derivable?

͏    ----

͏    Even if the other sources of the same scene were somehow recorded out-of-sync:
͏    Knowing when each recording started should suffice.
͏    No real need of duplicating the timestamps.

[MasterQuestionable]

͏    "MPEG2 has timecode on first and last frame as part of bitstream."
͏    Alike the useless prediction type meta in PNG..?
͏    ​https://trac.ffmpeg.org/ticket/11232#comment:1

[Balling]

But isn't that overlapping with presentation timestamps?

No, it is less precise, so typically PTS are preferred. ; vs : signal different types of timecodes. ; means DF, : NDF. And DF while more accurate one is still only accurate for 9 hours 15 minutes.

Alike the useless prediction type meta in PNG..?

There are a lot of strange things in MPEG2 bitstream. That thing alone. lol ​https://lists.ffmpeg.org/pipermail/ffmpeg-user/2024-May/058023.html

See, we do not implement those ​http://coverage.ffmpeg.org/index.cbs_mpeg2_syntax_template.c.9a102fef8e454157b19acdde3ec27f95.html#l290

[MasterQuestionable]

͏    So much tracing the legacy bizarrerie and quaintness..?

͏    More on the SMPTE timecode:
͏    ​https://trac.ffmpeg.org/attachment/ticket/11055/Ticket_11055.m2ts.xml#ref-1

[MaxEliaserAWS]

Why should the timestamps of GOP be additionally stored, when directly derivable?

I don't really disagree, for containers/codecs that can signal a nonzero starting PTS. I would just note that there are a lot of workflows/software out there which still rely on timecode and which do not do this derivation even though it is possible to do.

An example of where it might not be as easily derivable is in the QuickTime format, where the STTS/CTTS box structure cannot easily achieve a nonzero starting PTS, but where timecode can be signaled as a separate track of type tmcd to start the timeline at 10:00:00:00.

It is still standard in television production to start timecodes at 10 hours (1 hour for film.) This practice was established because mechanical tape/film equipment needs a little bit of time to get its motors spinning up to full speed, so the tape can't start at zero; if the pre-roll starts at 9:59:52:00, then the actual content can start at 10:00:00:00, a nice round number. Although we don't use this type of equipment anymore, this convention is still absolutely standard in television/broadcast, and there are a lot of files out there which signal this in timecode but not in PTS. Some information on this is here:
​https://youtu.be/C-aJEhZtnVo
(Apologies for video, I would have provided a written resource if I could find one.)

If we were to redesign all video software and file formats today, I agree it would be better to just focus on PTS and eliminate SMPTE timecode. But old technology still has a lot of influence from beyond the grave, and people still want to watch the old content. (Otherwise we'd have stopped using 29.976 FPS in north America a long time ago.)

For ffmpeg's use case as a self-contained transcoder, it's probably easy to avoid using timecode, but for ffprobe's use as a video analyzer tool or avcodec's use in other video software, I would argue for its continued usefulness.

Anyway, thanks for the opportunity to nerd out about video stuff, this was fun!

-Max

[MasterQuestionable]

͏    As I'm trying to sort these things out: I have to pay attention to every, even the slightest details.
͏    So to not retrace the very old mistakes.

͏    Based on your description the 10 h alike offset appears constant, and applies environment-wide.
͏    So there should be no difference just adding/subtracting it to satisfy specific use.
͏    [ Bear the havoc of alike, of course. ]

͏    Legacy workflows have their own expectations.
͏    That modern technologies may not satisfy, for each other.

͏    Old content may as well be restored in new definitions.
͏    And when properly done: likely of higher quality.
͏    [ Modern structure is VFR based. 29.976 whatsoever, doesn't really matter. ]

͏    Whatsoever you seem to miss the core question:
͏    "timestamps of GOP be additionally stored"
͏    ; not really about SMPTE timecodes in general.

[[
͏    "Why should the timestamps of GOP be additionally stored, when directly derivable?"
͏    I don't really disagree, for containers/codecs that can signal a non-0 starting PTS.

͏    I would just note that there are a lot of workflows/software out there, which still rely on timecode:
͏    And which do not do this derivation, even though it is possible to do.

͏    An example of where it might not be as easily derivable is in the QuickTime format:
͏    Where the STTS/CTTS box structure cannot easily achieve a non-0 starting PTS.
͏    But where timecode can be signaled as a separate track of type "tmcd", to start the timeline at 10:00:00:00.

͏    It is still standard in television production to start timecodes at 10 hours (1 hour for film).
͏    This practice was established because mechanical tape/film equipment needs a little bit of time, to get its motors spinning up to full speed: so the tape can't start at 0.
͏    If the pre-roll starts at 9:59:52:00, then the actual content can start at 10:00:00:00, a nice round number.
͏    .
͏    Although we don't use this type of equipment anymore, this convention is still absolutely standard in television/broadcast.
͏    And there are a lot of files out there which signal this in timecode but not in PTS.

͏    Some information on this is here:
͏    [20210407] Why Does TV Timecode Start At 10 HOURS???
͏    704 s (11:44)
͏    ​https://www.youtube.com/watch?v=C-aJEhZtnVo
͏    The Crow Hill Company (@TheCrowHillCo)
[[
͏    In a recent series of seminars about writing FACENT or "factual entertainment", a few of you asked why did the timecode for TV start at 10 hours???
͏    Well in this vlog I attempt to explain that with a few historical nuggets, that even the most seasoned of you may be surprised you didn't know!
]]
͏    (apologies for video, I would have provided a written resource if I could find one)

͏    If we were to redesign all video software and file formats today, I agree it would be better to just focus on PTS and eliminate SMPTE timecode.
͏    But old technology still has a lot of influence from beyond the grave, and people still want to watch the old content.
͏    (otherwise we'd have stopped using 29.976 FPS in North America a long time ago)

͏    For FFmpeg's use case as a self-contained transcoder, it's probably easy to avoid using timecode.
͏    But for FFprobe's use as a video analyzer tool or "avcodec" 's use in other video software, I would argue for its continued usefulness.

͏    Anyway, thanks for the opportunity to nerd out about video stuff. This was fun!
]]