h261dec doesn't mark keyframes

[Lastique]

h261dec does not set AVFrame::key_frame field on keyframes. As a side effect, frame skipping except keyframes doesn't work (all frames are skipped).

In H.261, keyframes are normally indicated by the Freeze Picture Release bit in the Picture Header of a frame, but h261dec ignores that bit. h261enc behavior is to set that bit on keyframes.

The problem was discovered in a client application that uses ffmpeg through its C API. I'm not sure if it can be reproduced through command line.

Attached is a patch that fixes the problem. The patch has been posted on ffmpeg-devel:

​https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2019-October/252145.html

[Lastique]

A patch to fix keyframe markup in h261dec

[Carl Eugen Hoyos]

The patch is ignored here and will not be backported.

[Lastique]

I disagree with "enhancement" qualification, this is a bug.

[Balling]

Replying to cehoyos:

The patch is ignored here and will not be backported.

The patch is here ​https://patchwork.ffmpeg.org/patch/15972/

[Balling]

I have a concern here from the code 02b7c630875c0bc63cee5ec597aa33baf9bf4e20

They say that they mark all I frames as key frames, even those that start the GOP. Is it possible in ancient H261 for such a frame to not be a key frame?

[Lastique]

Replying to Balling:

I have a concern here from the code 02b7c630875c0bc63cee5ec597aa33baf9bf4e20

They say that they mark all I frames as key frames, even those that start the GOP. Is it possible in ancient H261 for such a frame to not be a key frame?

This is not a question about the spec but rather the encoder implementation (i.e. h261enc.c).

The H.261 spec defines two modes of coding, INTER and INTRA. The latter ones do not use inter-frame prediction and frames entirely coded in this mode pretty much are what is understood as I-frames in the more modern MPEG codecs. The INTER mode uses motion compensation (MC) which references the previously coded picture. Pictures entirely coded in this mode are similar to the modern P-frames. I do not think the spec restricts the encoder to use only one mode for coding the entire picture, and in fact the spec doesn't specify pretty much anything about the encoder behavior.

But it does specify that upon a Fast update request (4.3.2) the encoder uses INTRA mode to encode the next picture and then uses the Freeze picture release (4.3.3) to signal the decoder that it may exit the freeze mode and start displaying decoded pictures after this one.

So, really, if the ffmpeg encoder always uses INTRA mode to encode entire pictures that were requested as AV_PICTURE_TYPE_I by the caller (or selected by the encoder as such), it has every reason to also mark these frames as Freeze picture release.

Respectively, if a decoder is seeing a Freeze picture release-marked frame, per H.261 spec, it should trust the encoder and assume this is an INTRA-only-coded picture and, in case of libavcodec, mark it as AV_PICTURE_TYPE_I and a keyframe. Which is what this patch does.

[Balling]

I frames are not the same as key frames. Only IDR frames are key frames in AVC/HEVC, e.g.

Key frame is not about decoding the frame. It is about decoding all frames after the key frame.

Marking all I frames as key frames is a bug in encoder.

[Lastique]

Replying to Balling:

I frames are not the same as key frames. Only IDR frames are key frames in AVC/HEVC, e.g.

Key frame is not about decoding the frame. It is about decoding all frames after the key frame.

An I-frame is a keyframe if there is no prediction across the I-frame. Which there isn't in the H.261 spec.

And in any case, the spec explicitly says the decoder should start displaying pictures starting with the one marked as Freeze picture release. Meaning that it guarantees that further pictures are decodable.

Marking all I frames as key frames is a bug in encoder.

It is not, if the encoder knows/guarantees that I-frames are actual keyframes.

[mkver]

Implemented in 88d8150de71cdb5588c42027b7fc516049a92c51.

[Balling]

Fixed in cf1c52c5c6cc82a27080fdaee53354f026401c7f

Yay

not, if the encoder knows/guarantees that I-frames are actual keyframes

Yes, in HEVC/AVC if closed GOP you just tag I frames that start the GOP as key frames, but then if open it is often the case that I frame is not a key frame. In practive even two I frames one after another in presentation order or decoding order are possible...

I know the case where even with closed (sic!!) GOP the decoding starts with two B frames in presentation order that only depend on 1 future pts I frame. See here: ​https://forum.doom9.org/showthread.php?p=2004952#post2004952

This whole topic is one big trip...