HEVC decoder applies alternative transfer function SEI in presentation order with leading pictures

[jeeb]

Summary of the bug:

In HEVC there is a concept called "leading pictures" (NAL unit types RASL, RADL), which can come after - among others - CRA ("clean random access") NAL units. This is where the pictures have a PTS before the random access sample, but come after it.

Sample: ​https://megumin.fushizen.eu/samples/2021-07-23-broadcast_with_alternative_transfer_sei.ts

Now, additionally - this sample utilizes a way of flagging both a "compatible" (as in, does not look too incorrect if interpreted as such) as well as the preferred (actual) transfer function with having:

1. A more compatible value in the VUI information (bt.2020, which is effectively the same as BT.709 etc - effectively BT.1886 on the rendering side).
2. The actual value in a separate SEI message at the CRA IRAP (ARIB STD-B67 aka HLG).

This SEI is defined that it should be applied in decoding order for the following sequence: "The alternative transfer characteristics SEI message persists for the current layer in decoding order from the current picture until the end of the CLVS"

Unfortunately, it seems like the libavcodec HEVC decoder applies these in presentation order, so the leading pictures which come after the CRA IRAP (but get output first in presentation order) will only get the "compatible" value, which leads to workflows that configure their output based on the first received frame (such as ffmpeg.c) to output the compatible value instead of the actual value.

How to reproduce:

% ffprobe -v verbose -i 'https://megumin.fushizen.eu/samples/2021-07-23-broadcast_with_alternative_transfer_sei.ts' -of json -show_frames -select_streams v > rasl_stream_frames.json

head -92 rasl_stream_frames.json 
{
    "frames": [
        {
            "media_type": "video",
            "stream_index": 0,
            "key_frame": 0,
            "pkt_pts": 7988680456,
            "pkt_pts_time": "88763.116178",
            "pkt_dts": 7988696656,
            "pkt_dts_time": "88763.296178",
            "best_effort_timestamp": 7988680456,
            "best_effort_timestamp_time": "88763.116178",
            "pkt_duration": 1800,
            "pkt_duration_time": "0.020000",
            "pkt_pos": "50384",
            "pkt_size": "38001",
            "width": 1920,
            "height": 1080,
            "pix_fmt": "yuv420p10le",
            "sample_aspect_ratio": "1:1",
            "pict_type": "B",
            "coded_picture_number": 0,
            "display_picture_number": 0,
            "interlaced_frame": 0,
            "top_field_first": 0,
            "repeat_pict": 0,
            "color_range": "tv",
            "color_space": "bt2020nc",
            "color_primaries": "bt2020",
            "color_transfer": "bt2020-10",
            "chroma_location": "left"
        },
        {
            "media_type": "video",
            "stream_index": 0,
            "key_frame": 0,
            "pkt_pts": 7988684056,
            "pkt_pts_time": "88763.156178",
            "pkt_dts": 7988698456,
            "pkt_dts_time": "88763.316178",
            "best_effort_timestamp": 7988684056,
            "best_effort_timestamp_time": "88763.156178",
            "pkt_duration": 1800,
            "pkt_duration_time": "0.020000",
            "pkt_pos": "117312",
            "pkt_size": "38000",
            "width": 1920,
            "height": 1080,
            "pix_fmt": "yuv420p10le",
            "sample_aspect_ratio": "1:1",
            "pict_type": "B",
            "coded_picture_number": 0,
            "display_picture_number": 0,
            "interlaced_frame": 0,
            "top_field_first": 0,
            "repeat_pict": 0,
            "color_range": "tv",
            "color_space": "bt2020nc",
            "color_primaries": "bt2020",
            "color_transfer": "bt2020-10",
            "chroma_location": "left"
        },
        {
            "media_type": "video",
            "stream_index": 0,
            "key_frame": 1,
            "pkt_pts": 7988700256,
            "pkt_pts_time": "88763.336178",
            "pkt_dts": 7988700256,
            "pkt_dts_time": "88763.336178",
            "best_effort_timestamp": 7988700256,
            "best_effort_timestamp_time": "88763.336178",
            "pkt_duration": 1800,
            "pkt_duration_time": "0.020000",
            "pkt_pos": "184992",
            "pkt_size": "38000",
            "width": 1920,
            "height": 1080,
            "pix_fmt": "yuv420p10le",
            "sample_aspect_ratio": "1:1",
            "pict_type": "I",
            "coded_picture_number": 0,
            "display_picture_number": 0,
            "interlaced_frame": 0,
            "top_field_first": 0,
            "repeat_pict": 0,
            "color_range": "tv",
            "color_space": "bt2020nc",
            "color_primaries": "bt2020",
            "color_transfer": "arib-std-b67",
            "chroma_location": "left"
        },

Expected result: all output frames should be flagged as "color_transfer": "arib-std-b67".
Received result: leading pictures get flagged with the base "compatible" transfer function.

[Balling]

Is this going to fix that? ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220218232001.345826-5-u@pkh.me/