Stream-level side data propagated to frames is indistinguishable from frame-level side data

[dbuitenh]

Summary of the bug:

Since ae22271620df90567ba276832b33607d64e9ba9c, stream level side data gets propagated to frames.

This causes issues, for example, when both the container and bitstream have the the same type of side data, such as, example, content light level metadata.

The issues arises because once the stream/container level side data is propagated to the frame, which already had side data of that type, the frame now has two side data of that type attached to it, with no way of figuring out which came from where.

This is made even worse due to the old av_frame_get_side_data API (which is still supported), only being able to return the first of any given side data attached to the frame.

After discussing with Anton and James, it was agreed that the solution is to only attach one of each type of side data to a frame, and to prefer frame-level/bitstream-level side data by default, if both exist, so that the user can choose which they prefer to use (if the stream-level was preferred by default, that would mean the default would be to not expose frame-level at all, which would be Bad(TM). There shall be an API to allow the user to prefer one or the other, which requires adding an AVList type to the API, according to Anton's preferences.

This blocks 7.0 since it is a regression with no work around. Bug opened per Michael's request.

How to reproduce:

Call av_frame_get_side_data() on frame(s) from the attached hdr10tags-both.mkv.

You can also see it easy like this:

% ffprobe -show_frames -of json hdr10tags-both.mkv

[... snip ...]

            "side_data_list": [
                {
                    "side_data_type": "Mastering display metadata",
                    "red_x": "11408507/16777216",
                    "red_y": "5368709/16777216",
                    "green_x": "2222981/8388608",
                    "green_y": "11576279/16777216",
                    "blue_x": "5033165/33554432",
                    "blue_y": "16106127/268435456",
                    "white_point_x": "10492471/33554432",
                    "white_point_y": "689963/2097152",
                    "min_luminance": "5368709/536870912",
                    "max_luminance": "1000/1"
                },
                {
                    "side_data_type": "Content light level metadata",
                    "max_content": 1000,
                    "max_average": 300
                },
                {
                    "side_data_type": "H.26[45] User Data Unregistered SEI message"
                },
                {
                    "side_data_type": "Mastering display metadata",
                    "red_x": "35400/50000",
                    "red_y": "14599/50000",
                    "green_x": "8500/50000",
                    "green_y": "39850/50000",
                    "blue_x": "6550/50000",
                    "blue_y": "2300/50000",
                    "white_point_x": "15634/50000",
                    "white_point_y": "16450/50000",
                    "min_luminance": "10/10000",
                    "max_luminance": "10000000/10000"
                }
            ]

Note the multiple indistinguishable side data on the first frame.

[elenril]

Fixed in commits preceding c0aee39412affb2a575f95e7a7d1f75274f7f8bc