TS encoding of H.264 omits the SPS and PPS metadata

[John Coiner]

Summary of the bug

Users of OBS who stream with HLS are transmitting non-compliant HLS streams, and the root cause is in FFmpeg's mpegtsenc.c.

Background

HLS

Each chunk of an HLS stream should be independently-decodable -- it should begin with a key frame and include any metadata (eg. SPS, PPS) needed to initialize the decoders. Section 3 of the HLS RFC requires this: "Any Media Segment that contains video SHOULD include enough information to initialize a video decoder and decode a continuous set of frames ..."

YouTube

I work at YouTube Live. YouTube allows creators to upload live streams with HLS: ​https://support.google.com/youtube/answer/10349430?hl=en

Sometimes creators transmit non-compliant HLS streams. Today YouTube is able to support them, with the caveat that noncompliant HLS uploads reduce the reliability of the resulting livestream broadcasts. In the future, it would be nice if OBS would produce compliant streams and YouTube could eventually reject noncompliant HLS.

The Bug

The OBS HLS implementation uses the muxer in mpegtsenc.c to format media into TS segments.

For H.264, this file has logic that intends to take the "extradata" (which includes the SPS and PPS) from the codec parser and re-emit it at the beginning of key-frame segments. This is the logic:

    if (st->codecpar->codec_id == AV_CODEC_ID_H264) {
        const uint8_t *p = buf, *buf_end = p + size;
        uint32_t state = -1;
        int extradd = (pkt->flags & AV_PKT_FLAG_KEY) ? st->codecpar->extradata_size : 0;
        int ret = ff_check_h264_startcode(s, st, pkt);
        if (ret < 0)
            return ret;

        if (extradd && AV_RB24(st->codecpar->extradata) > 1)
            extradd = 0;

        do {
            p = avpriv_find_start_code(p, buf_end, &state);
            av_log(s, AV_LOG_TRACE, "nal %"PRId32"\n", state & 0x1f);
            if ((state & 0x1f) == 7)
                extradd = 0;
        } while (p < buf_end && (state & 0x1f) != 9 &&
                 (state & 0x1f) != 5 && (state & 0x1f) != 1);

        if ((state & 0x1f) != 5)
            extradd = 0;
        if ((state & 0x1f) != 9) { // AUD NAL
            data = av_malloc(pkt->size + 6 + extradd);
            if (!data)
                return AVERROR(ENOMEM);
            memcpy(data + 6, st->codecpar->extradata, extradd);
            memcpy(data + 6 + extradd, pkt->data, pkt->size);
            AV_WB32(data, 0x00000001);
            data[4] = 0x09;
            data[5] = 0xf0; // any slice type (0xe) + rbsp stop one bit
            buf     = data;
            size    = pkt->size + 6 + extradd;
        }
    } else ...

This code scans the segment for NALs until it finds either an Access Unit Delimiter (9), an IDR picture (5), another SPS (7), or a non-IDR picture (1). If the first one found is an IDR picture (a key frame) it re-emits the "extradata" into its output buffer ahead of the key frame.

The problem is that encoders may preface the key frame with an Access Unit Delimiter, but no SPS/PPS. In that case this code does not repeat the "extradata" and the resulting HLS stream may be noncompliant and unjoinable.

How to reproduce

EDIT: See comment 15 for a simple method that works with any input file.

Comment 7 is an earlier path to reproduce, which relied on a particular input file.

I don't know how these content creators are configuring their OBS setups. (We know that it's OBS, thanks to its User-Agent of "libobs...") My guess is:

• They are using some VAAPI-supported hardware. Unlike the NVENC driver, the VAAPI driver in OBS has no way to hint to the underlying hardware that it should repeat SPS and PPS.
• The underlying hardware or driver defaults to never repeating SPS and PPS.
• The underlying hardware or driver defaults to emitting Access Unit Delimiters, similar to running x264 with its "aud" option enabled.
• The encoded bitstream interacts with this bug to produce an unjoinable HLS stream.

More than half of OBS+H.264+HLS users are producing compliant HLS streams -- perhaps because they are using an encoder that repeats SPS and PPS and thus doesn't rely on mpegtsenc.c to repeat it.

[Balling]

Are there any plans for Youtube to change resolutions mid segment? You know, download other resolution segment, decode not real time to needed frame and then change to new video stream mid segment.

Because I do not like how you need to skip to get to new resolution.

into M2TS segments.

Just TS segments, no?

[Balling]

See also ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_EE0E40DE9A569FE5AB454E6E700A2DA79A08@qq.com/

[John Coiner]

Thanks Balling. I've substituted M2TS with TS in the description.

To be clear, the problem concerns "ingestions" -- Live uploads to YouTube. For these, YouTube doesn't decide when the resolution changes -- the content creator does. I suspect YouTube supports resolution changes in the middle of an HLS ingestion and even in the middle of an HLS chunk, but haven't tested this. Certainly, we should not do anything that would preclude this.

I'm not sure how the posted patch would preclude a change of resolution; could you elaborate on the sequence you're concerned about? Thanks.

[John Coiner]

To reproduce, use the attached file and run:

 $ ffmpeg -re -i sample_input_ffmpeg_bug_10148.ts -codec copy -f hls -method PUT 'https://upload.youtube.com/http_upload_hls?cid=<<YOUR_STREAM_KEY>>&copy=0&file=out.m3u8'

(Is there an option to capture HLS output to local files? That'd work too; certainly uploading to YouTube should not really be necessary to check this. Just upload to any webserver that accepts PUT/POST.)

This upload a few HLS chunks. Only the first chunk is independently-decodable, all others are missing their SPS/PPS due to the bug. With the suggested fix, all chunks include SPS/PPS.

[John Coiner]

Proposed patch

[John Coiner]

Proposed patch, as plain text

[Balling]

You need to use git send-email. You cannot use anything else because of broken google email protocol.

See: ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/CAEP=B=FNT7X1cdnhHHTA60XZcnOnTtfR=w4vLDqsKVcdU0depg@mail.gmail.com/

[John Coiner]

Thanks for your patience with me.

git send-email produced ​http://ffmpeg.org/pipermail/ffmpeg-devel/2023-January/306028.html

[Balling]

Did you read the response in ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230127211841.59628-1-jpcoiner@gmail.com/

??

Can you apply the patch mentioned there (and here in comment 4).

[John Coiner]

Here's a simpler process to reproduce the bug with probably any input file:

ffmpeg -i <any_input_media> -vf scale=320:-1 -c:v libx264 -x264-params aud=1 -c:a aac -f hls -method PUT 'https://upload.youtube.com/http_upload_hls?cid=<your-stream-key>&copy=0&nightly=1&file=out.m3u8'

... where 'youtube.com' can be any HTTP server that accepts PUT, there's nothing specific to youtube here.

Without the fix, that produces an unjoinable and noncompliant upload where all HLS media segments after the first are not independently decodable. With the proposed fix, all are independently decodable.

I tested this with the input file already attached to this bug, as well as some others.

[John Coiner]

The latest proposed patch is here: ​http://ffmpeg.org/pipermail/ffmpeg-devel/2023-January/306143.html

This obsoletes the earlier proposed patch.

[John Coiner]

Balling, btw I confirmed that this is not a duplicate of ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_EE0E40DE9A569FE5AB454E6E700A2DA79A08@qq.com/

Adding prints to h264_mp4toannexb_bsf.c confirms that it's not reached in the sequence that reproduces this bug. It must be unrelated.

Thanks.

[John Coiner]

V2 patch posted: ​http://ffmpeg.org/pipermail/ffmpeg-devel/2023-February/306555.html

[Balling]

Fixed in e0cb89c354ea654bccc524a8a45321b93cb256d3.