#10148 closed defect (fixed)

TS encoding of H.264 omits the SPS and PPS metadata

Reported by: John Coiner Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by John Coiner)

Summary of the bug

Users of OBS who stream with HLS are transmitting non-compliant HLS streams, and the root cause is in FFmpeg's mpegtsenc.c.

Background

HLS

Each chunk of an HLS stream should be independently-decodable -- it should begin with a key frame and include any metadata (eg. SPS, PPS) needed to initialize the decoders. Section 3 of the HLS RFC requires this: "Any Media Segment that contains video SHOULD include enough information to initialize a video decoder and decode a continuous set of frames ..."

YouTube

I work at YouTube Live. YouTube allows creators to upload live streams with HLS: https://support.google.com/youtube/answer/10349430?hl=en

Sometimes creators transmit non-compliant HLS streams. Today YouTube is able to support them, with the caveat that noncompliant HLS uploads reduce the reliability of the resulting livestream broadcasts. In the future, it would be nice if OBS would produce compliant streams and YouTube could eventually reject noncompliant HLS.

The Bug

The OBS HLS implementation uses the muxer in mpegtsenc.c to format media into TS segments.

For H.264, this file has logic that intends to take the "extradata" (which includes the SPS and PPS) from the codec parser and re-emit it at the beginning of key-frame segments. This is the logic:

    if (st->codecpar->codec_id == AV_CODEC_ID_H264) {
        const uint8_t *p = buf, *buf_end = p + size;
        uint32_t state = -1;
        int extradd = (pkt->flags & AV_PKT_FLAG_KEY) ? st->codecpar->extradata_size : 0;
        int ret = ff_check_h264_startcode(s, st, pkt);
        if (ret < 0)
            return ret;

        if (extradd && AV_RB24(st->codecpar->extradata) > 1)
            extradd = 0;

        do {
            p = avpriv_find_start_code(p, buf_end, &state);
            av_log(s, AV_LOG_TRACE, "nal %"PRId32"\n", state & 0x1f);
            if ((state & 0x1f) == 7)
                extradd = 0;
        } while (p < buf_end && (state & 0x1f) != 9 &&
                 (state & 0x1f) != 5 && (state & 0x1f) != 1);

        if ((state & 0x1f) != 5)
            extradd = 0;
        if ((state & 0x1f) != 9) { // AUD NAL
            data = av_malloc(pkt->size + 6 + extradd);
            if (!data)
                return AVERROR(ENOMEM);
            memcpy(data + 6, st->codecpar->extradata, extradd);
            memcpy(data + 6 + extradd, pkt->data, pkt->size);
            AV_WB32(data, 0x00000001);
            data[4] = 0x09;
            data[5] = 0xf0; // any slice type (0xe) + rbsp stop one bit
            buf     = data;
            size    = pkt->size + 6 + extradd;
        }
    } else ...

This code scans the segment for NALs until it finds either an Access Unit Delimiter (9), an IDR picture (5), another SPS (7), or a non-IDR picture (1). If the first one found is an IDR picture (a key frame) it re-emits the "extradata" into its output buffer ahead of the key frame.

The problem is that encoders may preface the key frame with an Access Unit Delimiter, but no SPS/PPS. In that case this code does not repeat the "extradata" and the resulting HLS stream may be noncompliant and unjoinable.

How to reproduce

EDIT: See comment 15 for a simple method that works with any input file.

Comment 7 is an earlier path to reproduce, which relied on a particular input file.

I don't know how these content creators are configuring their OBS setups. (We know that it's OBS, thanks to its User-Agent of "libobs...") My guess is:

  • They are using some VAAPI-supported hardware. Unlike the NVENC driver, the VAAPI driver in OBS has no way to hint to the underlying hardware that it should repeat SPS and PPS.
  • The underlying hardware or driver defaults to never repeating SPS and PPS.
  • The underlying hardware or driver defaults to emitting Access Unit Delimiters, similar to running x264 with its "aud" option enabled.
  • The encoded bitstream interacts with this bug to produce an unjoinable HLS stream.

More than half of OBS+H.264+HLS users are producing compliant HLS streams -- perhaps because they are using an encoder that repeats SPS and PPS and thus doesn't rely on mpegtsenc.c to repeat it.

Attachments (3)

sample_input_ffmpeg_bug_10148.ts (1.0 MB ) - added by John Coiner 15 months ago.
0001-Proposed-fix-for-https-trac.ffmpeg.org-ticket-10148.patch (1.6 KB ) - added by John Coiner 15 months ago.
Proposed patch
0001-Proposed-fix-for-https-trac.ffmpeg.org-ticket-10148.patch.txt (1.6 KB ) - added by John Coiner 15 months ago.
Proposed patch, as plain text

Download all attachments as: .zip

Change History (23)

comment:1 by John Coiner, 15 months ago

Summary: M2TS encoding of H.264 omits the init segmentM2TS encoding of H.264 omits the SPS and PPS medata

comment:2 by John Coiner, 15 months ago

Description: modified (diff)

comment:3 by Balling, 15 months ago

Are there any plans for Youtube to change resolutions mid segment? You know, download other resolution segment, decode not real time to needed frame and then change to new video stream mid segment.

Because I do not like how you need to skip to get to new resolution.

into M2TS segments.

Just TS segments, no?

Last edited 15 months ago by Balling (previous) (diff)

comment:5 by John Coiner, 15 months ago

Description: modified (diff)
Summary: M2TS encoding of H.264 omits the SPS and PPS medataTS encoding of H.264 omits the SPS and PPS medata

comment:6 by John Coiner, 15 months ago

Thanks Balling. I've substituted M2TS with TS in the description.

To be clear, the problem concerns "ingestions" -- Live uploads to YouTube. For these, YouTube doesn't decide when the resolution changes -- the content creator does. I suspect YouTube supports resolution changes in the middle of an HLS ingestion and even in the middle of an HLS chunk, but haven't tested this. Certainly, we should not do anything that would preclude this.

I'm not sure how the posted patch would preclude a change of resolution; could you elaborate on the sequence you're concerned about? Thanks.

by John Coiner, 15 months ago

comment:7 by John Coiner, 15 months ago

To reproduce, use the attached file and run:

 $ ffmpeg -re -i sample_input_ffmpeg_bug_10148.ts -codec copy -f hls -method PUT 'https://upload.youtube.com/http_upload_hls?cid=<<YOUR_STREAM_KEY>>&copy=0&file=out.m3u8'

(Is there an option to capture HLS output to local files? That'd work too; certainly uploading to YouTube should not really be necessary to check this. Just upload to any webserver that accepts PUT/POST.)

This upload a few HLS chunks. Only the first chunk is independently-decodable, all others are missing their SPS/PPS due to the bug. With the suggested fix, all chunks include SPS/PPS.

comment:8 by John Coiner, 15 months ago

Description: modified (diff)

by John Coiner, 15 months ago

Proposed patch

by John Coiner, 15 months ago

Proposed patch, as plain text

comment:9 by Balling, 15 months ago

Status: newopen

You need to use git send-email. You cannot use anything else because of broken google email protocol.

See: https://patchwork.ffmpeg.org/project/ffmpeg/patch/CAEP=B=FNT7X1cdnhHHTA60XZcnOnTtfR=w4vLDqsKVcdU0depg@mail.gmail.com/

Last edited 15 months ago by Balling (previous) (diff)

comment:10 by John Coiner, 15 months ago

Thanks for your patience with me.

git send-email produced http://ffmpeg.org/pipermail/ffmpeg-devel/2023-January/306028.html

comment:11 by John Coiner, 15 months ago

Summary: TS encoding of H.264 omits the SPS and PPS medataTS encoding of H.264 omits the SPS and PPS metadata

comment:12 by John Coiner, 15 months ago

Description: modified (diff)

comment:13 by John Coiner, 15 months ago

Description: modified (diff)

comment:14 by Balling, 15 months ago

Did you read the response in https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230127211841.59628-1-jpcoiner@gmail.com/

??

Can you apply the patch mentioned there (and here in comment 4).

Last edited 15 months ago by Balling (previous) (diff)

comment:15 by John Coiner, 15 months ago

Here's a simpler process to reproduce the bug with probably any input file:

ffmpeg -i <any_input_media> -vf scale=320:-1 -c:v libx264 -x264-params aud=1 -c:a aac -f hls -method PUT 'https://upload.youtube.com/http_upload_hls?cid=<your-stream-key>&copy=0&nightly=1&file=out.m3u8'

... where 'youtube.com' can be any HTTP server that accepts PUT, there's nothing specific to youtube here.

Without the fix, that produces an unjoinable and noncompliant upload where all HLS media segments after the first are not independently decodable. With the proposed fix, all are independently decodable.

I tested this with the input file already attached to this bug, as well as some others.

Last edited 15 months ago by John Coiner (previous) (diff)

comment:16 by John Coiner, 15 months ago

Description: modified (diff)

comment:17 by John Coiner, 15 months ago

Description: modified (diff)

The latest proposed patch is here: http://ffmpeg.org/pipermail/ffmpeg-devel/2023-January/306143.html

This obsoletes the earlier proposed patch.

comment:18 by John Coiner, 15 months ago

Balling, btw I confirmed that this is not a duplicate of https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_EE0E40DE9A569FE5AB454E6E700A2DA79A08@qq.com/

Adding prints to h264_mp4toannexb_bsf.c confirms that it's not reached in the sequence that reproduces this bug. It must be unrelated.

Thanks.

comment:20 by Balling, 14 months ago

Resolution: fixed
Status: openclosed
Note: See TracTickets for help on using tickets.