Opened 22 months ago
Closed 21 months ago
#10148 closed defect (fixed)
TS encoding of H.264 omits the SPS and PPS metadata
Reported by: | John Coiner | Owned by: | |
---|---|---|---|
Priority: | normal | Component: | undetermined |
Version: | git-master | Keywords: | |
Cc: | Blocked By: | ||
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description (last modified by )
Summary of the bug
Users of OBS who stream with HLS are transmitting non-compliant HLS streams, and the root cause is in FFmpeg's mpegtsenc.c.
Background
HLS
Each chunk of an HLS stream should be independently-decodable -- it should begin with a key frame and include any metadata (eg. SPS, PPS) needed to initialize the decoders. Section 3 of the HLS RFC requires this: "Any Media Segment that contains video SHOULD include enough information to initialize a video decoder and decode a continuous set of frames ..."
YouTube
I work at YouTube Live. YouTube allows creators to upload live streams with HLS: https://support.google.com/youtube/answer/10349430?hl=en
Sometimes creators transmit non-compliant HLS streams. Today YouTube is able to support them, with the caveat that noncompliant HLS uploads reduce the reliability of the resulting livestream broadcasts. In the future, it would be nice if OBS would produce compliant streams and YouTube could eventually reject noncompliant HLS.
The Bug
The OBS HLS implementation uses the muxer in mpegtsenc.c to format media into TS segments.
For H.264, this file has logic that intends to take the "extradata" (which includes the SPS and PPS) from the codec parser and re-emit it at the beginning of key-frame segments. This is the logic:
if (st->codecpar->codec_id == AV_CODEC_ID_H264) { const uint8_t *p = buf, *buf_end = p + size; uint32_t state = -1; int extradd = (pkt->flags & AV_PKT_FLAG_KEY) ? st->codecpar->extradata_size : 0; int ret = ff_check_h264_startcode(s, st, pkt); if (ret < 0) return ret; if (extradd && AV_RB24(st->codecpar->extradata) > 1) extradd = 0; do { p = avpriv_find_start_code(p, buf_end, &state); av_log(s, AV_LOG_TRACE, "nal %"PRId32"\n", state & 0x1f); if ((state & 0x1f) == 7) extradd = 0; } while (p < buf_end && (state & 0x1f) != 9 && (state & 0x1f) != 5 && (state & 0x1f) != 1); if ((state & 0x1f) != 5) extradd = 0; if ((state & 0x1f) != 9) { // AUD NAL data = av_malloc(pkt->size + 6 + extradd); if (!data) return AVERROR(ENOMEM); memcpy(data + 6, st->codecpar->extradata, extradd); memcpy(data + 6 + extradd, pkt->data, pkt->size); AV_WB32(data, 0x00000001); data[4] = 0x09; data[5] = 0xf0; // any slice type (0xe) + rbsp stop one bit buf = data; size = pkt->size + 6 + extradd; } } else ...
This code scans the segment for NALs until it finds either an Access Unit Delimiter (9), an IDR picture (5), another SPS (7), or a non-IDR picture (1). If the first one found is an IDR picture (a key frame) it re-emits the "extradata" into its output buffer ahead of the key frame.
The problem is that encoders may preface the key frame with an Access Unit Delimiter, but no SPS/PPS. In that case this code does not repeat the "extradata" and the resulting HLS stream may be noncompliant and unjoinable.
How to reproduce
EDIT: See comment 15 for a simple method that works with any input file.
Comment 7 is an earlier path to reproduce, which relied on a particular input file.
I don't know how these content creators are configuring their OBS setups. (We know that it's OBS, thanks to its User-Agent of "libobs...") My guess is:
- They are using some VAAPI-supported hardware. Unlike the NVENC driver, the VAAPI driver in OBS has no way to hint to the underlying hardware that it should repeat SPS and PPS.
- The underlying hardware or driver defaults to never repeating SPS and PPS.
- The underlying hardware or driver defaults to emitting Access Unit Delimiters, similar to running x264 with its "aud" option enabled.
- The encoded bitstream interacts with this bug to produce an unjoinable HLS stream.
More than half of OBS+H.264+HLS users are producing compliant HLS streams -- perhaps because they are using an encoder that repeats SPS and PPS and thus doesn't rely on mpegtsenc.c to repeat it.
Attachments (3)
Change History (23)
comment:1 by , 22 months ago
Summary: | M2TS encoding of H.264 omits the init segment → M2TS encoding of H.264 omits the SPS and PPS medata |
---|
comment:2 by , 22 months ago
Description: | modified (diff) |
---|
comment:4 by , 22 months ago
comment:5 by , 21 months ago
Description: | modified (diff) |
---|---|
Summary: | M2TS encoding of H.264 omits the SPS and PPS medata → TS encoding of H.264 omits the SPS and PPS medata |
comment:6 by , 21 months ago
Thanks Balling. I've substituted M2TS with TS in the description.
To be clear, the problem concerns "ingestions" -- Live uploads to YouTube. For these, YouTube doesn't decide when the resolution changes -- the content creator does. I suspect YouTube supports resolution changes in the middle of an HLS ingestion and even in the middle of an HLS chunk, but haven't tested this. Certainly, we should not do anything that would preclude this.
I'm not sure how the posted patch would preclude a change of resolution; could you elaborate on the sequence you're concerned about? Thanks.
by , 21 months ago
Attachment: | sample_input_ffmpeg_bug_10148.ts added |
---|
comment:7 by , 21 months ago
To reproduce, use the attached file and run:
$ ffmpeg -re -i sample_input_ffmpeg_bug_10148.ts -codec copy -f hls -method PUT 'https://upload.youtube.com/http_upload_hls?cid=<<YOUR_STREAM_KEY>>©=0&file=out.m3u8'
(Is there an option to capture HLS output to local files? That'd work too; certainly uploading to YouTube should not really be necessary to check this. Just upload to any webserver that accepts PUT/POST.)
This upload a few HLS chunks. Only the first chunk is independently-decodable, all others are missing their SPS/PPS due to the bug. With the suggested fix, all chunks include SPS/PPS.
comment:8 by , 21 months ago
Description: | modified (diff) |
---|
by , 21 months ago
Attachment: | 0001-Proposed-fix-for-https-trac.ffmpeg.org-ticket-10148.patch added |
---|
Proposed patch
by , 21 months ago
Attachment: | 0001-Proposed-fix-for-https-trac.ffmpeg.org-ticket-10148.patch.txt added |
---|
Proposed patch, as plain text
comment:9 by , 21 months ago
Status: | new → open |
---|
You need to use git send-email. You cannot use anything else because of broken google email protocol.
comment:10 by , 21 months ago
Thanks for your patience with me.
git send-email produced http://ffmpeg.org/pipermail/ffmpeg-devel/2023-January/306028.html
comment:11 by , 21 months ago
Summary: | TS encoding of H.264 omits the SPS and PPS medata → TS encoding of H.264 omits the SPS and PPS metadata |
---|
comment:12 by , 21 months ago
Description: | modified (diff) |
---|
comment:13 by , 21 months ago
Description: | modified (diff) |
---|
comment:14 by , 21 months ago
Did you read the reponse in https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230127211841.59628-1-jpcoiner@gmail.com/
??
Can you apply the patch mentioned there (and here in comment 4).
comment:15 by , 21 months ago
Here's a simpler process to reproduce the bug with probably any input file:
ffmpeg -i <any_input_media> -vf scale=320:-1 -c:v libx264 -x264-params aud=1 -c:a aac -f hls -method PUT 'https://upload.youtube.com/http_upload_hls?cid=<your-stream-key>©=0&nightly=1&file=out.m3u8'
... where 'youtube.com' can be any HTTP server that accepts PUT, there's nothing specific to youtube here.
Without the fix, that produces an unjoinable and noncompliant upload where all HLS media segments after the first are not independently decodable. With the proposed fix, all are independently decodable.
I tested this with the input file already attached to this bug, as well as some others.
comment:16 by , 21 months ago
Description: | modified (diff) |
---|
comment:17 by , 21 months ago
Description: | modified (diff) |
---|
The latest proposed patch is here: http://ffmpeg.org/pipermail/ffmpeg-devel/2023-January/306143.html
This obsoletes the earlier proposed patch.
comment:18 by , 21 months ago
Balling, btw I confirmed that this is not a duplicate of https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_EE0E40DE9A569FE5AB454E6E700A2DA79A08@qq.com/
Adding prints to h264_mp4toannexb_bsf.c confirms that it's not reached in the sequence that reproduces this bug. It must be unrelated.
Thanks.
comment:19 by , 21 months ago
V2 patch posted: http://ffmpeg.org/pipermail/ffmpeg-devel/2023-February/306555.html
comment:20 by , 21 months ago
Resolution: | → fixed |
---|---|
Status: | open → closed |
Fixed in e0cb89c354ea654bccc524a8a45321b93cb256d3.
Are there any plans for Youtube to change resolutions mid segment? You know, download other resolution segment, decode not real time to needed frame and then change to new video stream mid segment.
Because I do not like how you need to skip to get to new resolution.
Just TS segments, no?