hls demuxer does not handle pid changes after EXT-X-DISCONTINUITY

[Aman]

Summary of the bug:
An HLS playlist can contain EXT-X-DISCONTINUITY tags. The segment after such a tag may contain a new PMT, which may contain new elementary streams on previously used pids.

For example, this m3u8:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-INDEPENDENT-SEGMENTS
#EXTINF:7.074,
segment_156270742.ts
#EXT-X-DISCONTINUITY
#EXTINF:10.101,
segment-1.ts
#EXTINF:10.008,
segment-2.ts
#EXTINF:10.194,
segment-3.ts
#EXT-X-DISCONTINUITY
#EXTINF:10.010,
segment_156270745.ts
#EXT-X-ENDLIST

where segment_156270742.ts before the X-DISCONTINUITY has streams as follows:

  Program 1
    Stream #0:0[0x101]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080, Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x102]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 93 kb/s
    Stream #0:2[0x103]: Data: timed_id3 (ID3  / 0x20334449)

and segment-1.ts after the X-DISCONTINUITY has streams:

  Program 1
    Stream #0:0[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 69 kb/s
    Stream #0:1[0x102]: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc

Before the discontinuity, 0x101 is a video PID but afterwards it is an audio PID.

ffmpeg does not detect this PMT change, and feeds audio packets into the video decoder and vice versa.

How to reproduce:

% ffmpeg -i https://tmm1.s3.amazonaws.com/hls-pidswap/test.m3u8 -f null -y /dev/null
ffmpeg version git-2019-02-13-bf78aa9 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/HEAD-bf78aa9 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 47.100 / 58. 47.100
  libavformat    58. 26.101 / 58. 26.101
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
[hls,applehttp @ 0x7f9dd0800000] Opening 'https://tmm1.s3.amazonaws.com/hls-pidswap/segment_156270742.ts' for reading
[hls,applehttp @ 0x7f9dd0800000] Opening 'https://tmm1.s3.amazonaws.com/hls-pidswap/segment-1.ts' for reading
Input #0, hls,applehttp, from 'https://tmm1.s3.amazonaws.com/hls-pidswap/test.m3u8':
  Duration: 00:00:47.39, start: 44058.812000, bitrate: 0 kb/s
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1920x1080, Closed Captions, 29.97 tbr, 90k tbn, 59.94 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
    Stream #0:2: Data: timed_id3 (ID3  / 0x20334449)
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
  Metadata:
    encoder         : Lavf58.26.101
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080, q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn, 29.97 tbc
    Metadata:
      variant_bitrate : 0
      encoder         : Lavc58.47.100 wrapped_avframe
    Stream #0:1: Audio: pcm_s16le, 44100 Hz, stereo, s16, 1411 kb/s
    Metadata:
      variant_bitrate : 0
      encoder         : Lavc58.47.100 pcm_s16le
[https @ 0x7f9dcf01b800] Opening 'https://tmm1.s3.amazonaws.com/hls-pidswap/segment-2.ts' for reading
[aac @ 0x7f9dd103a000] channel element 1.0 is not allocated
Error while decoding stream #0:1: Invalid data found when processing input
[aac @ 0x7f9dd103a000] More than one AAC RDB per ADTS frame is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x7f9dd103a000] Multiple frames in a packet.
[aac @ 0x7f9dd103a000] Number of bands (45) exceeds limit (32).
Error while decoding stream #0:1: Invalid data found when processing input
[aac @ 0x7f9dd103a000] Prediction is not allowed in AAC-LC.
Error while decoding stream #0:1: Invalid data found when processing input
[aac @ 0x7f9dd103a000] Reserved bit set.
[aac @ 0x7f9dd103a000] invalid band type
Error while decoding stream #0:1: Invalid data found when processing input
[aac @ 0x7f9dd103a000] Sample rate index in program config element does not match the sample rate index configured by the container.
[aac @ 0x7f9dd103a000] Inconsistent channel configuration.
[aac @ 0x7f9dd103a000] get_buffer() failed
Error while decoding stream #0:1: Invalid argument
[aac @ 0x7f9dd103a000] channel element 1.2 is not allocated
Error while decoding stream #0:1: Invalid data found when processing input
[aac @ 0x7f9dd103a000] Number of bands (23) exceeds limit (16).
Error while decoding stream #0:1: Invalid data found when processing input
[aac @ 0x7f9dd103a000] Reserved bit set.
[aac @ 0x7f9dd103a000] Number of bands (37) exceeds limit (19).
Error while decoding stream #0:1: Invalid data found when processing input

[Aman]

Note that popular HLS players such as Safari, Quicktime, hls.js etc have no problem with this playlist.

[Jun Zhao]

duplicate with ​http://trac.ffmpeg.org/ticket/6737

[Marton Balint]

How does a compliant player match the streams? E.g. what if two aac audio streams are in the PMT, and they swap pids?

[Vraz]

Ran into a variation of this while adding OTT live playback (like Pluto) to my player using 4.3.2 libs. During playback, the hls demuxer concatenates segments and passes to the subdemuxer as continuous media. At least for mpegts, concatenating segments is only valid for media that was mastered together. mpegts does not support the arbitrary concatenation of random segments and that is what happens at a discontinuity.

If you crack the log level up, you will likely see:
"Continuity check failed for pid %d expected %d got %d" at transitions of non-related segments.
You can find the corresponding code in mpegts.c by searching on the ironically wrong comment: /* continuity check (currently not used) */ (which is very much used). Once the corruption occurs, all kinds of "bad things" happen to the streams. Most common seems to be -1 stream types that eventually overflow the stream table.

The workaround is changing hls to advertise segment boundaries to mpegts so the latter can prepare for the change. In my experiments, making mpegts detect a seek between discontinuous segments solved the issue. My proof of concept triggers the mpegts seek code between every segment though don't think that is fully hls compliant.

In hls.c/read_data:

  ret = read_from_url(v, seg, buf, buf_size);
  if (ret > 0) { ... }
+ if (avio_feof(v->input))
+   v->ctx->io_repositioned = (v->pb.pos+ret) & 0x7fffffff;

In mpegts.c/handle_packets:

    ts->last_pos = avio_tell(s->pb);
+   if (s->io_repositioned == (ts->last_pos & 0x7fffffff))
+     ts->last_pos = 0;
    return ret;

Reusing AVFormatContext::io_repositioned required stripping the stream position to 31-bits. With a couple extra lines of code, setting the advertisement could be conditional on a prior #EXT-X-DISCONTINUITY to be more standards compliant. Your app must disable AVFormatContext::ts_overflow (else it will trigger improperly) and the app must performs its own DTS/PTS resequencing.

Also, additional streams will get added during playback. After a couple hours of Pluto and its commercials, there were 16 h264 streams, 1 aac and 1 timed_id3. The "merge_pmt_versions" might help, though its non-trivial to set as hls does not pass options down to the subdemuxer.

[Vraz]

Before the discontinuity, 0x101 is a video PID but afterwards it is an audio PID.

ffmpeg does not detect this PMT change, and feeds audio packets into the video decoder and vice versa.

Enabled merge_pmt_versions within hls.c, and instead of preventing extra streams, it duplicates the above issue. The problem is mpegts.c::find_matching_stream() is super simplistic and unclear if it ever does anything useful. It fails to verify the stream-type prior to merging thus often resulting in damaged streams.

Changed find_matching_stream() to match on program_num and stream_type and that seemed to resolve it. Might have other side-effects, but OTT Pluto worked well with limited testing.

My player needs to work with OS distributed builds so hooked the read-packet calls to trick mpegts into seek-after-segment. Will leave it to others to decide whether updating find_matching_streams makes sense. (Not possible to retrofit that behavior.)