HLS source with multiple variants is not optimally handled

[Andrew]

Currently when ffmpeg parses an HLS source it it performs a scan of all the available variants resulting in a very slow startup time. It also does not adapt to changes in network conditions to switch between the variants.

Both of these cases are some of purposes of the HLS protocol.

For startup the common convention with in formatting HLS master playlists is to make the first variant in the list the one that should be played first. So the reader should attempt to start playing this variant first without parsing all the variant playlists.

The reader should then maintain a buffer to detect how fast it is reading segments in the variant playlist and if the buffer is not filling fast enough to maintain playback for the current bitrate it should switch to a lower bitrate, if the low buffer continues then the reader should switch to a lower rate until it is as the lowest bitrate. If the buffer is full for a specified period then the reader should attempt to read the next higher bitrate and so on until the top rate is reached.

Note that the reader should also look at the bitrates recorded in the BANDWIDTH= field of the playlist as opposed to processing the segments as it should be assumed that the media honours the specification so as to reduce processing overhead and make for a faster experience.

A reasonable sample of a master playlist with multiple bitrate variants is available here: ​https://devimages.apple.com.edgekey.net/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8

If someone starts on this I am happy to provide more detail.

[Anssi Hannula]

I agree this is probably something that should be implemented, though it should be made optional (not yet sure if opt-in or opt-out) to allow manual selection if the user wants it.

I'll put this in my TODO, but I don't expect to have time for this in the near future, so others are welcome to try to implement this :)

[Andrew]

Replying to anssi:

I'll put this in my TODO, but I don't expect to have time for this in the near future, so others are welcome to try to implement this :)

Are you able to advise where in hls.c the switching between variants is currently handled?

[Anssi Hannula]

Replying to sinkers:

Are you able to advise where in hls.c the switching between variants is currently handled?

Currently, hls.c simply passes along all the variants as different AVPrograms to the user. The user can use AVDISCARD_ALL in AVStream->discard to temporarily disable retrieval of those variants that the user is not currently using (i.e. all but the currently playing one).

In an adaptive streaming mode, I'd expect that the demuxer will instead have to provide single "virtual" audio and video streams (plus alternative renditions), and seamlessly switch the actual backing audio/video streams (playlist->ctx->streams) that those streams are fed from by hls_read_packet().

[Andrew]

Thanks and can see that now. One simple option could be to take all the streams based on the BANDWIDTH param in the m3u8 (assuming it is available) and then order the streams in an array based on this.

Start playing the lowest bandwidth and if the state of the stream doesn't enter "buffering" (I can't see exactly where this is tracked but am sure it is there somewhere) then step up to the next highest stream until you reach the top. If the state then enters buffering it should step down a stream.

It should also be configurable how big the buffer that maintained is as well.

Some of what has been done in VLC could also be useful for this.
​http://git.videolan.org/?p=vlc.git;a=blob;f=modules/stream_filter/httplive.c;h=fa745af1052ff13c1f9dd22a088fffcc441ff904;hb=e6e09d2bf4f9f802aa27e3ca0cb75a3cc787e2cb

[Anssi Hannula]

It sounds simple, yes, but it may not be, as doing trickery like this one often encounters unforeseen issues to be solved.

Currently the hls demuxer does not maintain any buffers (except the current segment), the user is responsible for keeping enough buffer themselves. That may be changed if it is necessary or beneficial, of course.

[mav]

Hi, guys!

I am trying to implement the HLS adaptive streaming.

I am disabling unneeded variants with setting AVDISCARD_ALL in AVStream->discard.

But once I enable stream, I cannot disable it again back. The check in recheck_discard_flags() is performed only for the first packet.
Why is it there? Has it something to do with stream timestamps?

How can I disable fetching of live streams?

[Anssi Hannula]

@mav, You should be able to re-disable a stream by setting AVDISCARD_ALL again. read_data() checks discard status before requesting the download of a new segment.

This question probably belongs in the mailing lists, though...

[Andrew]

How are you defining what are unneeded variants?

If you like I am happy to contribute some kind of spec for this if you think it would help and also happy to assist with samples and testing.

Generally the behaviour for this should be as follows:

1. Parse the master playlist and extract all variants (including alternate audio and text)
2. Start with the lowest bitrate variant
3. Calculate what sort of bandwidth we have by measuring the download of the lowest variant. If the variant playlist doesn't include the bitrate against each stream then until the bitrates have been downloaded assume that they are in order e.g. lowest first
4. As quickly as possibly move to the variant that most closely matches the current bandwidth
5. Monitor the fill rate of the buffer to get an idea of current bandwidth and if required switch to a variant with a bandwidth closest to the observed bandwidth

Note that the dash.js client isn't a bad reference on handling the heuristics for ABR (but not the HLS part) and probably better than the VLC example above, the latest version is here: ​http://dashif.org/reference/players/javascript/1.2.0/index.html
and the code is here: ​https://github.com/Dash-Industry-Forum/dash.js/tree/development/app

For HLS parsing there is a good example (in Javascript) here: ​https://github.com/videojs/videojs-contrib-hls/tree/master/src

[mav]

My player will check buffer fill percentage every 30 seconds or so. If buffer fill > 90% it will switch to higher bitrate, if buffer fill < 20%, it will switch to lower bitrate.

The problem with ffmpeg currently - I cannot disable stream by setting AVDISCARD_ALL flag after I get packets from it, it still downloads segments from unneeded bitrates.

See the check in hls.c recheck_discard_flags() function. It checks for discard flag only on first packet in stream. After this - it just ignores discard flag.

[Anssi Hannula]

Disabling after first packet is done in read_data() as per comment:8, _not_ in recheck_discard_flags().

Since you don't seem to be working on fixing this ticket (this is about built-in adaptive streaming support in libavformat itself), to avoid cluttering the comments I suggest you post on ffmpeg-user@ instead or open a new ticket if you have found a bug :)

[Danny Zhao]

has anyone implemented the HLS adaptive streaming ?

[Zo2m4bie]

Hi. I made some work around this ticket. If you need you can use my implementation ​http://zo2m4bie.blogspot.com/2016/08/force-ffmpeg-to-work-with-hls-adaptive.html