Unable to read most of HLS M3U8 from Arte TV

[Thomas ERNEST]

Hello,

I am trying to improve Kodi plugin Arte+7. The improvement aims at reading videos from new Arte TV API instead of "old" Arte HBB TV API. Old content is full mp4 file. New content is HTTP Live Streaming (HLS) packaged in m3u8 files. Some m3u8 files can be read, some cannot.

How to reproduce:

% FAILURE
% ./ffplay.exe -v 9 -loglevel 99 https://arte-cmafhls.akamaized.net/am/cmaf/089000/089100/089171-000-A/230120220655/089171-000-A_VF-STF_XQ.m3u8 2> ko.log
...
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] type:'sidx' parent:'root' sz: 32248 847 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] sidx reference_type 1 is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] error reading header
[AVIOContext @ 00000228e9aae980] Statistics: 35459 bytes read, 0 seeks
...


% SUCCESS
% ./ffplay.exe -v 9 -loglevel 99 https://arte-cmafhls.akamaized.net/am/cmaf/105000/105600/105610-000-A/230111071325/105610-000-A_VOF-STF_XQ.m3u8 2> ok.log

I reproduced the erroneous and success cases with Kodi 19.5 Matrix and 20.0 Nexus. I reproduced the issue directly with ffplay also and attach the logs of ffplay command here.

EDIT : Both streams have WEBVTT SPRITES. But webvtt for successful playback is not found 404.
Failing one :

• in m3u8 #SPRITES: medias/089171-000-A_SPR.vtt
• deeplink ​https://arte-cmafhls.akamaized.net/am/cmaf/089000/089100/089171-000-A/230120220655/medias/089171-000-A_SPR.vtt
• content

  WEBVTT
  00:00:00.000 --> 00:00:06.000
  089171-000-A_SPR_000000.jpg#xywh=0,0,240,135
  ...
  

Successful one :

• in m3u8 #SPRITES: medias/105610-000-A_SPR.vtt
• deeplink ​https://arte-cmafhls.akamaized.net/am/cmaf/105000/105600/105610-000-A/230111071325/medias/105610-000-A_SPR.vtt
• content

  Not found
  

Link to webvtt related tickets :

• ​https://trac.ffmpeg.org/ticket/5067
• ​https://trac.ffmpeg.org/ticket/5881
• ​https://trac.ffmpeg.org/ticket/4048

Thank you in advance

[Thomas ERNEST]

Logs of failing case

[Thomas ERNEST]

Logs of a succesful case

[Thomas ERNEST]

failing m3u8 with hls and webvtt

[Thomas ERNEST]

valid m3u8 with hls and webvtt

[quinkblack]

sidx is the direct reason leading to the failure, but it's not the root cause.

1. Media initialization section byterange includes sidx. There is no benefical in HLS's case, although DASH can use sidx explicitly. How to interpret the sidx is a question, HLS spec doesn't help. If the byterange excluding sidx, the issue should be gone.

#EXT-X-MAP:URI="089171-000-A_v360.mp4",BYTERANGE="33087@0"

2. FFmpeg mp4 demux has no issue to handle the sidx. You can test it with

ffplay  'https://arte-cmafhls.akamaized.net/am/cmaf/089000/089100/089171-000-A/230120220655/medias/089171-000-A_v360.mp4'

The issue happens when playback fmp4 inside HLS. The log message "sidx reference_type 1 is not implemented" is misleading. It's some IO error leads to this error, the fmp4 doesn't use sidx reference_type 1.

I don't have much time to dig further now.

[zebul666]

there is also that ticket related to webvtt ​https://trac.ffmpeg.org/ticket/10169

I am able to play such erroneous case by downloading separately video mp4 stream, audio mp4 stream, and webvtt (by following each m3u8) (removing any STYLE blocks in webvtt like said in #10169) and combine/merge them into a file; for local reading.

[Thomas ERNEST]

Indeed the problem is not related to webvtt or subtitles. Thank you for your help quinkblack and zebul666.

The root cause is related to the multivariant playlist ​https://developer.apple.com/documentation/http-live-streaming/creating-a-multivariant-playlist or byterange addressing.
It is possible to build a set of new m3u to index streams without multivariant and byterange addressing so that FFMPEG can play it.

Nevertheless, there is still a problem with these new m3u : the content is not seekable. It is not possible to fast-forward, rewind or jump to a moment in video.

To reproduce:

1. GET ​https://www.arte.tv/api/rproxy/emac/v4/fr/tv/pages/HOME/
2. pick any programId from the response e.g. 020528-000-A
3. GET ​https://api.arte.tv/api/player/v2/config/fr/020528-000-A
4. Read URLs to m3u in data.attributes.streams.url

Notes : Each m3u has different combinaition audio and subtibles e.g. original version with french subtitles, german language without subtitles... m3u have the same multivariant video streams. Different audio stream and subtitles combinaison.

5. Pick a video stream and every audios streams. Create a m3u to encapsulate each stream.
6. Create a parent m3u to index and add metadata to stream m3u.

Notes : See m3u attached to the ticket as an example. m3u from step 5 and 6 follow the pattern in ​https://developer.apple.com/documentation/http-live-streaming/adding-alternate-media-to-a-playlist.

[Thomas ERNEST]

german audio stream m3u for program 20528

[Thomas ERNEST]

english audio stream m3u for program 20528

[Thomas ERNEST]

french audio stream m3u for program 20528

[Thomas ERNEST]

720p video stream m3u for program 20528

[Thomas ERNEST]

main parent m3u for program 20528

[Thomas ERNEST]

ffplay logs playing 20528 m3u set - not seekable

[Thomas ERNEST]

If I add in the m3u8 files #EXT-X-TARGETDURATION:6360 - 6360 being content duration in seconds - then I have the ability to seek the playlist with ffplay 6.0, but not with Kodi 20.1 (integrating ffplay 4.4.1 as far as I recall).

• #EXT-X-TARGETDURATION:6360

Please find in attachements a set of m3u8 which I can navigate through.

I added other fields after a first successfull test as far as I remember. I guess they don't influence ffplay to seek or not the playlist :

• #EXT-X-PLAYLIST-TYPE:VOD
• #EXT-X-MEDIA-SEQUENCE:0
• #EXT-X-ENDLIST

[Thomas ERNEST]

seekable example - german audio stream

[Thomas ERNEST]

seekable example - english audio stream

[Thomas ERNEST]

seekable example - french audio stream

[Thomas ERNEST]

seekable example - video stream

[Thomas ERNEST]

seekable example - main content index

[pedrojvpedrosa]

I have been getting this error on some of my HLS streams. I noticed it only happens on streams that are longer than around 8000 seconds (or 2666 segments).

Initially I thought this could be an issue with unsigned 32-bit integer overflow of the byterange addresses, but it also happens on the lighter variants such as 432p which don't have addresses that overflow a 32bit unsigned int.