Opened 9 years ago

Last modified 2 years ago

#4048 new enhancement

segment: Handle WebVTT as per HLS spec

Reported by: Ridley Combs Owned by:
Priority: normal Component: avformat
Version: git-master Keywords: hls webvtt
Cc: mkuron Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

As per the HLS spec:

   Subtitle segments MUST be formatted as WebVTT [WebVTT] files.  Each
   subtitle segment MUST contain all subtitle cues that are intended to
   be displayed during the period indicated by the segment EXTINF
   duration.  The start time offset and end time offset of each cue MUST
   indicate the total display time for that cue, even if that time range
   extends beyond the EXTINF duration.  A WebVTT segment MAY contain no
   cues; this indicates that no subtitles are to be displayed during
   that period.

This means that the segment (or HLS) muxer needs to handle two particular cases, when muxing WebVTT for HLS:

  1. No frames exist during a segment's duration, so a blank segment should be written.
  2. A frame spans two or more different segments, so it should be written in all of them.

This behavior could be triggered by an option, or automatically when the underlying format is WebVTT.

Relatedly, the WebVTT muxer should provide an option to write the X-TIMESTAMP-MAP header for HLS synchronization.

Change History (6)

comment:1 by Carl Eugen Hoyos, 9 years ago

Iiuc, there is a bug: Our hls muxer does not conform to the specification and produces output that is incorrect.
Is that correct? If yes, please explain how this can be reproduced. (You know what I mean with explain?)
I know that you understand the ticket and I realize that whoever will fix this already understands the issue (and possibly already knows how to fix it). but ideally every ticket should be reproducible for every user. This is particularly important in the future: The commit fixing the ticket might (in theory) introduce another problem and whoever wants to fix it may not know so much about hls...
Or said differently: It is great that you added a link to the specification and quoted the relevant paragraph, it will make somebody's life much easier. But as said previously this analysis (no matter if helpful like in this case or disturbing like in other tickets not created by you) should never replace the actual bug report.

comment:2 by Carl Eugen Hoyos, 9 years ago

Keywords: hls webvtt added

comment:3 by Ridley Combs, 9 years ago

Keywords: hls webvtt removed

It's not so much a bug as the lack of support for a newly-specified behavior. But let's use this test.vtt file as an example:

00:00.000 --> 00:07.000
Test line 1, spans segments 0 and 1

00:07.000 --> 00:08.000
Test line 2, contained in segment 1

00:16.000 --> 00:18.000
Test line 3, should be in segment 3 after an empty segment 2

00:18.000 --> 00:40.000
Test line 4, should span several segments

Let's run ffmpeg -i in.vtt -f segment -segment_time 5 -segment_format webvtt -scodec copy out-%05d.vtt.

We get:
out-00000.vtt

WEBVTT

00:00.000 --> 00:07.000
Test line 1, spans segments 0 and 1

out-00001.vtt

WEBVTT

00:07.000 --> 00:08.000
Test line 2, contained in segment 1

out-00002.vtt

WEBVTT

00:16.000 --> 00:18.000
Test line 3, should be in segment 3 after an empty segment 2

out-00003.vtt

WEBVTT

00:18.000 --> 00:40.000
Test line 4, should span several segments

The correct output for HLS would be:
out-00000.vtt

WEBVTT

00:00.000 --> 00:07.000
Test line 1, spans segments 0 and 1

out-00001.vtt

WEBVTT

00:00.000 --> 00:07.000
Test line 1, spans segments 0 and 1

00:07.000 --> 00:08.000
Test line 2, contained in segment 1

out-00002.vtt

WEBVTT

out-00003.vtt

WEBVTT

00:18.000 --> 00:40.000
Test line 4, should span several segments

out-00004.vtt

WEBVTT

00:18.000 --> 00:40.000
Test line 4, should span several segments

<etc…, with more duplicates up to out-00007.vtt>

The two behaviors that need adding (writing empty segments, and duplicating packets spanning multiple segments) seem like they could reasonably be separate options. The writing-empty-segments behavior seems sensible for generic subtitle segmenting (in order to have predictable segment durations), whereas duplicating packets is pretty WebVTT-in-HLS-specific.

comment:4 by Ridley Combs, 9 years ago

Keywords: hls webvtt added

comment:5 by Djfe, 7 years ago

I guess this kinda belongs to this ticket:
https://amd.cdn.turner.com/adultswim/episodes/no_uk_geo/bobsburgers/2017/03/bobsburgers_cc_505_Digital_IttyBittyDittyCommittee/hls/bobsburgers_cc_505_Digital_IttyBittyDittyCommittee_stream_webvtt.m3u8

is the webvtt Stream (subtitle only stream) related to https://amd.cdn.turner.com/adultswim/episodes/no_uk_geo/bobsburgers/2017/03/bobsburgers_cc_505_Digital_IttyBittyDittyCommittee/hls/bobsburgers_cc_505_Digital_IttyBittyDittyCommittee_stream_full.m3u8

from http://www.adultswim.com/videos/bobs-burgers/the-itty-bitty-ditty-committee/

Though I'm not sure. If this is only an issue about the muxer but not the demuxer, then say so and I'll open a new ticket.

FFmpeg only saves the first segment instead of merging all segments together, like it's described in this ticket.
ffmpeg -i url -c copy example.vtt

Since the m3u8 won't be up forever, here's the content of it:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:300
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:300.299,
bobsburgers_cc_505_Digital_IttyBittyDittyCommittee_stream_webvttvtt_00001.vtt
#EXTINF:300.300,
bobsburgers_cc_505_Digital_IttyBittyDittyCommittee_stream_webvttvtt_00002.vtt
#EXTINF:300.300,
bobsburgers_cc_505_Digital_IttyBittyDittyCommittee_stream_webvttvtt_00003.vtt
#EXTINF:300.300,
bobsburgers_cc_505_Digital_IttyBittyDittyCommittee_stream_webvttvtt_00004.vtt
#EXTINF:104.104,
bobsburgers_cc_505_Digital_IttyBittyDittyCommittee_stream_webvttvtt_00005.vtt
#EXT-X-ENDLIST

FFmpeg only saves the first 5mins, so I guess that's the segment time.

comment:6 by mkuron, 2 years ago

Cc: mkuron added
Note: See TracTickets for help on using tickets.