WebVTT decoder breaks when ::cue elements are used

[derrod]

Summary of the bug: WebVTT input is not read past ::cue tags in input file.
How to reproduce:

% ffmpeg -y -v 9 -loglevel 99 -i test_input.vtt test_output.ass
ffmpeg version git-2020-05-25-6268034 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200523
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 49.100 / 56. 49.100
  libavcodec     58. 87.101 / 58. 87.101
  libavformat    58. 43.100 / 58. 43.100
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 83.100 /  7. 83.100
  libswscale      5.  6.101 /  5.  6.101
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Splitting the commandline.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input url with argument 'test_input.vtt'.
Reading option 'test_output.ass' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input url test_input.vtt.
Successfully parsed a group of options.
Opening an input file: test_input.vtt.
[NULL @ 0000025fa3a9ccc0] Opening 'test_input.vtt' for reading
[file @ 0000025fa3a9dd40] Setting default whitelist 'file,crypto,data'
Probing webvtt score:100 size:139
[webvtt @ 0000025fa3a9ccc0] Format webvtt probed with size=2048 and score=100
[webvtt @ 0000025fa3a9ccc0] Before avformat_find_stream_info() pos: 44 bytes read:139 seeks:0 nb_streams:1
[webvtt @ 0000025fa3a9ccc0] All info found
[webvtt @ 0000025fa3a9ccc0] stream 0: start_time: -9223372036854776.000 duration: -9223372036854776.000
[webvtt @ 0000025fa3a9ccc0] format: start_time: -9223372036854.775 duration: -9223372036854.775 (estimate from bit rate) bitrate=0 kb/s
[webvtt @ 0000025fa3a9ccc0] After avformat_find_stream_info() pos: 44 bytes read:139 seeks:0 frames:0
Input #0, webvtt, from 'test_input.vtt':
  Duration: N/A, bitrate: N/A
    Stream #0:0, 0, 1/1000: Subtitle: webvtt
Successfully opened the file.
Parsing a group of options: output url test_output.ass.
Successfully parsed a group of options.
Opening an output file: test_output.ass.
[file @ 0000025fa3aa1b80] Setting default whitelist 'file,crypto,data'
Successfully opened the file.
Output #0, ass, to 'test_output.ass':
  Metadata:
    encoder         : Lavf58.43.100
    Stream #0:0, 0, 1/100: Subtitle: ass (ssa)
    Metadata:
      encoder         : Lavc58.87.101 ssa
Stream mapping:
  Stream #0:0 -> #0:0 (webvtt (native) -> ass (ssa))
Press [q] to stop, [?] for help
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
No more output streams to write to, finishing.
size=       1kB time=00:00:00.00 bitrate=N/A speed=   0x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:1kB muxing overhead: unknown
Input file #0 (test_input.vtt):
  Input stream #0:0 (subtitle): 0 packets read (0 bytes); 0 frames decoded;
  Total: 0 packets (0 bytes) demuxed
Output file #0 (test_output.ass):
  Output stream #0:0 (subtitle): 0 frames encoded; 0 packets muxed (0 bytes);
  Total: 0 packets (0 bytes) muxed
0 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0000025fa3aa2d40] Statistics: 0 seeks, 1 writeouts
[AVIOContext @ 0000025fa3aa5fc0] Statistics: 139 bytes read, 0 seeks

The entire content of the input file is as follows (also attached):

WEBVTT

::cue(.white){ color: white; }

1
00:00:01.000 --> 00:00:05.000 line:77% position:50% align:middle
<c.white>Hello World</c>

The output file only contains the boilerplate ASS information and no events. (see attached file).

Removing the line(s) starting with ::cue will result in a successfully converted output (albeit without the styling).

Note that even a codec copy from WebVTT to WebVTT will fail and produce an output file that contains only the "WEBVTT" header (other attached file).

[derrod]

sample input file

[derrod]

sample ASS output

[derrod]

sample stream copy output

[Carl Eugen Hoyos]

Am I correct that your input file is not valid webvtt?

[derrod]

The file was created by an external source and indeed appears to be missing the "STYLE" keyword of the style block (​https://www.w3.org/TR/webvtt1/#webvtt-style-block). However, even if the STYLE keyword is added the file is not parsed correctly and the output remains empty.

At least Chrome still appears to render the output correctly however

[derrod]

test input with STYLE keyword

[tpikonen]

There is patch fixing this issue here: ​http://ffmpeg.org/pipermail/ffmpeg-devel/2022-May/296353.html

[tpikonen]

More patches fixing this are here: ​https://patchwork.ffmpeg.org/project/ffmpeg/list/?q=webvttdec

This one even claims to do the actual parsing:
​https://patchwork.ffmpeg.org/project/ffmpeg/patch/CA+_k4RJ1CNriPbhxxzLf1O6V1VuP2OXch3E8LqOsMNSRUyEnTw@mail.gmail.com/

[logv]

Note to add that this issue currently seems to break processing of all recently produced VTT subtitles served by SVT Play (<​https://play.svt.se>). I'm not sure if this is perhaps considered malformed, or if it is something ffmpeg optimally should be able to handle. When trying to convert subtitle format to SRT or ASS, ffmpeg fails silently and produces a 0 byte file (SRT) or just boilerplate (ASS).

E.g., a typical start of a VTT from SVT Play:

WEBVTT

STYLE
::cue {
    color: #DFDFDF;
}
::cue(.huvudpratare) {
    color: #00FFFF;
}
::cue(.ljud) {
    color: #FF00FF;
}
::cue(.ljudskylt) {
    color: #FF00FF;
}
::cue(.relivemark) {
    color: #FF0000;
}
::cue(.sang) {
    color: #00FF00;
}

5da33626-c8a3-48d2-b40a-d5878ea108fa
00:01:34.320 --> 00:01:39.160 align:left position:18%
<c.teletext>Han kommer senare.
Vi kan inte göra nånting åt det nu.</c>

Another (world accessible) example lives here (for a limited time): ​https://ed4.cdn.svt.se/d0/world/20220811/b9e3c11c-6cdd-46b7-8947-228492d686a5/text/text-0.vtt (also attached)

[logv]

Replying to tpikonen:

There is patch fixing this issue here: ​http://ffmpeg.org/pipermail/ffmpeg-devel/2022-May/296353.html

I can verify that this patch seems to fix the problem for me.

[varenc]

Any hope of getting that patch to webvttdec merged? I can confirm it fixes this issue!

In the meantime here's a hacky solution that helped me workaround the bug by removing the styling:

 ffmpeg -i <(cat styled-subs.vtt | sd '^STYLE\n::cue\s*\{[^}]*\}' '') out.srt