Opened 5 months ago

Last modified 5 months ago

#9979 new enhancement

webvtt subtitles not properly detected in DASH manifest

Reported by: cookieguru Owned by:
Priority: wish Component: avformat
Version: git-master Keywords: dash webvtt
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Summary of the bug:

If an MPD manifest contains WebVTT subtitles FFmpeg will not properly recognize or decode them.

How to reproduce:
Given a MPD with webvtt files, ffmpeg does not recognize it as a subtitle stream and instead labels it as an unknown data stream:
ffprobe -hide_banner ""

Input #0, dash, from '':
  Duration: 00:42:00.00, start: 0.000000, bitrate: 0 kb/s
  Program 0
  Stream #0:0: Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 416x234 [SAR 1:1 DAR 16:9], 162 kb/s, 29.97 fps, 29.97 tbr, 60k tbn (default)
      variant_bitrate : 139000
      id              : video=139000
  (snip: more video streams)
  Stream #0:8(en): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 126 kb/s (default)
      variant_bitrate : 125000
      id              : audio_eng=125000
  Stream #0:9(en): Data: none (wvtt / 0x74747677), 0 kb/s (default)
      id              : 1
Unsupported codec with id 0 for input stream 9

In this example, stream 9 contains the subtitles but is marked as Data.

Here is the contents of the MPD file:

<?xml version="1.0"?>
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" profiles="urn:mpeg:dash:profile:isoff-live:2011,urn:com:dashif:dash264" type="static" mediaPresentationDuration="PT42M0.818300S" minBufferTime="PT10S">
	<Period id="1" duration="PT42M0.8183S">
		<!-- audio and video AdaptationSet removed for brevity -->
		<AdaptationSet mimeType="application/mp4" codecs="wvtt" startWithSAP="1" segmentAlignment="true" lang="en" contentType="text">
			<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
			<SegmentTemplate initialization="caption_uuid_init.mp4" media="caption_uuid_$Time$.m4s" startNumber="1" timescale="1000">
					<S t="0" d="30000" r="15"/>
					<S d="27273"/>
					<S d="30000" r="16"/>
					<S d="21498"/>
					<S d="30000" r="9"/>
					<S d="7941"/>
					<S d="30000" r="9"/>
					<S d="967"/>
					<S d="30000" r="17"/>
					<S d="15688"/>
					<S d="30000" r="9"/>
					<S d="9733"/>
			<Representation mimeType="application/mp4" bandwidth="256" codecs="wvtt" id="1"/>
		<EventStream schemeIdUri="urn:scte:scte35:2014:xml+bin" timescale="30000">
			<Event id="1" presentationTime="15218203" duration="0"/>
			<Event id="2" presentationTime="31163132" duration="0"/>
			<Event id="3" presentationTime="40401361" duration="0"/>
			<Event id="4" presentationTime="49430381" duration="0"/>
			<Event id="5" presentationTime="66101035" duration="0"/>

Note codecs="wvtt" located twice in this AdaptationSet.

If we manually download the contents of this stream:
ffmpeg -hide_banner -i "" -map 0:d -c copy -copy_unknown -f data raw.bin

We can see that it contains the subs, but note it is not a valid webVTT file (first byte is a NUL byte):

xxd raw.bin | head
00000000: 0000 005f 7674 7463 0000 0009 6964 656e  ..._vttc....iden
00000010: 3800 0000 4e70 6179 6c3c 693e 5b6e 6172  8...Npayl<i>[nar
00000020: 7261 746f 725d 2041 6e64 2061 2066 616e  rator] And a fan
00000030: 7461 7374 6963 616c 3c2f 693e 0a3c 693e  tastical</i>.<i>
00000040: 776f 726c 6420 6869 6464 656e 2069 6e20  world hidden in
00000050: 7468 6520 6a75 6e67 6c65 2e3c 2f69 3e00  the jungle.</i>.
00000060: 0000 5f76 7474 6300 0000 0969 6465 6e38  .._vttc....iden8
00000070: 0000 004e 7061 796c 3c69 3e5b 6e61 7272  ...Npayl<i>[narr
00000080: 6174 6f72 5d20 416e 6420 6120 6661 6e74  ator] And a fant
00000090: 6173 7469 6361 6c3c 2f69 3e0a 3c69 3e77  astical</i>.<i>w

The URL of the stream in question is time-limited (it's a signed URL with an expiration parameter) and thus is not included here. US-based users can probably find similar MPD files on but note that the site uses both HLS and DASH streaming (newer videos tend to be DASH; older videos never are).

ffmpeg version:

ffmpeg version N-108711-g3141dbb7ad-20221018 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 12.1.0 (crosstool-NG
configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20221018
libavutil      57. 39.101 / 57. 39.101
libavcodec     59. 51.100 / 59. 51.100
libavformat    59. 34.101 / 59. 34.101
libavdevice    59.  8.101 / 59.  8.101
libavfilter     8. 49.101 /  8. 49.101
libswscale      6.  8.112 /  6.  8.112
libswresample   4.  9.100 /  4.  9.100
libpostproc    56.  7.100 / 56.  7.100

Attachments (1)

raw.bin (60.8 KB ) - added by cookieguru 5 months ago.
raw vtt stream obtained from

Download all attachments as: .zip

Change History (4)

comment:1 by cookieguru, 5 months ago

Keywords: mpd added; mp removed

Possibly related: #9839

comment:2 by Carl Eugen Hoyos, 5 months ago

Please attach the subtitle file.

comment:3 by Carl Eugen Hoyos, 5 months ago

Component: undeterminedavformat
Keywords: DASH dashdec mpd removed
Priority: normalwish
Type: defectenhancement

by cookieguru, 5 months ago

Attachment: raw.bin added
Note: See TracTickets for help on using tickets.