Unable to parse some subtitles with a system ffmpeg compiled HandBrake

[Simone Caronni]

Summary of the bug: Different behaviours with subtitles in HandBrake compiled with libav vs ffmpeg.

As part of my daily usage, I've seen that HandBrake compiled with a system ffmpeg (unsupported by HandBrake) gives complete feature parity except for the parsing of UTF-8 subtitles in movie streams. Since Fedora defaults to ffmpeg, I'm trying to make it work with that.

How to reproduce: Build HandBrake with the following patch applied:

​https://github.com/negativo17/HandBrake/blob/master/cf1571f3bf314638a608784f19f80edb736e8144.patch

To allow building HandBrake with a system installed ffmpeg and not its own bundled copy of libav. Behaviour is then different. I'm sure that there is something in the way that HandBrake uses libav for parsing the subtitles. Unfortunately the HandBrake developers are not interested in making the program work with anything but their libav bundle; so maybe someone here can shed some light.

Importing a movie which has UTF-8 subtitles in a system ffmpeg compiled HandBrake, result in the subtitle stream not being available (like if it wasn't there):

[10:51:05] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/home/slaanesh/Downloads/convert/Macross - Full/1.1 Macross II/Macross II Lovers Again/Macross II Lovers Again.mkv':
  Metadata:
    title           : Macross II: Lovers Again - Part 1
    encoder         : libebml-0.7.5 & libmatroska-0.7.7
    creation_time   : 2010-09-01T18:03:39.000000Z
  Duration: 02:17:03.90, start: 0.000000, bitrate: 1393 kb/s
    Stream #0:0(Macross II: Lovers Again - Part 1): Video: mpeg4 (DX50 / 0x30355844), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 23.98 fps, 23.98 tbr, 1k tbn, 30k tbc (default)
    Metadata:
      SUBJECT         : www.anime-legion.net
      COMPOSER        : Nicholi
      COPYRIGHT       : #Anime-Legion@mIRCX.com
      LANGUAGE        : Macross II: Lovers Again - Part 1
    Stream #0:1(eng): Audio: vorbis, 48000 Hz, stereo, fltp (default)
    Stream #0:2(jpn): Audio: vorbis, 48000 Hz, stereo, fltp
    Stream #0:3(eng): Subtitle: subrip (default)
[10:51:05] add_ffmpeg_subtitle: unknown subtitle stream type: 0x17808

Doing the same with a bundled libav compiled HandBrake allows the subtitle stream to be parsed correctly:

[10:29:12] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/home/slaanesh/Downloads/convert/Macross - Full/1.1 Macross II/Macross II Lovers Again/Macross II Lovers Again.mkv':
  Metadata:
    title           : Macross II: Lovers Again - Part 1
  Duration: 02:17:03.89, start: 0.000000, bitrate: N/A
    Stream #0:0(Macross II: Lovers Again - Part 1): Video: mpeg4 (Simple Profile) [DX50 / 0x30355844]
      yuv420p, 640x480 [PAR 1:1 DAR 4:3], PAR 1:1 DAR 4:3
      23.98 fps, 1k tbn (default)
    Metadata:
      SUBJECT         : www.anime-legion.net
      COMPOSER        : Nicholi
      COPYRIGHT       : #Anime-Legion@mIRCX.com
      LANGUAGE        : Macross II: Lovers Again - Part 1
    Stream #0:1(eng): Audio: vorbis
      48000 Hz, stereo, fltp (default)
    Stream #0:2(jpn): Audio: vorbis
      48000 Hz, stereo, fltp
    Stream #0:3(eng): Subtitle: srt (default)

If I convert the video with the system ffmpeg skipping HandBrake the subtitle stream is manipulated correctly:

$ ffmpeg -v verbose -i Macross\ II\ Lovers\ Again.mkv -strict experimental -ac 2 -c:a aac -c:v hevc_nvenc -preset medium output.mkvffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.1 (GCC) 20161221 (Red Hat 6.3.1-1)
  configuration: --arch=x86_64 --bindir=/usr/bin --datadir=/usr/share/ffmpeg --disable-debug --disable-static --disable-stripping --enable-avfilter --enable-avresample --enable-bzlib --enable-cuda --enable-cuvid --enable-libnpp --enable-doc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gpl --enable-iconv --enable-libass --enable-libbluray --enable-libcdio --enable-libdc1394 --enable-libebur128 --enable-libfdk-aac --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libkvazaar --enable-libmfx --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libzvbi --enable-lzma --enable-nonfree --enable-openal --enable-opencl --enable-nvenc --enable-opengl --enable-postproc --enable-pthreads --enable-sdl2 --enable-shared --enable-version3 --enable-x11grab --enable-xlib --enable-zlib --extra-cflags='-I/usr/include/nvenc -I/usr/include/cuda' --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --optflags='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --prefix=/usr --shlibdir=/usr/lib64 --enable-runtime-cpudetect
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Routing option strict to both codec and muxer layer
Input #0, matroska,webm, from 'Macross II Lovers Again.mkv':
  Metadata:
    title           : Macross II: Lovers Again - Part 1
    encoder         : libebml-0.7.5 & libmatroska-0.7.7
    creation_time   : 2010-09-01T18:03:39.000000Z
  Duration: 02:17:03.90, start: 0.000000, bitrate: 1393 kb/s
    Stream #0:0(Macross II: Lovers Again - Part 1): Video: mpeg4, 1 reference frame (DX50 / 0x30355844), yuv420p(left), 640x480 [SAR 1:1 DAR 4:3], 23.98 fps, 23.98 tbr, 1k tbn, 30k tbc (default)
    Metadata:
      SUBJECT         : www.anime-legion.net
      COMPOSER        : Nicholi
      COPYRIGHT       : #Anime-Legion@mIRCX.com
      LANGUAGE        : Macross II: Lovers Again - Part 1
    Stream #0:1(eng): Audio: vorbis, 48000 Hz, stereo, fltp (default)
    Stream #0:2(jpn): Audio: vorbis, 48000 Hz, stereo, fltp
    Stream #0:3(eng): Subtitle: subrip (default)
[graph 0 input from stream 0:0 @ 0x87a740] w:640 h:480 pixfmt:yuv420p tb:1/1000 fr:13978/583 sar:1/1 sws_param:flags=2
[graph 1 input from stream 0:1 @ 0x8a4520] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
[hevc_nvenc @ 0x8b80e0] Loaded Nvenc version 8.0
[hevc_nvenc @ 0x8b80e0] Nvenc initialized successfully
[hevc_nvenc @ 0x8b80e0] 1 CUDA capable devices found
[hevc_nvenc @ 0x8b80e0] [ GPU #0 - < GeForce GTX 1060 6GB > has Compute SM 6.1 ]
[hevc_nvenc @ 0x8b80e0] supports NVENC
[hevc_nvenc @ 0x8b80e0] Lookahead enabled: depth 0, scenecut enabled, B-adapt disabled.
Output #0, matroska, to 'output.mkv':
  Metadata:
    title           : Macross II: Lovers Again - Part 1
    encoder         : Lavf57.56.101
    Stream #0:0(Macross II: Lovers Again - Part 1): Video: hevc (hevc_nvenc) (Main), 1 reference frame, yuv420p(left), 640x480 [SAR 1:1 DAR 4:3], q=-1--1, 2000 kb/s, 23.98 fps, 1k tbn, 23.98 tbc (default)
    Metadata:
      SUBJECT         : www.anime-legion.net
      COMPOSER        : Nicholi
      COPYRIGHT       : #Anime-Legion@mIRCX.com
      LANGUAGE        : Macross II: Lovers Again - Part 1
      encoder         : Lavc57.64.101 hevc_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
    Stream #0:1(eng): Audio: aac (LC) ([255][0][0][0] / 0x00FF), 48000 Hz, stereo, fltp, delay 1024, 128 kb/s (default)
    Metadata:
      LANGUAGE        : eng
      encoder         : Lavc57.64.101 aac
    Stream #0:2(eng): Subtitle: ass (ssa) (default)
    Metadata:
      LANGUAGE        : eng
      encoder         : Lavc57.64.101 ssa
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> hevc (hevc_nvenc))
  Stream #0:1 -> #0:1 (vorbis (native) -> aac (native))
  Stream #0:3 -> #0:2 (subrip (srt) -> ass (ssa))
Press [q] to stop, [?] for help
[mpeg4 @ 0x8a8760] Video uses a non-standard and wasteful way to store B-frames ('packed B-frames'). Consider using the mpeg4_unpack_bframes bitstream filter without encoding but stream copy to fix it.
No more output streams to write to, finishing.e=02:17:03.15 bitrate=2128.7kbits/s speed=21.7x     
frame=197176 fps=519 q=9.0 Lsize= 2140250kB time=02:17:03.90 bitrate=2131.9kbits/s speed=21.7x    
video:2006247kB audio:129778kB subtitle:70kB other streams:0kB global headers:1kB muxing overhead: 0.194519%
Input file #0 (Macross II Lovers Again.mkv):
  Input stream #0:0 (video): 197176 packets read (1213717124 bytes); 197176 frames decoded; 
  Input stream #0:1 (audio): 504330 packets read (107663547 bytes); 504329 frames decoded (394747584 samples); 
  Input stream #0:2 (audio): 1 packets read (1 bytes); 
  Input stream #0:3 (subtitle): 1174 packets read (46193 bytes); 1174 frames decoded; 
  Total: 702681 packets (1321426865 bytes) demuxed
Output file #0 (output.mkv):
  Output stream #0:0 (video): 197176 frames encoded; 197176 packets muxed (2054397078 bytes); 
  Output stream #0:1 (audio): 385496 frames encoded (394747584 samples); 385497 packets muxed (132892434 bytes); 
  Output stream #0:2 (subtitle): 1174 frames encoded; 1174 packets muxed (71835 bytes); 
  Total: 583847 packets (2187361347 bytes) muxed
[hevc_nvenc @ 0x8b80e0] Nvenc unloaded
[aac @ 0x8b8cc0] Qavg: 593.729

Again, putting it into the HandBrake with bundled libav:

[10:53:23] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/home/slaanesh/Downloads/convert/Macross - Full/1.1 Macross II/Macross II Lovers Again/output.mkv':
  Metadata:
    title           : Macross II: Lovers Again - Part 1
    ENCODER         : Lavf57.56.101
  Duration: 02:17:03.95, start: 0.000000, bitrate: N/A
    Stream #0:0(Macross II: Lovers Again - Part 1): Video: hevc (Main)
      yuv420p, tv, 640x480 [PAR 1:1 DAR 4:3], PAR 1:1 DAR 4:3
      23.98 fps, 1k tbn (default)
    Metadata:
      SUBJECT         : www.anime-legion.net
      COMPOSER        : Nicholi
      COPYRIGHT       : #Anime-Legion@mIRCX.com
      ENCODER         : Lavc57.64.101 hevc_nvenc
      DURATION        : 02:17:03.956000000
    Stream #0:1(eng): Audio: aac (LC)
      48000 Hz, stereo, fltp (default)
    Metadata:
      ENCODER         : Lavc57.64.101 aac
      DURATION        : 02:17:03.917000000
    Stream #0:2(eng): Subtitle: ssa (default)
    Metadata:
      ENCODER         : Lavc57.64.101 ssa
      DURATION        : 02:13:59.346000000

But then again, if I import the movie to an HandBrake compiled with a system ffmpeg, the result is similar and the subtitle stream is not parsed correctly:

[11:00:59] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/home/slaanesh/Downloads/convert/Macross - Full/1.1 Macross II/Macross II Lovers Again/output.mkv':
  Metadata:
    title           : Macross II: Lovers Again - Part 1
    ENCODER         : Lavf57.56.101
  Duration: 02:17:03.96, start: 0.000000, bitrate: 2131 kb/s
    Stream #0:0(Macross II: Lovers Again - Part 1): Video: hevc (Main), yuv420p(tv, progressive), 640x480 [SAR 1:1 DAR 4:3], 23.98 fps, 23.98 tbr, 1k tbn, 23.98 tbc (default)
    Metadata:
      SUBJECT         : www.anime-legion.net
      COMPOSER        : Nicholi
      COPYRIGHT       : #Anime-Legion@mIRCX.com
      ENCODER         : Lavc57.64.101 hevc_nvenc
      DURATION        : 02:17:03.956000000
    Stream #0:1(eng): Audio: aac (LC), 48000 Hz, stereo, fltp (default)
    Metadata:
      ENCODER         : Lavc57.64.101 aac
      DURATION        : 02:17:03.917000000
    Stream #0:2(eng): Subtitle: ass (default)
    Metadata:
      ENCODER         : Lavc57.64.101 ssa
      DURATION        : 02:13:59.346000000
[11:00:59] add_ffmpeg_subtitle: unknown subtitle stream type: 0x1780d

Thanks & regards,
--Simone

PS: I've remove the chapters from above output to reduce a bit the information pasted.

[Carl Eugen Hoyos]

Replying to slaanesh:

Summary of the bug: Different behaviours with subtitles in HandBrake compiled with libav vs ffmpeg.

avconv contains several hundred known bugs that are not present in FFmpeg, some of them subtitle-related. Handbrake contains work-arounds for many of those bugs (the Handbrake developers know of these bugs but prefer to use avconv, they are currently very active in fixing some of the bugs in avconv). I suspect you have to remove some of the work-arounds to make subtitle handling work correctly.

[10:51:05] add_ffmpeg_subtitle: unknown subtitle stream type: 0x17808

For this issue, I suspect you have to map AV_CODEC_ID_SUBRIP against Handbrake's internal subtitle codec.

[Carl Eugen Hoyos]

I consider this rather important, it is needed for all distributors...

[Simone Caronni]

Replying to cehoyos:

I consider this rather important, it is needed for all distributors...

True, but it's not an ffmpeg issue but a HandBrake problem, as the developers do not seem to be interested in adding support for it. I just posted here seeking some advice, which very kindly I got. Thank you :)

I've seen that we would need to patch support for the AV_CODEC_ID_SUBRIP format which is completely missing in HandBrake; just adding it along AV_CODEC_ID_{TEXT,SRT} is not enough. HandBrake it's then able to recognize the UTF8 subtitle stream but dies with:

[matroska @ 0x7fd1f08ca660] Subtitle codec 94212 is not supported.

[JohnAStebbins]

ffmpeg has defines for both AV_CODEC_ID_SUBRIP and AV_CODEC_ID_SRT. libav only has a define for AV_CODEC_ID_SRT which is what HandBrake uses. I'm guessing ffmpeg got the define from AV_CODEC_ID_SRT from a merge with libav. You might want to consider making them the same value for compatibility.

[JohnAStebbins]

Or maybe not. I just noticed there is a difference in the description of them in ffmpeg. SRT is described as "SubRip subtitle with embedded timing" whereas SUBRIP is "SubRip subtitle". I'm uncertain whether this difference is really expressed in the code however. SRT seems not to be used aside from codec_descriptors.

[Carl Eugen Hoyos]

Sven Göthel has kindly fixed this issue, I attached the patch here, it was originally posted in ​https://github.com/HandBrake/HandBrake/issues/974

[Simone Caronni]

Still getting the same message, even with the patch.

Subtitle codec 94212 is not supported.

[Carl Eugen Hoyos]

Impressive.

[Simone Caronni]

I think this should be closed here, it is not an FFMpeg issue.

Aniway, trying to replace also the ASS/SSA subtitles as described here:

​https://github.com/HandBrake/HandBrake/issues/974#issuecomment-389577931

[Carl Eugen Hoyos]

HandBrake has switched to FFmpeg some time ago, this should not be an issue anymore.