DVB subtitles render incorrectly

[leoenc]

Summary of the bug:

Conversion of sup subtitles (pgssub) to DVB subtitles and muxing to mpeg-ts results in bad looking subtitles.
When using the overlay filter instead, they render correctly.

I am using the latest nightly build 20180331-be502ec and testing with VLC Player 3.0.1 (Windows). I also tried playing the dvb output with ffplay but subtitles aren't showing but instead the screen is just flashing from black to video.
I tried with or without canvas_size 1920x1080 and/or s:s 1920x1080, makes no difference.

How to reproduce:
I've attached the input video and subtitles needed to reproduce + both outputs I got on my end.

Output with DVB subtitles which looks wrong:

ffmpeg -y -i input.mp4 -fix_sub_duration -i subtitles.sup -vcodec libx264 -crf 24 -f mpegts -scodec dvbsub output_dvb.ts
ffmpeg version N-90553-gbe502ec6cd Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 12.100 / 56. 12.100
  libavcodec     58. 16.100 / 58. 16.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 13.100 /  7. 13.100
  libswscale      5.  0.102 /  5.  0.102
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2018-04-01T16:37:29.000000Z
  Duration: 00:00:30.12, start: 0.000000, bitrate: 20 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 18 kb/s, 25 fps, 25 tbr, 100 tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2018-04-01T16:37:29.000000Z
      handler_name    : 264@GPAC0.7.0-rev0-gbd5c9af-master
Input #1, sup, from 'subtitles.sup':
  Duration: N/A, start: 3.440000, bitrate: N/A
    Stream #1:0: Subtitle: hdmv_pgs_subtitle, 1920x1080
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (hdmv_pgs_subtitle (pgssub) -> dvb_subtitle (dvbsub))
Press [q] to stop, [?] for help
[sup @ 00000271594a0140] Invalid timestamps stream=0, pts=0, dts=303570, size=4974
[libx264 @ 00000271594833c0] using SAR=1/1
[libx264 @ 00000271594833c0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 00000271594833c0] profile High, level 4.0
Output #0, mpegts, to 'output_dvb.ts':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    encoder         : Lavf58.10.100
    Stream #0:0(und): Video: h264 (libx264), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 25 fps, 90k tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2018-04-01T16:37:29.000000Z
      handler_name    : 264@GPAC0.7.0-rev0-gbd5c9af-master
      encoder         : Lavc58.16.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1: Subtitle: dvb_subtitle (dvbsub), 1920x1080
    Metadata:
      encoder         : Lavc58.16.100 dvbsub
[sup @ 00000271594a0140] Invalid timestamps stream=0, pts=0, dts=767997, size=6238
[sup @ 00000271594a0140] Invalid timestamps stream=0, pts=0, dts=1135096, size=9926
frame=  150 fps=0.0 q=29.0 size=      27kB time=00:00:06.68 bitrate=  33.1kbits/s sp[sup @ 00000271594a0140] Invalid timestamps stream=0, pts=0, dts=1635183, size=17502
frame=  310 fps=309 q=29.0 size=      80kB time=00:00:11.80 bitrate=  55.6kbits/s sp[sup @ 00000271594a0140] Invalid timestamps stream=0, pts=0, dts=2218393, size=18698
frame=  478 fps=317 q=29.0 size=     133kB time=00:00:18.60 bitrate=  58.7kbits/s spframe=  639 fps=318 q=29.0 size=     187kB time=00:00:24.20 bitrate=  63.1kbits/s spframe=  753 fps=310 q=-1.0 Lsize=     221kB time=00:00:30.00 bitrate=  60.3kbits/s speed=12.3x
video:54kB audio:0kB subtitle:65kB other streams:0kB global headers:0kB muxing overhead: 85.350807%
[libx264 @ 00000271594833c0] frame I:4     Avg QP: 8.50  size:   642
[libx264 @ 00000271594833c0] frame P:190   Avg QP:10.10  size:    77
[libx264 @ 00000271594833c0] frame B:559   Avg QP:13.67  size:    69
[libx264 @ 00000271594833c0] consecutive B-frames:  0.9%  0.3%  0.0% 98.8%
[libx264 @ 00000271594833c0] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 00000271594833c0] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 00000271594833c0] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%
[libx264 @ 00000271594833c0] 8x8 transform intra:0.0%
[libx264 @ 00000271594833c0] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 00000271594833c0] i16 v,h,dc,p: 99%  0%  1%  0%
[libx264 @ 00000271594833c0] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 00000271594833c0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 00000271594833c0] kb/s:14.75

Output with overlay subtitles which looks correct:

ffmpeg -y -i input.mp4 -i subtitles.sup -filter_complex "[1:s]scale=width=1920:height=1080[sub];[0:v][sub]overlay" -vcodec libx264 -crf 24 -f mpegts output_overlay.ts
ffmpeg version N-90553-gbe502ec6cd Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 12.100 / 56. 12.100
  libavcodec     58. 16.100 / 58. 16.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 13.100 /  7. 13.100
  libswscale      5.  0.102 /  5.  0.102
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2018-04-01T16:37:29.000000Z
  Duration: 00:00:30.12, start: 0.000000, bitrate: 20 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 18 kb/s, 25 fps, 25 tbr, 100 tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2018-04-01T16:37:29.000000Z
      handler_name    : 264@GPAC0.7.0-rev0-gbd5c9af-master
Input #1, sup, from 'subtitles.sup':
  Duration: N/A, start: 3.440000, bitrate: N/A
    Stream #1:0: Subtitle: hdmv_pgs_subtitle, 1920x1080
Stream mapping:
  Stream #0:0 (h264) -> overlay:main
  Stream #1:0 (pgssub) -> scale
  overlay -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[sup @ 00000292aeae0600] Invalid timestamps stream=0, pts=0, dts=303570, size=4974
[sup @ 00000292aeae0600] Invalid timestamps stream=0, pts=0, dts=767997, size=6238
[sup @ 00000292aeae0600] sub2video: using 720x576 canvas
[libx264 @ 00000292ae6e2700] using SAR=1/1
[libx264 @ 00000292ae6e2700] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 00000292ae6e2700] profile High, level 4.0
Output #0, mpegts, to 'output_overlay.ts':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    encoder         : Lavf58.10.100
    Stream #0:0: Video: h264 (libx264), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 25 fps, 90k tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.16.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=   57 fps=0.0 q=29.0 size=       7kB time=-00:00:00.07 bitrate=N/A speed=N/A  frame=  123 fps=123 q=29.0 size=      20kB time=00:00:02.56 bitrate=  62.9kbits/s sp[sup @ 00000292aeae0600] Invalid timestamps stream=0, pts=0, dts=1135096, size=9926
frame=  189 fps=126 q=29.0 size=      40kB time=00:00:05.20 bitrate=  63.6kbits/s sp[sup @ 00000292aeae0600] Invalid timestamps stream=0, pts=0, dts=1635183, size=17502
frame=  255 fps=127 q=29.0 size=      53kB time=00:00:07.84 bitrate=  55.2kbits/s spframe=  322 fps=128 q=29.0 size=      78kB time=00:00:10.52 bitrate=  60.5kbits/s sp[sup @ 00000292aeae0600] Invalid timestamps stream=0, pts=0, dts=2218393, size=18698
frame=  388 fps=129 q=29.0 size=      91kB time=00:00:13.16 bitrate=  56.5kbits/s spframe=  454 fps=129 q=29.0 size=     125kB time=00:00:15.80 bitrate=  64.6kbits/s spframe=  518 fps=129 q=29.0 size=     137kB time=00:00:18.36 bitrate=  61.2kbits/s spframe=  583 fps=129 q=29.0 size=     149kB time=00:00:20.96 bitrate=  58.4kbits/s spframe=  649 fps=129 q=29.0 size=     184kB time=00:00:23.60 bitrate=  64.0kbits/s spframe=  716 fps=129 q=29.0 size=     198kB time=00:00:26.28 bitrate=  61.6kbits/s spframe=  754 fps=127 q=-1.0 Lsize=     216kB time=00:00:30.04 bitrate=  58.8kbits/s speed=5.07x
video:116kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 85.782951%
[libx264 @ 00000292ae6e2700] frame I:5     Avg QP: 8.08  size: 12890
[libx264 @ 00000292ae6e2700] frame P:190   Avg QP: 9.85  size:    81
[libx264 @ 00000292ae6e2700] frame B:559   Avg QP:13.61  size:    70
[libx264 @ 00000292ae6e2700] consecutive B-frames:  1.1%  0.3%  0.0% 98.7%
[libx264 @ 00000292ae6e2700] mb I  I16..4: 95.4%  3.1%  1.5%
[libx264 @ 00000292ae6e2700] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 00000292ae6e2700] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%  L0:42.3% L1:57.7% BI: 0.0%
[libx264 @ 00000292ae6e2700] 8x8 transform intra:3.0% inter:6.6%
[libx264 @ 00000292ae6e2700] coded y,uvDC,uvAC intra: 1.0% 1.5% 1.5% inter: 0.0% 0.0% 0.0%
[libx264 @ 00000292ae6e2700] i16 v,h,dc,p: 98%  0%  2%  0%
[libx264 @ 00000292ae6e2700] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 55%  4% 41%  0%  0%  0%  0%  0%  0%
[libx264 @ 00000292ae6e2700] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 43% 18% 14%  3%  4%  6%  5%  4%  3%
[libx264 @ 00000292ae6e2700] i8c dc,h,v,p: 99%  1%  0%  0%
[libx264 @ 00000292ae6e2700] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 00000292ae6e2700] ref P L0: 90.5%  6.0%  3.6%
[libx264 @ 00000292ae6e2700] ref B L0: 75.5% 24.5%
[libx264 @ 00000292ae6e2700] ref B L1: 98.5%  1.5%
[libx264 @ 00000292ae6e2700] kb/s:0.00

[leoenc]

Please ignore attachment dvbsub.2.jpg, its the same as dvbsub.jpg and was added by mistake. couldn't find how to delete attachments here.

[Carl Eugen Hoyos]

Can you confirm that the issue is not reproducible with old (!) FFplay?

[Carl Eugen Hoyos]

Afaict, the issue is that the dvbsub encoder does not write a Display definition segment, see also 1fc20af6af3ec31d0ee665557885b6c20c6d3686
Reproducible with the following command, I will delete most attachments:

$ ffmpeg -f lavfi -i color=s=hd1080 -fix_sub_duration -i subtitles.sup -scodec dvbsub -qscale 2 -mbd 2 -t 30 out.ts
ffmpeg version N-90583-g5710fda Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
  configuration: --enable-gpl
  libavutil      56. 12.100 / 56. 12.100
  libavcodec     58. 17.100 / 58. 17.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 14.100 /  7. 14.100
  libswscale      5.  0.102 /  5.  0.102
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, lavfi, from 'color=s=hd1080':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Input #1, sup, from 'subtitles.sup':
  Duration: N/A, start: 3.440000, bitrate: N/A
    Stream #1:0: Subtitle: hdmv_pgs_subtitle, 1920x1080
Please use -q:a or -q:v, -qscale is ambiguous
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg2video (native))
  Stream #1:0 -> #0:1 (hdmv_pgs_subtitle (pgssub) -> dvb_subtitle (dvbsub))
Press [q] to stop, [?] for help
[sup @ 0x21bf140] Invalid timestamps stream=0, pts=0, dts=303570, size=4974
Output #0, mpegts, to 'out.ts':
  Metadata:
    encoder         : Lavf58.10.100
    Stream #0:0: Video: mpeg2video (Main), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.17.100 mpeg2video
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
    Stream #0:1: Subtitle: dvb_subtitle (dvbsub), 1920x1080
    Metadata:
      encoder         : Lavc58.17.100 dvbsub
[sup @ 0x21bf140] Invalid timestamps stream=0, pts=0, dts=767997, size=6238
[sup @ 0x21bf140] Invalid timestamps stream=0, pts=0, dts=1135096, size=9926
[sup @ 0x21bf140] Invalid timestamps stream=0, pts=0, dts=1635183, size=17502
[sup @ 0x21bf140] Invalid timestamps stream=0, pts=0, dts=2218393, size=18698
frame=  750 fps=213 q=2.0 Lsize=    2809kB time=00:00:29.92 bitrate= 769.0kbits/s speed= 8.5x
video:2470kB audio:0kB subtitle:65kB other streams:0kB global headers:0kB muxing overhead: 10.795754%

[leoenc]

Replying to cehoyos:

Can you confirm that the issue is not reproducible with old (!) FFplay?

I can confirm. Tested with a build from 2013 and the output looks perfect.

ffplay_old.exe out.ts
ffplay version N-54161-gc329713 Copyright (c) 2003-2013 the FFmpeg developers
  built on Jun 23 2013 14:25:36 with gcc 4.7.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 37.101 / 52. 37.101
  libavcodec     55. 16.100 / 55. 16.100
  libavformat    55.  9.100 / 55.  9.100
  libavdevice    55.  2.100 / 55.  2.100
  libavfilter     3. 77.101 /  3. 77.101
  libswscale      2.  3.100 /  2.  3.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
max_analyze_duration 5000000 reached at 5000000 microsecondsf=0/0
Input #0, mpegts, from 'out.ts':
  Duration: 00:00:29.96, start: 1.440000, bitrate: 774 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](und): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
  24.74 M-V: -0.028 fd=   0 aq=    0KB vq=  693KB sq=    0B f=0/0

BTW, better use some color for the background when testing, so the subtitle border is visible, e.g.:

ffmpeg -f lavfi -i color=green:s=hd1080 -fix_sub_duration -i subtitles.sup -scodec dvbsub -qscale 2 -mbd 2 -t 30 out.ts

[Carl Eugen Hoyos]

Fixed by Jernej Fijacko in f86f213a6d3dd749152a105df839172fe4c772d1