Opened 11 years ago

Last modified 3 years ago

#1778 reopened enhancement

EIA-608 / EIA-708 Closed Captions disappear when transcoding/reencoding

Reported by: Mike Owned by: Anshul
Priority: wish Component: undetermined
Version: git-master Keywords: cc
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
When transoding/reencoding video ffmepg loses the CC data that was embedded within the actual video stream itself. This type of CC is referred to as EIA-608/EIA-708 and is muxed to the video stream following guidelines in SCTE 128 from my research. If you use '-c:v copy' the CC remains intact. I'm capturing live video from the gige port of a Motorola DSR-6100 IRD that is putting out UDP multicast TS.

This PDF give more details regarding how this method of CC works:
http://www.evertz.com/resources/eia_608_708_cc.pdf

Here are the files produced from the below command. They are larger than the 10MB requested so I've hosted them on my site, here are the direct links.

http://mikecheat.com/disjrhd.ts - Original(18.2MB)
http://mikecheat.com/disjrsd.ts - Reencoded(8.9MB)

There is another open source project that seems to have figured out how to pull this CC data from video. They have the source code on their website, I've included the link.
http://zapping.sourceforge.net/ZVBI/index.html

How to reproduce:
root@hdmux:/home/mike# ffmpeg -i 'udp://239.1.1.3:6100?fifo_size=9000000' -map 0:p:1:0 -c:v mpeg2video -s 704x480 -r ntsc -b:v 3000k -map 0:p:1:1 -c:a mp3 -ac 2 -ar 48000 -b:a 128k -f mpegts disjrsd.ts -map 0:p:1:0 -c:v copy -map 0:p:1:1 -c:a copy -f mpegts disjrhd.ts
ffmpeg version 1.0 Copyright (c) 2000-2012 the FFmpeg developers

built on Sep 28 2012 14:24:44 with gcc 4.4.5 (Debian 4.4.5-8)
configuration: --enable-gpl --enable-nonfree --enable-shared --enable-runtime-cpudetect --enable-libmp3lame --enable-libx264
libavutil 51. 73.101 / 51. 73.101
libavcodec 54. 59.100 / 54. 59.100
libavformat 54. 29.104 / 54. 29.104
libavdevice 54. 2.101 / 54. 2.101
libavfilter 3. 17.100 / 3. 17.100
libswscale 2. 1.101 / 2. 1.101
libswresample 0. 15.100 / 0. 15.100
libpostproc 52. 0.100 / 52. 0.100

[mpegts @ 0x2169240] Unable to seek back to the start
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] non-existing PPS referenced
[h264 @ 0x2193a80] non-existing PPS 0 referenced
[h264 @ 0x2193a80] decode_slice_header error
[h264 @ 0x2193a80] no frame!
[h264 @ 0x2193a80] mmco: unref short failure

Last message repeated 2 times

[mpegts @ 0x2169240] max_analyze_duration 5000000 reached at 5003333
[mpegts @ 0x2169240] Estimating duration from bitrate, this may be inaccurate
Input #0, mpegts, from 'udp://239.1.1.3:6100?fifo_size=9000000':

Duration: N/A, start: 11940.555644, bitrate: 768 kb/s
Program 1

Stream #0:0[0x1e00]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 61.76 fps, 59.94 tbr, 90k tbn, 119.88 tbc
Stream #0:1[0x1020](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), s16, 384 kb/s
Stream #0:2[0x1021](spa): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, s16, 192 kb/s
Stream #0:3[0x1022](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, s16, 192 kb/s

File 'disjrhd.ts' already exists. Overwrite ? [y/N] y
muxrate VBR, pcr every 2 pkts, sdt every 200, pat/pmt every 40 pkts
[mpegts @ 0x22aad60] muxrate VBR, pcr every 5 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mpegts, to 'disjrsd.ts':

Metadata:

encoder : Lavf54.29.104
Stream #0:0: Video: mpeg2video, yuv420p, 704x480 [SAR 40:33 DAR 16:9], q=2-31, 3000 kb/s, 90k tbn, 29.97 tbc
Stream #0:1(eng): Audio: mp3, 48000 Hz, stereo, s16, 128 kb/s

Output #1, mpegts, to 'disjrhd.ts':

Metadata:

encoder : Lavf54.29.104
Stream #1:0: Video: h264 ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 61.76 fps, 90k tbn, 59.94 tbc
Stream #1:1(eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), 384 kb/s

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> mpeg2video)
Stream #0:1 -> #0:1 (ac3 -> libmp3lame)
Stream #0:0 -> #1:0 (copy)
Stream #0:1 -> #1:1 (copy)

Press [q] to stop, ? for help
[h264 @ 0x28a5740] Missing reference picture
[h264 @ 0x28a5740] decode_slice_header error
[h264 @ 0x28a5740] concealing 3600 DC, 3600 AC, 3600 MV errors in B frame
[h264 @ 0x28a63a0] Missing reference picture
[h264 @ 0x28a63a0] decode_slice_header error
[h264 @ 0x28a63a0] concealing 3600 DC, 3600 AC, 3600 MV errors in B frame
[h264 @ 0x28a68e0] reference picture missing during reorder
[h264 @ 0x28a68e0] Missing reference picture
[h264 @ 0x28a68e0] decode_slice_header error
[h264 @ 0x28a68e0] concealing 3600 DC, 3600 AC, 3600 MV errors in P frame
[h264 @ 0x28a7360] mmco: unref short failure
[h264 @ 0x2c4ddc0] mmco: unref short failure
frame= 656 fps= 47 q=2.0 Lq=-1.0 size= 9102kB time=00:00:21.93 bitrate=3398.6kbits/s dup=22 drop=628
video:24157kB audio:1326kB subtitle:0 global headers:0kB muxing overhead -64.284273%
Received signal 2: terminating.

You can't use my command exactly as is because I'm capturing from a live source. You can use one of the above files to emulate the problem. If any other info is needed please let me know.

Thank you

Change History (15)

comment:1 by Mike, 11 years ago

PS I can provide as many samples as needed. This is only one feed of about 30 that I have access to.

comment:2 by Carl Eugen Hoyos, 11 years ago

Component: FFmpegundetermined
Keywords: closed caption subtitle teletext eia 608 708 removed
Priority: normalwish
Status: newopen
Type: defectenhancement
Version: 1.0git-master

FFmpeg does not support closed captions, see also tickett #846.

comment:3 by Mike, 11 years ago

Wow all this time I was thinking it did. Made the assumption that subtitles and closed caption were the same thing. Okay well thanks for the update.

comment:4 by reimar, 11 years ago

The situation is not quite as simple.
The problem in this specific case is that it does not have CC data in a separate stream but embedded in the video frame data.
FFmpeg currently has no way of handling this mess where one stream (video) contains another one (subtitles/CC).
CC in e.g. wtv, mxf and even mov as in ticket #846 is actually supported in so far as extracting it is possible, we just can't decode it into plain text or such and we can't mux it either.

comment:5 by Mike, 11 years ago

I see. Could any of the code from the libzvbi project maybe help with this? It seems they've found a way to gather that information from the video stream and then mux it into a TS. There is also another project called OB-Encoder, they use libav, libx264, and libzvbi with a third party mpegts muxer to produce the desired output. I could use their program but they don't currently support mpeg2 and some of the more advance features of ffmepg that I've accustomed to.

libzvbi - http://zapping.sourceforge.net/ZVBI/Download.html
obencoder - http://code.google.com/p/open-broadcast-encoder/downloads/list

comment:6 by Anshul, 9 years ago

Owner: set to Anshul

comment:7 by Anshul, 9 years ago

It is in pipeline of reviews, someone willing to review and comment please do so.

comment:8 by Anshul, 9 years ago

Resolution: fixed
Status: openclosed

comment:9 by Matt, 9 years ago

This is only for extracting EIA-607/708 captions from the input, correct? Any chance you can also add the ability to insert captions (in EIA-608/708 form) as well? I'd like to be able to produce an mpeg2 ts which contains EIA-608/708 captions instead of having them burned into the video.

comment:10 by Anshul, 9 years ago

Resolution: fixed
Status: closedreopened

yes there is way, cc608 encoder need to be implemented

comment:11 by CJ, 9 years ago

I'm not sure if this should constitute a new ticket, but the "lavfi" method in the revision doesn't work exactly right on a FinalCut-Captioned MOV file. FFMPEG didn't extract anything regardless of seeing the input track. Sample MOV available, but is 4.8GB and only 5mins long.

ffmpeg.exe -f lavfi -i "movie=OPEN_HOUSE_5275_WITH_CC_5mins.mov[out0+subcc]"
-map s "OPEN_HOUSE_5275_WITH_CC_5mins.mov.srt"

ffmpeg version N-73002-gf6c3f1e Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.9.2 (GCC)
  configuration: --arch=x86_64 --target-os=mingw32 --cross-prefix=/mnt/st2/ffmpe
g_macosx/sandbox/mingw-w64-x86_64/bin/x86_64-w64-mingw32- --pkg-config=pkg-confi
g --enable-gpl --disable-decoder=jpeg2000 --enable-libsoxr --enable-fontconfig -
-enable-libass --enable-libutvideo --enable-libbluray --enable-iconv --enable-li
btwolame --extra-cflags=-DLIBTWOLAME_STATIC --enable-libzvbi --enable-libcaca --
enable-libmodplug --extra-libs=-lstdc++ --extra-libs=-lpng --enable-libvidstab -
-enable-libx265 --enable-decklink --extra-libs=-loleaut32 --enable-libx264 --ena
ble-libxvid --enable-libmp3lame --enable-version3 --enable-zlib --enable-librtmp
 --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libopenjpeg --
enable-gnutls --enable-libgsm --enable-libfreetype --enable-libopus --disable-w3
2threads --enable-frei0r --enable-filter=frei0r --enable-libvo-aacenc --enable-b
zlib --enable-libxavs --enable-libopencore-amrnb --enable-libopencore-amrwb --en
able-libvo-amrwbenc --enable-libschroedinger --enable-libvpx --enable-libilbc --
enable-libwavpack --enable-libwebp --enable-libgme --enable-dxva2 --enable-libdc
adec --enable-avisynth --enable-libmfx --enable-static --disable-shared --extra-
cflags= --prefix=/mnt/st2/ffmpeg_macosx/sandbox/mingw-w64-x86_64/x86_64-w64-ming
w32 --extra-libs=-lpsapi --enable-nonfree --enable-libfdk-aac --disable-libfaac
--enable-nvenc --enable-runtime-cpudetect
  libavutil      54. 27.100 / 54. 27.100
  libavcodec     56. 41.100 / 56. 41.100
  libavformat    56. 37.100 / 56. 37.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 17.100 /  5. 17.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.100 /  1.  2.100
  libpostproc    53.  3.100 / 53.  3.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000004183a0] Could not find codec parameters for
 stream 3 (Subtitle: none (c708 / 0x38303763), 1920x1080, 23 kb/s): unknown code
c
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, lavfi, from 'movie=OPEN_HOUSE_5275_WITH_CC_5mins.mov[out0+subcc]':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (Y3[10][10] / 0xA0A3359), yuv422p10le, 1920x108
0 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc
    Stream #0:1: Subtitle: eia_608
File 'OPEN_HOUSE_5275_WITH_CC_5mins.mov.srt' already exists. Overwrite ? [y/N] y

Output #0, srt, to 'OPEN_HOUSE_5275_WITH_CC_5mins.mov.srt':
  Metadata:
    encoder         : Lavf56.37.100
    Stream #0:0: Subtitle: subrip (srt)
    Metadata:
      encoder         : Lavc56.41.100 srt
Stream mapping:
  Stream #0:1 -> #0:0 (eia_608 (cc_dec) -> subrip (srt))
Press [q] to stop, [?] for help
size=       0kB time=00:00:00.00 bitrate=N/A
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing ove
rhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters i
f used)

Without lavfi, you see a c708 codec detection:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'OPEN_HOUSE_5275_WITH_CC_5mins.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2015-04-15 21:12:03
  Duration: 00:05:00.03, start: 0.000000, bitrate: 138315 kb/s
    Stream #0:0(und): Video: prores (apcn / 0x6E637061), yuv422p10le(bt709), 192
0x1080, 135984 kb/s, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc (d
efault)
    Metadata:
      creation_time   : 2015-04-15 21:12:03
      handler_name    : Apple Alias Data Handler
      encoder         : Apple ProRes 422
      timecode        : 00:59:00;02
    Stream #0:1(und): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, stereo, s3
2 (24 bit), 2304 kb/s (default)
    Metadata:
      creation_time   : 2015-04-15 21:12:04
      handler_name    : Apple Alias Data Handler
    Stream #0:2(und): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2015-04-15 21:12:04
      handler_name    : Apple Alias Data Handler
      timecode        : 00:59:00;02
    Stream #0:3(eng): Subtitle: none (c708 / 0x38303763), 1920x1080, 23 kb/s (de
fault)
    Metadata:
      creation_time   : 2015-04-15 21:12:04
      handler_name    : Apple Alias Data Handler
At least one output file must be specified

I can't even do a "rawvideo" or -c:s copy on that stream to anywhere.

Last edited 9 years ago by CJ (previous) (diff)

comment:12 by hemube, 7 years ago

If the source has only single CC in 608/708 format then latest FFmpeg have the option to retain the 608/70d data after transcoding.

working file Media Info


Text #1
ID                                       : 256 (0x100)-CC1
Menu ID                                  : 1 (0x1)
Format                                   : EIA-608
Muxing mode                              : SCTE 128 / DTVCC Transport
Muxing mode, more info                   : Muxed in Video #1
Duration                                 : 20s 938ms
Bit rate mode                            : Constant
Stream size                              : 0.00 Byte (0%)

Text #2
ID                                       : 256 (0x100)-1
Menu ID                                  : 1 (0x1)
Format                                   : EIA-708
Muxing mode                              : SCTE 128 / DTVCC Transport
Muxing mode, more info                   : Muxed in Video #1
Duration                                 : 20s 938ms
Bit rate mode                            : Constant
Stream size                              : 0.00 Byte (0%)

At the same time if the source has multiple CC streams then FFmpeg is not retaining the CC.

Not working file Media Info

Text #1
ID                                       : 256 (0x100)-CC1
Menu ID                                  : 1 (0x1)
Format                                   : EIA-608
Muxing mode                              : SCTE 128 / DTVCC Transport
Muxing mode, more info                   : Muxed in Video #1
Duration                                 : 26s 593ms
Bit rate mode                            : Constant
Stream size                              : 0.00 Byte (0%)

Text #2
ID                                       : 256 (0x100)-CC3
Menu ID                                  : 1 (0x1)
Format                                   : EIA-608
Muxing mode                              : SCTE 128 / DTVCC Transport
Muxing mode, more info                   : Muxed in Video #1
Duration                                 : 26s 593ms
Bit rate mode                            : Constant
Stream size                              : 0.00 Byte (0%)

Text #3
ID                                       : 256 (0x100)-1
Menu ID                                  : 1 (0x1)
Format                                   : EIA-708
Muxing mode                              : SCTE 128 / DTVCC Transport
Muxing mode, more info                   : Muxed in Video #1
Duration                                 : 26s 593ms
Bit rate mode                            : Constant
Stream size                              : 0.00 Byte (0%)

Text #4
ID                                       : 256 (0x100)-2
Menu ID                                  : 1 (0x1)
Format                                   : EIA-708
Muxing mode                              : SCTE 128 / DTVCC Transport
Muxing mode, more info                   : Muxed in Video #1
Duration                                 : 26s 593ms
Bit rate mode                            : Constant
Stream size                              : 0.00 Byte (0%)

comment:13 by Brian Leighty, 7 years ago

hemube, can you give an example of syntax required to make the above sample work? Trying to look in to this. We eventually would like the ability to use the decklink input sending in both video and 608 captions to be encoded into both rtmp outputs and hls outputs. Thanks.

comment:15 by Balling, 3 years ago

Also this patch series should be applied too: https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=1057

Note: See TracTickets for help on using tickets.