Closed Captions: Data Ignored since exceeding screen width

[Carl Eugen Hoyos]

An isom file was uploaded that is supposed to contain Closed Captions as a separate stream.
Playing the file with ffplay shows many warnings, I don't know how to use Closed Captions with ffmpeg.

$ ffplay -i Full\ Episode.mp4
ffplay version N-72859-g5b2f979 Copyright (c) 2003-2015 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      54. 27.100 / 54. 27.100
  libavcodec     56. 41.100 / 56. 41.100
  libavformat    56. 36.100 / 56. 36.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 16.101 /  5. 16.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.100 /  1.  2.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Full Episode.mp4': f=0/0
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1iso6
    creation_time   : 2015-06-12 12:48:45
  Duration: 00:23:28.02, start: 0.000000, bitrate: 1083 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1024x576, 951 kb/s, 23.98 fps, 23.98 tbr, 11988 tbn, 47.95 tbc (default)
    Metadata:
      creation_time   : 2015-06-12 12:48:45
      handler_name    : AVC Video
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      creation_time   : 2015-06-12 10:15:38
      handler_name    : AAC Audio (2.0)
    Stream #0:2(eng): Subtitle: eia_608 (c608 / 0x38303663), 1920x1080, 0 kb/s (default)
    Metadata:
      creation_time   : 2015-03-26 05:58:44
      handler_name    : Core Media Closed Caption
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 39 times
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 5 times
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 5 times
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 9 times
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 3 times
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 1 times
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 3 times
[Closed caption Decoder @ 0x7f9eb43edcc0] Data Ignored since exceeding screen width
    Last message repeated 3 times
  11.65 A-V: -0.033 fd=   0 aq= 1635KB vq=13724KB sq=    0B f=0/0

[Carl Eugen Hoyos]

​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket4616/

[Carl Eugen Hoyos]

videolan ticket 12685

[Carl Eugen Hoyos]

$ ffmpeg -i Full\ Episode.mp4 -qscale 2 out.mkv
ffmpeg version N-73448-g7efe81a Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      54. 28.100 / 54. 28.100
  libavcodec     56. 46.101 / 56. 46.101
  libavformat    56. 40.100 / 56. 40.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 20.100 /  5. 20.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.100 /  1.  2.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Full Episode.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1iso6
    creation_time   : 2015-06-12 12:48:45
  Duration: 00:23:28.02, start: 0.000000, bitrate: 1083 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1024x576, 951 kb/s, 23.98 fps, 23.98 tbr, 11988 tbn, 47.95 tbc (default)
    Metadata:
      creation_time   : 2015-06-12 12:48:45
      handler_name    : AVC Video
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      creation_time   : 2015-06-12 10:15:38
      handler_name    : AAC Audio (2.0)
    Stream #0:2(eng): Subtitle: eia_608 (c608 / 0x38303663), 1920x1080, 0 kb/s (default)
    Metadata:
      creation_time   : 2015-03-26 05:58:44
      handler_name    : Core Media Closed Caption
Please use -q:a or -q:v, -qscale is ambiguous
Output #0, matroska, to 'out.mkv':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1iso6
    encoder         : Lavf56.40.100
    Stream #0:0(eng): Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 1024x576, q=2-31, 200 kb/s, 23.98 fps, 1k tbn, 23.98 tbc (default)
    Metadata:
      creation_time   : 2015-06-12 12:48:45
      handler_name    : AVC Video
      encoder         : Lavc56.46.101 mpeg4
    Stream #0:1(eng): Audio: ac3 ([0] [0][0] / 0x2000), 48000 Hz, stereo, fltp, 192 kb/s (default)
    Metadata:
      creation_time   : 2015-06-12 10:15:38
      handler_name    : AAC Audio (2.0)
      encoder         : Lavc56.46.101 ac3
    Stream #0:2(eng): Subtitle: ass, 1920x1080 (default)
    Metadata:
      creation_time   : 2015-03-26 05:58:44
      handler_name    : Core Media Closed Caption
      encoder         : Lavc56.46.101 ass
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mpeg4 (native))
  Stream #0:1 -> #0:1 (aac (native) -> ac3 (native))
  Stream #0:2 -> #0:2 (eia_608 (cc_dec) -> ass (native))
Press [q] to stop, [?] for help
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 5 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 3 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 29 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 11 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 19 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 21 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 3 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 17 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 24 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 5 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 25 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 13 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 29 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 25 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 34 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 23 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 5 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 29 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 19 times
[Closed caption Decoder @ 0x33cd140] Data Ignored since exceeding screen width/s
    Last message repeated 25 times
frame= 7118 fps=579 q=2.0 Lsize=  198180kB time=00:04:57.33 bitrate=5460.1kbits/s
video:191071kB audio:6969kB subtitle:0kB other streams:0kB global headers:1kB muxing overhead: 0.070861%

[ndjamena]

Someone beat me to it...

​http://forum.doom9.org/showthread.php?p=1718273#post1718273

I've been trying to get Subtitle Edit, CCextractor and VLC to decode these things properly and I'm not sure how much progress has been made. Pretty much everything you need to know to decode them is scattered throughout the linked to Subtitle Edit page.

Basically, 608 captions = Line 21 captions. They're designed to be written into line 21 two bytes at a time at 30 frames per second and should be decoded as such. I don't know if that's the problem here, but it seems to be what everyone else is missing.

​https://trac.videolan.org/vlc/ticket/12685

[Anshul]

From Mp4 Correct data is not received.
through GDB, I saw that first packet is of 12 bytes
(gdb) x /12xb avpkt->data
0x26dbec0: 0x00 0x00 0x00 0x0c 0x63 0x64 0x61 0x74
0x26dbec8: 0x00 0x00 0x00 0x00

where only last 4 bytes are of cc rest is mp4 header

from same packet if I ignore first 4 bytes and print
#1 0x0000000000cac5c0 in avcodec_decode_subtitle2 (avctx=0x254aec0, sub=0x7fffffffd510, got_sub_ptr=0x7fffffffd63c, avpkt=0x7fffffffd5d0) at libavcodec/utils.c:2833
2833 ret = avctx->codec->decode(avctx, sub, got_sub_ptr, &pkt_recoded);
(gdb) p avpkt->data+4
$9 = (uint8_t *) 0x254c9f4 "cdat\224 \224\256\224@\227\242>> \310u\354\354 damag\345\272 2\260%, bu\364 \364h\345\224\340\227\242sys\364\345m\247s s\364\351\354\354 \357p\345\362a\364\351\357na쮀\200\200\200\200\224/"

Any one can see cdat string at start which is specific to Mp4 protocol

[Carl Eugen Hoyos]

Thank you for looking into this!

If the header were removed inside the mov demuxer, would the muxer be able to recreate it?

[Anshul]

anshul@linux-z9q9:~/Project/MUltimedia/FFmpeg> ./ffmpeg -i /d/TestVideos/mp4/Full\ Episode.mp4 -scodec copy -acodec copy test.mp4
ffmpeg version N-74672-g540f280 Copyright (c) 2000-2015 the FFmpeg developers

built with gcc 4.8 (SUSE Linux)
configuration: --enable-gpl --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-nonfree --enable-postproc --enable-version3 --enable-x11grab --enable-libvpx --disable-optimizations
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 58.100 / 56. 58.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 38.100 / 5. 38.100
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/d/TestVideos/mp4/Full Episode.mp4':

Metadata:

major_brand : isom
minor_version : 1
compatible_brands: isomavc1iso6
creation_time : 2015-06-12 12:48:45

Duration: 00:23:28.02, start: 0.000000, bitrate: 1083 kb/s

Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1024x576, 951 kb/s, 23.98 fps, 23.98 tbr, 11988 tbn, 47.95 tbc (default)
Metadata:

creation_time : 2015-06-12 12:48:45
handler_name : AVC Video

Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
Metadata:

creation_time : 2015-06-12 10:15:38
handler_name : AAC Audio (2.0)

Stream #0:2(eng): Subtitle: eia_608 (c608 / 0x38303663), 1920x1080, 0 kb/s (default)
Metadata:

creation_time : 2015-03-26 05:58:44
handler_name : Core Media Closed Caption

File 'test.mp4' already exists. Overwrite ? [y/N] y
[libx264 @ 0x3fca4a0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX XOP FMA4 FMA3 LZCNT BMI1
[libx264 @ 0x3fca4a0] profile High, level 3.1
[libx264 @ 0x3fca4a0] 264 - core 148 r2 73ae2d1 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - ​http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=23 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[mp4 @ 0x3fc9a00] Codec for stream 1 does not use global headers but container format requires global headers
[mp4 @ 0x3fc9a00] Codec for stream 2 does not use global headers but container format requires global headers
[mp4 @ 0x3fc9a00] Could not find tag for codec eia_608 in stream #2, codec not currently supported in container
Output #0, mp4, to 'test.mp4':

Metadata:

major_brand : isom
minor_version : 1
compatible_brands: isomavc1iso6
encoder : Lavf56.40.101
Stream #0:0(eng): Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1024x576, q=-1--1, 23.98 fps, 11988 tbn, 23.98 tbc (default)
Metadata:

creation_time : 2015-06-12 12:48:45
handler_name : AVC Video
encoder : Lavc56.58.100 libx264

Stream #0:1(eng): Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 127 kb/s (default)
Metadata:

creation_time : 2015-06-12 10:15:38
handler_name : AAC Audio (2.0)

Stream #0:2(eng): Subtitle: eia_608 (c608 / 0x38303663), 1920x1080, 0 kb/s (default)
Metadata:

creation_time : 2015-03-26 05:58:44
handler_name : Core Media Closed Caption

Stream mapping:

Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (copy)
Stream #0:2 -> #0:2 (copy)

Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument

[Anshul]

cant say anything about MOV muxer and demuxer never worked on it, but just now header is there and it is still not able to mux it, more understanding of mov protocol is required to remux that stream.

[Anshul]

In Mov demuxer following function for First frame of Closed caption.

Inside mov_read_packet
while calling av_get_packet sample->size must be 4byte and 8 bytes before that must be discarded or parsed. In present scenario one atom is not parsed and that atom header act as garbage for Closed caption decoder. Present size of sample id 12 which should be 4 byte
ret = av_get_packet(sc->pb, pkt, sample->size);

[Carl Eugen Hoyos]

Works fine (?) here:

$ ffmpeg -i Full\ Episode\ 2.mp4 -c copy -map 0 out.mov

But that wasn't my question, what I meant was: If the mov demuxer removes the header bytes that the cc decoder doesn't understand, would the mov muxer have all necessary information to recreate the header if somebody implemented it? Or would the information be lost?
Or in other words: Is a separate codec id necessary for closed captions in mov because it contains different data than closed captions in mpeg streams or is it possible to stream copy both from mov and mpeg to mov and therefore one codec id is correct?

[Anshul]

With your command "-c copy " it ignores eia-608 stream and copy only audio and video.

In mov as I know there is lot of atoms inside atoms so correct way would be to see why cdat atom is ignored and copied in pkt->data. cdat atom should be parsed while demuxing and inserted while muxing in my opinion.

In CCextractor we usually check 2 atoms for Closed caption ("cdat" and "cdt2")

[Carl Eugen Hoyos]

Replying to er.anshul.maheshwari@…:

With your command "-c copy " it ignores eia-608 stream and copy only audio and video.

Mistake on copy & paste, sorry.

[Carl Eugen Hoyos]

The beginning of the Closed Captions looks like the following:

00000000  00 00 00 0c 63 64 61 74  00 00 00 00 00 00 00 5c  |....cdat.......\|
00000010  63 64 61 74 94 20 94 52  c8 49 2c 20 54 c8 45 52  |cdat. .R.I, T.ER|
00000020  45 ae 20 4a d5 d3 54 20  57 c1 ce 54 45 c4 94 f2  |E. J..T W..TE...|
00000030  54 4f 20 57 45 4c 43 4f  cd 45 20 d9 4f d5 20 54  |TO WELCO.E .O. T|
00000040  4f 20 cd d9 20 d3 c8 4f  57 80 94 20 94 2c 94 2f  |O .. ..OW.. .,./|
00000050  94 20 94 f2 d3 54 c1 52  52 49 ce c7 20 cd 45 ad  |. ...T.RRI.. .E.|
00000060  ad cb d5 da 43 4f ae 80  00 00 00 3c 63 64 61 74  |....CO.....<cdat|

I suspect this (the second packet) should be decoded as "HI THERE JUST WANTED TO WELCOME YOU TO MY SHOW STARRING ME..." (or similar). If I cut away the first eight bytes (size + "cdat") and feed the shortened packet to the cc_dec decoder the result is "T WCO". What do I miss?

[Carl Eugen Hoyos]

Replying to cehoyos:

The beginning of the Closed Captions looks like the following:

Looking at Full\ Episode\ 2.mp4 in ​http://streams.videolan.org/issues/12685/

[Carl Eugen Hoyos]

For the other sample, the Closed Captions start with:

00000000  00 00 00 0c 63 64 61 74  00 00 00 00 00 00 00 52  |....cdat.......R|
00000010  63 64 61 74 94 20 94 ae  94 40 97 a2 3e 3e 20 c8  |cdat. ...@..>> .|
00000020  75 ec ec 20 64 61 6d 61  67 e5 ba 20 32 b0 25 2c  |u.. damag.. 2.%,|
00000030  20 62 75 f4 20 f4 68 e5  94 e0 97 a2 73 79 73 f4  | bu. .h.....sys.|
00000040  e5 6d a7 73 20 73 f4 e9  ec ec 20 ef 70 e5 f2 61  |.m.s s.... .p..a|
00000050  f4 e9 ef 6e 61 ec ae 80  80 80 80 80 94 2f 00 00  |...na......../..|

This should be decoded as ">> Hull damage: 20%, but the system's still operational."
When cutting away the first eight bytes, I get no output from cc_dec.

[Anshul]

can you share what changes did you made in mov.c to cut the data, though I will try hardcoding it in ccaption.c for this video and come back tommarow with some solution.
I do have working solution known as CCextractor, which works perfectly on this file.
There might be some other feature also missing in Closed caption implementation. If there are something missing I will implement it and let you know.

[Carl Eugen Hoyos]

diff --git a/libavformat/mov.c b/libavformat/mov.c
index 45367d3..132de3d 100644
--- a/libavformat/mov.c
+++ b/libavformat/mov.c
@@ -4626,6 +4626,10 @@ static int mov_read_packet(AVFormatContext *s, AVPacket *pkt)
     if (mov->aax_mode)
         aax_filter(pkt->data, pkt->size, mov);

+if (st->codec->codec_id == AV_CODEC_ID_EIA_608) {
+memmove(pkt->data, pkt->data + 8, pkt->size - 8);
+pkt->size -= 8;
+}
     return 0;
 }

[Carl Eugen Hoyos]

The following has the same effect:

diff --git a/libavcodec/ccaption_dec.c b/libavcodec/ccaption_dec.c
index 9f67caa..0534e8b 100644
--- a/libavcodec/ccaption_dec.c
+++ b/libavcodec/ccaption_dec.c
@@ -23,6 +23,7 @@
 #include "ass.h"
 #include "libavutil/internal.h"
 #include "libavutil/opt.h"
+#include "libavutil/intreadwrite.h"

 #define SCREEN_ROWS 15
 #define SCREEN_COLUMNS 32
@@ -528,6 +529,7 @@ static int decode(AVCodecContext *avctx, void *data, int *got_sub, AVPacket *avp
     int len = avpkt->size;
     int ret = 0;
     int i;
+int offset = 0;

     if ( ctx->pktbuf->size < len) {
         ret = av_buffer_realloc(&ctx->pktbuf, len);
@@ -537,7 +539,9 @@ static int decode(AVCodecContext *avctx, void *data, int *got_sub, AVPacket *avp
             ret = 0;
         }
     }
+    if (AV_RB32(avpkt->data + 4) == 0x63646174)
+        offset = 8;
-    memcpy(ctx->pktbuf->data, avpkt->data, len);
+    memcpy(ctx->pktbuf->data, avpkt->data + offset, len-offset);
     bptr = ctx->pktbuf->data;

Btw: Why is this memcpy() needed?

[Elon Musk]

Those patches are nonsense, ffmpeg doesn't have decoder for this type of closed captions.

[Carl Eugen Hoyos]

An additional sample from mpv issue 6121 is in ​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket4616/
A link to a very large sample is in ticket #6448.

[Carl Eugen Hoyos]

Another sample is attached to ticket #7694.

[Carl Eugen Hoyos]

Fixed by Paul in f91906973c62459f31beb5683a8e2105758af0bb