EIA-608 Closed Captions distortion on the output.

[Will Boba]

Summary of the bug:
How to reproduce:

% ffmpeg.exe -y -v 99 -loglevel 99 -i sample-1min.ts -c:a aac -c:v libx264 sample-1min.mp4

ffmpeg version N-92202-gf72b9904fe Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20181017
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 19.101 / 56. 19.101
  libavcodec     58. 33.101 / 58. 33.101
  libavformat    58. 19.100 / 58. 19.100
  libavdevice    58.  4.105 / 58.  4.105
  libavfilter     7. 33.101 /  7. 33.101
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
  libpostproc    55.  2.100 / 55.  2.100

When I try to re-encode some MPEG2 TS files into h.264/mp4 using libx264 for encoder, embedded closed captions are not properly passed through to output file. It looks to me like CCs are doubled for some reason.

Original CCs (got them with CCextractor tool) look like this:

1
00:00:01,751 --> 00:00:03,469
        Yup, so you just
       want a nice layer.

2
00:00:03,536 --> 00:00:04,670
       That's good there.

3
00:00:04,721 --> 00:00:06,305
      Potatoes are ready.

4
00:00:06,372 --> 00:00:08,090
  This gets covered in
 your famous pot crack?

While extracted from MP4 file after re-encoding - they look like this:

1
00:00:01,234 --> 00:00:03,034
   So we'e're makaking B.C. h h?

2
00:00:03,036 --> 00:00:04,703
        YuYup, so you u just
       want a  nice layerer.

3
00:00:04,871 --> 00:00:06,004
       That''s goodod there.

4
00:00:06,105 --> 00:00:07,672
      Potatoes a are ready..

[Will Boba]

FFmpeg debug log

[Will Boba]

CCs from original file

[Will Boba]

CCs from output file

[Will Boba]

All files required to reproduce this issue can be downloaded from here:
(240MB) ​http://64.137.230.232/cc-distortion.zip

[The_Bart_The]

I recently found out FFMPEG preserves closed caption data by default and am currently in the process of switching my scripts over from HandBrakeCLI. I also discovered this, searched for this bug online and found this post the same day it went up.

Anyway, I'm getting this same bug when I convert an NBC broadcast (captured via antenna, so ATSC 1080i MPEG-2 TS; specifically from WMAQ in Chicago) but I haven't seen this bug yet from CBS (WBBM, 1080i), FOX (WFLD, 720p) PBS (WTTW, 1080i), CW (WPWR, 720p) or ABC (WLS, 720p).

I've only been using FFMPEG seriously for a few days, so I'll report back with any new relevant information. (Maybe it's only certain shows instead of networks; who knows?)

[The_Bart_The]

I've done some research and I think I've cracked it.

Each frame of video can hold up to two characters of closed caption data. In line 5921 of your log, you'll see 367 duplicated frames and zero dropped frames. That's why you see duplicated letters.

If you use -vsync 0 like so:

% ffmpeg.exe -y -v 99 -loglevel 99 -i sample-1min.ts -c:a aac -c:v libx264 sample-1min.mp4 -vsync 0

FFmpeg won't try to adjust the framerate of your video by duplicating & dropping frames and your subtitles should remain intact.

I figured I was on to something when I remembered my local NBC affiliate was causing me the same problem. I remembered that all those years I was using HandBrakeCLI, NBC was the only network doing this to me:

Once I realized NBC was the only network changing framerates back and forth (Video and Film are code for 30 fps and 24 fps, respectively), and saw duplicated frames in my own FFmpeg logs I went looking for how to preserve the original framerate. I came across "-vsync 0" and here we are.

Of course, the bug's still valid despite this workaround.

[Carl Eugen Hoyos]

Please test this patch:

diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
index a12208c..c7e0806 100644
--- a/fftools/ffmpeg.c
+++ b/fftools/ffmpeg.c
@@ -1286,6 +1286,8 @@ static void do_video_out(OutputFile *of,
         ret = avcodec_send_frame(enc, in_picture);
         if (ret < 0)
             goto error;
+        // Make sure the Closed Captions will not be duplicated
+        av_frame_remove_side_data(in_picture, AV_FRAME_DATA_A53_CC);

         while (1) {
             ret = avcodec_receive_packet(enc, &pkt);

[Will Boba]

Replying to cehoyos:

Please test this patch:
diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c

Thank you very much!

Looks like this patch works just as expected. I will do some more testing, but on the first try I've got this:

1
00:00:01,234 --> 00:00:03,001
   So we're making B.C. hash?   

2
00:00:03,003 --> 00:00:04,703
        Yup, so you just        
       want a nice layer.       

3
00:00:04,838 --> 00:00:05,970
       That's good there.       

4
00:00:06,072 --> 00:00:07,639
      Potatoes are ready.       

Comparing to "old" version:

1
00:00:01,234 --> 00:00:03,034
   So we'e're makaking B.C. h h?

2
00:00:03,036 --> 00:00:04,703
        YuYup, so you u just
       want a  nice layerer.

3
00:00:04,871 --> 00:00:06,004
       That''s goodod there.

4
00:00:06,105 --> 00:00:07,672
      Potatoes a are ready..

[Will Boba]

SRT file extracted from output MP4 file after patch was applied

[Will Boba]

This patch works, when FPS filter is not being applied. With the following command problem persists.

I used same set of files from before for testing: ​​http://64.137.230.232/cc-distortion.zip

See code blocks:

ffmpeg version N-92494-ge3a9630982 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)

ffmpeg -threads 0 -y -i sample-1min-ts.ts -c:a aac -c:v libx264 -filter:v yadif,fps=ntsc sample-1min-patched.mp4

1
00:00:01,234 --> 00:00:03,001
   So w we're m making B.C.C. h?

2
00:00:03,003 --> 00:00:04,769
        Yup, so yoyou just      
       want a a nice layayer.   

3
00:00:04,871 --> 00:00:06,137
       Thatat's gogood there..  

4
00:00:06,239 --> 00:00:07,639
      Potatoeses are readydy.   

[Carl Eugen Hoyos]

Please also test this patch:

diff --git a/libavfilter/vf_fps.c b/libavfilter/vf_fps.c
index 9167a00..33a6079 100644
--- a/libavfilter/vf_fps.c
+++ b/libavfilter/vf_fps.c
@@ -249,6 +249,7 @@ static int write_frame(AVFilterContext *ctx, FPSContext *s, AVFilterLink *outlin
         frame = av_frame_clone(s->frames[0]);
         if (!frame)
             return AVERROR(ENOMEM);
+        av_frame_remove_side_data(s->frames[0], AV_FRAME_DATA_A53_CC);
         frame->pts = s->next_pts++;

         av_log(ctx, AV_LOG_DEBUG, "Writing frame with pts %"PRId64" to pts %"PRId64"\n",

[Will Boba]

Replying to cehoyos:

Please also test this patch:
diff --git a/libavfilter/vf_fps.c b/libavfilter/vf_fps.c
index 9167a00..33a6079 100644
--- a/libavfilter/vf_fps.c
+++ b/libavfilter/vf_fps.c

This one worked great! Thank you very much!

[Carl Eugen Hoyos]

Original ticket fixed in 698574ac5d68e806279a8d28aeff48a407377538, fps issue fixed in 1893c72086024e53ac32312ecb96693c5be023d0

[Will Boba]

Exactly same problem with closed captions when using deinterlacing with frame doubler:

% ffmpeg.exe -y -v 99 -loglevel 99 -i sample-1min.ts -c:a aac -vf yadif=1:-1:0 -c:v libx264 sample-1min.mp4

Same CC data is attached to each frame produced from 2 fields.

[Carl Eugen Hoyos]

Please open a new ticket, provide the command line you tested including the complete, uncut console output. (Sorry that I didn't tell you originally.)