[Intel Caption] Microsoft RTMP Ingest - FFmpeg fails to decode closed-caption to srt/webvtt

[Zied Aouina]

Summary of the bug: All Windows FFmpeg versions after 4.2.3 fail to extract the Closed-caption eia-608 and convert it to srt or webvtt. It's stuck without generating any output.
How to reproduce:

% ffmpeg.exe -f lavfi -i movie=flvdecoder_input223.flv[out+subcc] -y  -map 0:1  ./output_p.srt
ffmpeg version N-102809-gde8e6e67e7-20210630 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10-win32 (GCC) 20210408
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libvmaf --enable-vulkan --enable-amf --enable-libaom --disable-avisynth --enable-libdav1d --disable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-libglslang --enable-libgme --enable-libass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librav1e --disable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libvidstab --disable-libx264 --disable-libx265 --disable-libxavs2 --disable-libxvid --enable-libzimg --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20210630
  libavutil      57.  0.100 / 57.  0.100
  libavcodec     59.  3.100 / 59.  3.100
  libavformat    59.  3.101 / 59.  3.101
  libavdevice    59.  0.100 / 59.  0.100
  libavfilter     8.  0.103 /  8.  0.103
  libswscale      6.  0.100 /  6.  0.100
  libswresample   4.  0.100 /  4.  0.100
[flv @ 0000022bbf6bbb40] Estimating duration from bitrate, this may be inaccurate
Input #0, lavfi, from 'movie=flvdecoder_input223.flv[out+subcc]':
  Duration: N/A, start: 0.000367, bitrate: N/A
  Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn
  Stream #0:1: Subtitle: eia_608
Output #0, srt, to './output_p.srt':
  Metadata:
    encoder         : Lavf59.3.101
  Stream #0:0: Subtitle: subrip
    Metadata:
      encoder         : Lavc59.3.100 srt
Stream mapping:
  Stream #0:1 -> #0:0 (eia_608 (cc_dec) -> subrip (srt))
Press [q] to stop, [?] for help
size=       0kB time=00:00:00.00 bitrate=N/A speed=   0x

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Zied Aouina]

Flv file contains h264 with closed caption embedded into SEI packets

[Zied Aouina]

Version 4.2.3 was able to convert the file :
ffmpeg.exe -loglevel debug -f lavfi -i movie=flvdecoder_input223-1.flv[out+subcc] -y -map 0:1 ./output_p.srt
ffmpeg version git-2020-06-17-0b3bd00 Copyright (c) 2000-2020 the FFmpeg developers

built with gcc 9.3.1 (GCC) 20200523
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
libavutil 56. 55.100 / 56. 55.100
libavcodec 58. 92.100 / 58. 92.100
libavformat 58. 46.101 / 58. 46.101
libavdevice 58. 11.100 / 58. 11.100
libavfilter 7. 86.100 / 7. 86.100
libswscale 5. 8.100 / 5. 8.100
libswresample 3. 8.100 / 3. 8.100
libpostproc 55. 8.100 / 55. 8.100

Input file #0 (movie=flvdecoder_input223-1.flv[out+subcc]):

Input stream #0:0 (video): 128 packets read (176947200 bytes);
Input stream #0:1 (subtitle): 128 packets read (7650 bytes); 3 frames decoded;
Total: 256 packets (176954850 bytes) demuxed

Output file #0 (./output_p.srt):

Output stream #0:0 (subtitle): 3 frames encoded; 3 packets muxed (258 bytes);
Total: 3 packets (258 bytes) muxed

3 frames successfully decoded, 0 decoding errors
[AVIOContext @ 000002018da92900] Statistics: 0 seeks, 1 writeouts
[AVIOContext @ 000002018d0c2900] Statistics: 2119113 bytes read, 0 seeks

[Zied Aouina]

The old build from zeranoe works fine: ​https://archive.org/download/zeranoe/win64/static/

[Zied Aouina]

The issue was introduced starting from build n4.4:
Here are the builds I compared (using the file I uploaded): ​https://sourceforge.net/projects/ffmpeg-windows-builds/files/release/

Here is how n4.3.2 performs:
ffmpeg.exe -f lavfi -i movie=flvdecoder_input223-1.flv[out+subcc] -y -map 0:1 ./output_p.srt
ffmpeg version n4.3.2-ffmpeg-windows-build-helpers Copyright (c) 2000-2021 the FFmpeg developers

built with gcc 10.2.0 (GCC)
configuration: --pkg-config=pkg-config --pkg-config-flags=--static --extra-version=ffmpeg-windows-build-helpers --enable-version3 --disable-debug --disable-w32threads --arch=x86_64 --target-os=mingw32 --cross-prefix=/home/runner/work/ffmpeg-windows-builds/ffmpeg-windows-builds/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32- --enable-libcaca --enable-gray --enable-libtesseract --enable-fontconfig --enable-gmp --enable-gnutls --enable-libass --enable-libbluray --enable-libbs2b --enable-libflite --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libvorbis --enable-libwebp --enable-libzimg --enable-libzvbi --enable-libmysofa --enable-libopenjpeg --enable-libopenh264 --enable-liblensfun --enable-libvmaf --enable-libsrt --enable-demuxer=dash --enable-libxml2 --enable-opengl --enable-libdav1d --enable-cuda-llvm --enable-libaom --enable-libvpx --enable-nvenc --enable-nvdec --extra-libs=-lharfbuzz --extra-libs=-lm --extra-libs=-lpthread --extra-cflags=-DLIBTWOLAME_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DCACA_STATIC --enable-amf --enable-libmfx --enable-gpl --enable-frei0r --enable-filter=frei0r --enable-librubberband --enable-libvidstab --enable-libx264 --enable-libx265 --enable-avisynth --enable-libaribb24 --enable-libxvid --enable-libdavs2 --enable-libxavs2 --enable-libxavs --enable-avresample --extra-cflags='-mtune=generic' --extra-cflags=-O3 --enable-static --disable-shared --prefix=/home/runner/work/ffmpeg-windows-builds/ffmpeg-windows-builds/sandbox/cross_compilers/mingw-w64-x86_64/x86_64-w64-mingw32
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100

Input #0, lavfi, from 'movie=flvdecoder_input223-1.flv[out+subcc]':

Duration: N/A, start: 0.000356, bitrate: N/A

Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc
Stream #0:1: Subtitle: eia_608

Output #0, srt, to './output_p.srt':

Metadata:

encoder : Lavf58.45.100
Stream #0:0: Subtitle: subrip (srt)
Metadata:

encoder : Lavc58.91.100 srt

Stream mapping:

Stream #0:1 -> #0:0 (eia_608 (cc_dec) -> subrip (srt))

Press [q] to stop, ? for help
[Parsed_movie_0 @ 0000023fe4cc0740] EOF timestamp not reliable
size= 0kB time=00:00:00.02 bitrate= 106.7kbits/s speed=0.205x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 39.534885%

[Zied Aouina]

Hi, This is a high priority ticket and the FFmpeg version is currently used in a highly visible product in Microsoft. We have customers experience issues with Caption during Teams Live Event. Please help,

[Elon Musk]

Use -data_field first as decoder option in CLI. Default value was changed from first to auto in latest FFmpeg version.
Or modify AVOption of same name in API for this decoder.

[Zied Aouina]

Thanks @Elon for the reply, This is the command we are currently using:
ffmpeg.exe -f lavfi -i movie=flvdecoder_input223.flv[out+subcc] -y -map 0:1 ./output_p.srt

I will be looking to see any updates in the FFmpeg documentation. Can you please elaborate and provide pointers the right decoding options or the right FF command er can use.
Thank you!

[Elon Musk]

ffmpeg.exe -data_field first -f lavfi -i movie=flvdecoder_input223.flv[out+subcc] -y -map 0:1 ./output_p.srt

[Elon Musk]

ffmpeg -h decoder=cc_dec

[Zied Aouina]

The Command you provided worked fine. Thank you so much for the help! Really appreciated!
We are going to proceed to make a release today and test with customers. Will post the updates here.
Meanwhile, I was looking into any FFmpeg documentation that can show how to use the CLI to decode embedded caption using data_field and couldn't find any.