A/V desync when transcoding MPEG-TS streams if an audio track disappears

[Aurélien Nephtali]

Hi,

I have some streams that for some unknown reasons (I don't have control on them since it's mainly DVB-S streams) lose audio from time to time.

In some cases a new PMT with only a video track can be found or the audio packets could just disappear.
This can last a few seconds or multiple hours (delay for the broadcaster to fix the problem I guess).

In both cases get_input_packet() from process_input() won't see packets with stream_index=audio anymore.
ffmpeg will keep encoding the video frames.

The problem is with av_interleaved_write_frame() in write_frame(). This functions waits for all streams to have at least one packet before writing to the output.
In this case it will buffer video packets (thus leaking) until another audio packet comes ; and then it will start to write the video packets again but they won't match the audio packets.

I crafted (once again) a sample 'bbb_gap_sound.ts' by deleting some audio packets from an MPEG-TS stream (I don't have a sample where the PMT changes).
It's from Big Buck Bunny so there are only music and noises but it's quite easy to spot the A/V desync (the audio disappears at ~37s).

How to reproduce:

ffmpeg -i bbb_gap_sound.ts  -acodec libfdk_aac -b:a 64k -ac 2 -ar 44100 -vcodec libx264 -b:v 800k -s 320x240 /tmp/bbb_encoded.ts
ffmpeg version N-58591-ga677fb6 Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 29 2013 17:58:08 with gcc 4.7 (Debian 4.7.2-5)
  configuration: --prefix=/home/aurelien/local --enable-postproc --enable-nonfree --enable-gpl --enable-libfaac --enable-shared --enable-libx264 --extra-cflags=-I/home/aurelien/local/include --extra-ldflags=-L/home/aurelien/local/lib --enable-version3 --disable-optimizations --enable-libfreetype --disable-stripping --enable-libaacplus --enable-libfdk-aac
  libavutil      52. 55.100 / 52. 55.100
  libavcodec     55. 44.100 / 55. 44.100
  libavformat    55. 21.102 / 55. 21.102
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     3. 91.100 /  3. 91.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
[h264 @ 0x16a9aa0] non-existing PPS referenced
[h264 @ 0x16a9aa0] non-existing PPS 0 referenced
[h264 @ 0x16a9aa0] decode_slice_header error
[h264 @ 0x16a9aa0] no frame!
[... non-existing PPS messages skipped ...]
[mpegts @ 0x16a5a00] PES packet size mismatch
Input #0, mpegts, from 'bbb_gap_sound.ts':
  Duration: 00:01:58.26, start: 211.763244, bitrate: 988 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 320x240, 24 fps, 24 tbr, 90k tbn, 48 tbc
    Stream #0:1[0x101](eng): Audio: aac ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 64 kb/s
[libx264 @ 0x16abb40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 @ 0x16abb40] profile High, level 2.0
Output #0, mpegts, to '/tmp/bbb_encoded.ts':
  Metadata:
    encoder         : Lavf55.21.102
    Stream #0:0: Video: h264 (libx264), yuv420p, 320x240, q=-1--1, 800 kb/s, 90k tbn, 24 tbc
    Stream #0:1(eng): Audio: aac (libfdk_aac), 44100 Hz, stereo, s16, 64 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx264)
  Stream #0:1 -> #0:1 (aac -> libfdk_aac)
Press [q] to stop, [?] for help
[mpegts @ 0x16a5a00] PES packet size mismatchme=00:01:53.16 bitrate= 616.6kbits/s dup=32 drop=0
[h264 @ 0x228af80] error while decoding MB 4 13, bytestream (-40)
[h264 @ 0x228af80] concealing 85 DC, 85 AC, 85 MV errors in P frame
frame= 2839 fps=601 q=-1.0 Lsize=   14116kB time=00:01:58.20 bitrate= 978.2kbits/s dup=33 drop=0
video:12165kB audio:565kB subtitle:0 global headers:0kB muxing overhead 10.884066%

[Carl Eugen Hoyos]

Please provide a real-world sample.

[Aurélien Nephtali]

I managed to get a real sample this night on a Chinese channel.
I uploaded it on the FTP (ntd_gap_sound.ts).

There is the end of a Nissan commercial, the start of an unknown one then a test card, then a black screen and no sound for ~47s.
After some time a news broadcast starts.

Here are the timestamps :
00:12s : test card + black screen and no sound
00:59s : news broadcast starts

When encoded with ffmpeg I get :
00:12s : test card + black screen and no sound
00:32s : news broadcast _sound_ starts
00:59s : news broadcast _video_ starts

Is this sample OK for you ?

[Carl Eugen Hoyos]

Thank your for the sample!
For future tickets: Please do not use external libraries like libx264 and libfdk-aac if they are not needed to reproduce the problem.

The following produces an output file that plays severely out-of-sync, -async 1 does not help, I don't think this is a regression:

$ ffmpeg -i ntd_gap_sound.ts -qscale 10 out.ts
ffmpeg version N-58656-gbd9de4f Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 30 2013 11:55:29 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      52. 55.100 / 52. 55.100
  libavcodec     55. 44.100 / 55. 44.100
  libavformat    55. 21.102 / 55. 21.102
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     3. 91.100 /  3. 91.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
[mpeg2video @ 0x2e0b780] Invalid frame dimensions 0x0.
    Last message repeated 19 times
Input #0, mpegts, from 'ntd_gap_sound.ts':
  Duration: 00:01:08.87, start: 50432.154678, bitrate: 3972 kb/s
  Program 1701
    Metadata:
      service_name    : MTV Portugal
      service_provider: Arqiva
  Program 1715
    Metadata:
      service_name    : D1 DAB
      service_provider: Arqiva
  Program 1720
    Metadata:
      service_name    : ANN
      service_provider: Arqiva
  Program 1740
    Metadata:
      service_name    : DCRO
      service_provider: Arqiva
  Program 1741
    Metadata:
      service_name    : DCBU
      service_provider: Arqiva
  Program 1752
    Metadata:
      service_name    : NTDTV
      service_provider: Arqiva
    Stream #0:0[0xa47]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv), 544x576 [SAR 24:17 DAR 4:3], max. 15000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0xa48](eng): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16p, 128 kb/s
  Program 1760
    Metadata:
      service_name    : Il Sole
      service_provider: Arqiva
  Program 1765
    Metadata:
      service_name    : Deepam TV
      service_provider: Arqiva
  Program 1775
    Metadata:
      service_name    : GTV
      service_provider: Arqiva
  Program 1780
    Metadata:
      service_name    : Fox Sports HD
      service_provider: Arqiva
Please use -q:a or -q:v, -qscale is ambiguous
Output #0, mpegts, to 'out.ts':
  Metadata:
    encoder         : Lavf55.21.102
    Stream #0:0: Video: mpeg2video, yuv420p, 544x576 [SAR 24:17 DAR 4:3], q=2-31, 200 kb/s, 90k tbn, 25 tbc
    Stream #0:1(eng): Audio: mp2, 48000 Hz, stereo, s16, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video -> mpeg2video)
  Stream #0:1 -> #0:1 (mp2 -> mp2)
Press [q] to stop, [?] for help
frame=  416 fps=0.0 q=10.0 size=    3267kB time=00:00:16.56 bitrate=1616.3kbits/s dup=31 drop=0   [mpegts @ 0x2de7900] PES packet size mismatch
[mp2 @ 0x2e0d100] Header missing
Error while decoding stream #0:1: Invalid data found when processing input
frame= 1218 fps=1217 q=10.0 size=    3498kB time=00:00:48.64 bitrate= 589.1kbits/s dup=31 drop=0  [mpegts @ 0x2de7900] PES packet size mismatch
[mp2 @ 0x2e0d100] Header missing
Error while decoding stream #0:1: Invalid data found when processing input
[mpeg2video @ 0x2e0b780] ac-tex damaged at 25 34
[mpeg2video @ 0x2e0b780] Warning MVs not available
[mpeg2video @ 0x2e0b780] concealing 68 DC, 68 AC, 68 MV errors in P frame
frame= 1723 fps=1169 q=10.0 Lsize=    5619kB time=00:01:08.88 bitrate= 668.2kbits/s dup=33 drop=0 
video:4434kB audio:592kB subtitle:0 global headers:0kB muxing overhead 11.780382%

The following commands work as expected:

$ ffmpeg -ss 30 -i ntd_gap_sound.ts -qscale 10 out.ts

$ ffmpeg -skip_initial_bytes 25000000 -i ntd_gap_sound.ts -qscale 10 out.ts