A/V desync when transcoding A/V files with timestamps non starting from 0 with -copyts

[Stefano Sabatini]

Using the command:

ffmpeg -vsync 0 -copyts -i columbia-intro.ts -acodec libfaac -y columbia-intro.out.ts

(same without -vsync) the A/V streams in the output file result desynched, and will result in the video being unplayable. This seems due to the first audio packet timestamp information not being correctly propagated to the encoder (and to the muxer), and being set to 0, so that ffmpeg will fill the first part of the file with audio packets, and the probe will fail since ffmpeg can't detect video frames in the probed initial part of the file.

Same result in case I use the -copyts with no -vsync. If the output file only contains video (e.g. if the -an switch is used) the output file seems to be correctly generated (playable and with correct timestamp information).

I'm attaching a sample file and log obtained by adding debugging print in ffmpeg.c.

[Stefano Sabatini]

log file containing packets/frames timestamp information at various transcoding stages

[Stefano Sabatini]

Added the same file two times, please remove the duplicate.

[Carl Eugen Hoyos]

Please provide the complete, uncut console output together with your command line.

[Stefano Sabatini]

Replying to cehoyos:

Please provide the complete, uncut console output together with your command line.

Not that I think that the console output gives more hints, but here it is:

$ ffmpeg -copyts -vsync 0 -an -i columbia-intro.ts -acodec libfaac -y columbia-intro.out.ts -nostats
ffmpeg version N-37013-ge7321ef Copyright (c) 2000-2012 the FFmpeg developers
  built on Jan 22 2012 17:39:37 with gcc 4.6.1
  configuration: --enable-version3 --enable-avconv --prefix=/home/stefano --enable-librtmp --enable-libx264 --enable-libfaac --disable-shared --enable-static --enable-debug=3 --enable-pthreads --enable-libvorbis --enable-gpl --enable-nonfree --enable-libmp3lame --enable-libtheora --enable-gpl --enable-x11grab --enable-frei0r --disable-optimizations --disable-mmx --enable-libfreetype --enable-libopencv --enable-libopencore-amrnb --enable-libopencore-amrnb --enable-libass
  libavutil      51. 34.101 / 51. 34.101
  libavcodec     53. 57.105 / 53. 57.105
  libavformat    53. 30.100 / 53. 30.100
  libavdevice    53.  4.100 / 53.  4.100
  libavfilter     2. 59.101 /  2. 59.101
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0.  6.100 /  0.  6.100
  libpostproc    52.  0.100 / 52.  0.100
[mpegts @ 0x1a5dd80] max_analyze_duration 5000000 reached at 5000000
Input #0, mpegts, from 'columbia-intro.ts':
  Duration: 00:00:08.03, start: 12.928000, bitrate: 1005 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, s16, 176 kb/s
    Stream #0:2[0x102]: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, s16, 12 kb/s
[buffer @ 0x1a641d0] w:720 h:576 pixfmt:yuv420p tb:1/1000000 sar:64/45 sws_param:
[mpegts @ 0x1a633d0] muxrate VBR, pcr every 2 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mpegts, to 'columbia-intro.out.ts':
  Metadata:
    encoder         : Lavf53.30.100
    Stream #0:0: Video: mpeg2video, yuv420p, 720x576 [SAR 64:45 DAR 16:9], q=2-31, 200 kb/s, 90k tbn, 25 tbc
    Stream #0:1: Audio: aac, 48000 Hz, 5.1, s16, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> mpeg2video)
  Stream #0:1 -> #0:1 (aac -> libfaac)
Press [q] to stop, [?] for help
frame=  200 fps= 44 q=31.0 Lsize=     717kB time=00:00:00.00 bitrate=   0.0kbits/s    
video:642kB audio:0kB global headers:0kB muxing overhead 11.598826%

[Carl Eugen Hoyos]

Is this only reproducible using an external library (libfaac) or also with one of the native audio encoders?

What happens without -copyts? (Why are you using -copyts?)

If -vsync is not needed to reproduce the problem, please remove it from your command line.

[Mike]

From what I can see in the code, -copyts does the right thing only when you also use '-acodec copy'. This command line works for me:

./ffmpeg -vsync 0 -copyts -i columbia-intro.ts -vcodec mpeg2video -b:v 3000000 -acodec copy out.ts

...and the proper time offset between the first audio and video packets is maintained.

So, the short answer is that ffmpeg (the app) doesn't yet support what you want to do.

The more detailed answer is that, if you try to actually encode the audio stream, ffmpeg.c looks like it always calls avcodec_encode_audio*() with the AVFrame pts timestamp set to AV_NOPTS_VALUE. Later on, this will cause av_interleaved_write_frame() to make up timestamps, starting at 0.

[Mike]

One other thing to be aware of: the mpegts mux does not honor the -copyts flag, and adds it's own constant offset to every timestamp.

[Stefano Sabatini]

First of all thanks for the reply, I confirm that -copyts -acodec copy works just fine, and fixes my particular problem.

Still the problem stands when audio transcoding is required, from my analysis it looks like a problem in the ffmpeg.c code (decoded audio frame timestamp is ignored).

[Carl Eugen Hoyos]

I was unable to reproduce the original problem - iiuc that -vsync 0 together with a start time >0 leads to unplayable output files because audio timestamps are incorrect - with both current FFmpeg and an ancient version. Testing the same version as the OP is impossible, "e7321ef" does not exist.

[Stefano Sabatini]

Command:

ffmpeg -copyts -i ~/s/columbia-intro.ts -acodec libfaac -y columbia-intro.out.ts
ffmpeg version N-46969-g64dc383 Copyright (c) 2000-2012 the FFmpeg developers
  built on Nov 21 2012 20:32:53 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
  configuration: --enable-pic --enable-fontconfig --enable-libschroedinger --enable-libass --enable-version3 --prefix=/home/stefano --enable-librtmp --enable-libx264 --enable-libfaac --disable-shared --enable-static --enable-debug=3 --enable-pthreads --enable-libvorbis --enable-gpl --enable-nonfree --enable-libmp3lame --enable-libtheora --enable-gpl --enable-x11grab --enable-frei0r --enable-libspeex --enable-libcaca --enable-libflite --enable-libfreetype --enable-libopencv --enable-libopencore-amrnb --enable-libopencore-amrnb --disable-optimizations --disable-mmx
  libavutil      52.  8.100 / 52.  8.100
  libavcodec     54. 74.100 / 54. 74.100
  libavformat    54. 37.100 / 54. 37.100
  libavdevice    54.  3.100 / 54.  3.100
  libavfilter     3. 23.101 /  3. 23.101
  libswscale      2.  1.102 /  2.  1.102
  libswresample   0. 17.101 /  0. 17.101
  libpostproc    52.  2.100 / 52.  2.100
[mpegts @ 0x1ba1950] max_analyze_duration 5000000 reached at 5000000
Input #0, mpegts, from '/home/stefano/s/columbia-intro.ts':
  Duration: 00:00:08.03, start: 12.928000, bitrate: 1005 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, s16, 176 kb/s
    Stream #0:2[0x102]: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, s16, 12 kb/s
Output #0, mpegts, to 'columbia-intro.out.ts':
  Metadata:
    encoder         : Lavf54.37.100
    Stream #0:0: Video: mpeg2video, yuv420p, 720x576 [SAR 64:45 DAR 16:9], q=2-31, 200 kb/s, 90k tbn, 25 tbc
    Stream #0:1: Audio: aac, 48000 Hz, 5.1, s16, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> mpeg2video)
  Stream #0:1 -> #0:1 (aac -> libfaac)
Press [q] to stop, [?] for help
frame=  525 fps= 55 q=31.0 Lsize=    1545kB time=00:00:20.96 bitrate= 604.0kbits/s dup=325 drop=0    
video:1172kB audio:188kB subtitle:0 global headers:0kB muxing overhead 13.591660%

Now if you ffprobe the file with:

ffprobe columbia-intro.out.ts -show_packets -of compact | less

you see there are a lot of video packets at the beginning, and the first audio packet is in:

packet|codec_type=audio|stream_index=1|pts=1287600|pts_time=14.306667|dts=1287600|dts_time=14.306667|duration=N/A|duration_time=N/A|convergence_duration=N/A|convergence_duration_time=N/A|size=448|pos=910108|flags=K

The file is now playable with ffplay (I remember it wasn't), but if you ffprobe the file you get:

ffprobe columbia-intro.out.ts 
ffprobe version N-46969-g64dc383 Copyright (c) 2007-2012 the FFmpeg developers
  built on Nov 21 2012 20:32:53 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
  configuration: --enable-pic --enable-fontconfig --enable-libschroedinger --enable-libass --enable-version3 --prefix=/home/stefano --enable-librtmp --enable-libx264 --enable-libfaac --disable-shared --enable-static --enable-debug=3 --enable-pthreads --enable-libvorbis --enable-gpl --enable-nonfree --enable-libmp3lame --enable-libtheora --enable-gpl --enable-x11grab --enable-frei0r --enable-libspeex --enable-libcaca --enable-libflite --enable-libfreetype --enable-libopencv --enable-libopencore-amrnb --enable-libopencore-amrnb --disable-optimizations --disable-mmx
  libavutil      52.  8.100 / 52.  8.100
  libavcodec     54. 74.100 / 54. 74.100
  libavformat    54. 37.100 / 54. 37.100
  libavdevice    54.  3.100 / 54.  3.100
  libavfilter     3. 23.101 /  3. 23.101
  libswscale      2.  1.102 /  2.  1.102
  libswresample   0. 17.101 /  0. 17.101
  libpostproc    52.  2.100 / 52.  2.100
[mpegts @ 0x34fe270] max_analyze_duration 5000000 reached at 5000000
[mpegts @ 0x34fe270] Could not find codec parameters for stream 1 (Audio: aac ([15][0][0][0] / 0x000F), 0 channels, s16): unspecified sample rate
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[NULL @ 0x35027e0] start time is not set in estimate_timings_from_pts
Input #0, mpegts, from 'columbia-intro.out.ts':
  Duration: 00:00:20.96, start: 1.400000, bitrate: 604 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: aac ([15][0][0][0] / 0x000F), 0 channels, s16

so the problem is still there.

[Carl Eugen Hoyos]

Is this only reproducible with -acodec libfaac or also with -strict -2 -acodec aac ?

[Stefano Sabatini]

Replying to cehoyos:

Is this only reproducible with -acodec libfaac or also with -strict -2 -acodec aac ?

Tracked the problem to do_video_out() in ffmpeg.c, if vsync != passthrough the input value is ignored (this is significantly different from what happened 10 months ago), with the command:

ffmpeg -vsync passthrough -copyts -i columbia-intro.ts -acodec libfaac -y columbia-intro.out.ts

I got synched timestamps, yet there is a mismatch of around ~1s between input and output timestamps.
Also documentation for -copyts should be fixed (it only disables input timestamps processing).

[Carl Eugen Hoyos]

Replying to saste:

ffmpeg -vsync passthrough -copyts -i columbia-intro.ts -acodec libfaac -y columbia-intro.out.ts

Is this only reproducible with -acodec libfaac or also with -strict -2 -acodec aac ?

[Roger Pack]

Is this still a problem?