Opened 5 years ago

Last modified 3 years ago

#2445 new defect

Live audio/video gets out of sync when the output (file, network, pipe) lags or is too slow.

Reported by: Max-P Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Hello everyone, first please excuse my lenghty bug report. I'm trying to be as presise and helpful as possible, please correct me if I miss anything here.

Context: I was trying to stream a game from my desktop with the audio that comes with it, but always got a random delay with the audio. I couldn't get it working right, so I searched Google a lot and kind of figured the problem by taking apart each step of the encoding.

Summary of the bug: When converting live sources (here x11grab and pulse sources), the audio gets out of sync with the video no matter which settings are used (-vsync, -af aresample, -async, -isync, -af asetpts=PTS-STARTPTS, -vf setpts=PTS-STARTPTS, none of them change anything). I found that this is due to the output being unresponsive or too slow at start, because it stays in sync when I output to a file, and I found a way to reliably tell it's not the tcp:// or rtmp:// protocol of FFmpeg that causes problems.

How to reproduce:

In order to simulate the buggy or laggy output, I made three C programs (attached with the ffmpeg -report log to this ticket), and all examples were ran with this command. I understand it's not as good, but it allows me to rule out any FFmpeg internal protocol bugs:

% ffmpeg -report \
	-f x11grab -s 1280x720 -r 30 -show_region 1 -i $DISPLAY \
	-f pulse -ac 2 -ar 44100 -i default \
	-vcodec libx264 -b:v 500k -preset veryfast \
	-pix_fmt yuv420p -threads 4 -r 30 -s 1280x720 \
	-acodec libmp3lame -ab 192k \
	-f flv -y - | tee test.flv | ./slow

So, to explain why I do this: I tell the first ffmpeg instance to output to stdout, so I can pipe it to any other program I want. I did this, because I noticed local files were always in perfect sync without any trouble, but as soon as I tried to pipe a ffmpeg to another ffmpeg to get it to my server via RTMP, it got out of sync again.

So I pipe it to tee, which will output the result to a local file on disk, and pipe it again to another program.

1- I pipe it to ./void, which is the program in void.c. It reads STDIN as fast as it can. The resulting test.flv file is in perfect sync for as long as I tried it (5 minutes). It works the exact same was as just letting ffmpeg outputting to test.flv directly. This is the expected behavior.

2- I pipe it to ./slow, which is a variant of the void.c program, but it wait 3 seconds before reading the data, reads it all and continue reading it a bit, then wait 3 seconds again and repeat over and over forever. I was lazy, so it doesn't do proper rate limiting, but it's good enough for the tests. It simulated periodic lags in the output, which seems to happen a lot when streaming to slow networks or servers. The resulting test.flv file gets progressively out of sync. After only 20 seconds, the audio is already a second off from the video, and it gets worse and worse. The ffmpeg command is left untouched the whole time, only the piped command is changed. This lets me suppose that when the output buffer is completely filled, the input buffers will get filled, but one of the video or audio buffers can hold more data than the other, so when the output clears out, both sources are now out of sync.

3- Just for testing sake, I piped it to my last program, idelay.c, which simply waits 10 seconds before reading as fast as it can like the void.c program. It simulates an initial network setup delay, which is possibly what rtmp:// does for me and gets my stream all out of sync. The output test.flv file is out of sync by a second or two, but delay seems constant afterward.

4- Of course, piping to "ffmpeg -i - -codec copy -f flv rtmp://my.server.tld" causes the same initial delay, which breaks my live stream just as much as using rtmp://server directly in the original ffmpeg command.

I seriously hope this is clear enough, I'm doing my best :) If someone can give me pointers on where to look at, I could also do some test, C is not my best language but I sure can dig into it if needed.

Attachments (4)

ffmpeg-20130407-151441.log (272.8 KB) - added by Max-P 5 years ago.
ffmpeg -report log
void.c (79 bytes) - added by Max-P 5 years ago.
The void.c program that drops all its input
slow.c (233 bytes) - added by Max-P 5 years ago.
The slow.c program that simulates netowork lag on the output.
idelay.c (113 bytes) - added by Max-P 5 years ago.
The idelay.c program that waits 10 seconds before dropping all it input, simulating a network connection delay.

Download all attachments as: .zip

Change History (5)

Changed 5 years ago by Max-P

ffmpeg -report log

Changed 5 years ago by Max-P

The void.c program that drops all its input

Changed 5 years ago by Max-P

The slow.c program that simulates netowork lag on the output.

Changed 5 years ago by Max-P

The idelay.c program that waits 10 seconds before dropping all it input, simulating a network connection delay.

comment:1 Changed 3 years ago by ferdi265

I have had that problem as well. Using your setup (using other ffmpeg parameters but the same piping setup) I was able to reproduce the problem.

My command:

ffmpeg -y \
	-f pulse -i alsa_input.pci-0000_00_14.2.analog-stereo \
	-f pulse -i alsa_output.pci-0000_00_14.2.analog-stereo.monitor \
	-filter_complex amerge \
	-f x11grab -s 1280x1024 -r 24 -i :0.0 \
	-c:a libmp3lame -ac 2 -ar 44100 \
	-c:v libx264 -b:v 2000k -minrate 2000k -maxrate 2000k -bufsize 2000k \
	-g $((24 * 2)) -vf format=yuv420p -crf 22 \
	-threads 4 -preset veryfast -strict normal \
	-f flv - | tee test.flv | ./idelay

FFMPEG version:

ffmpeg version 2.4.3-1ubuntu1~utopic1 Copyright (c) 2000-2014 the FFmpeg developers
  built on Nov 15 2014 20:19:29 with gcc 4.9.1 (Ubuntu 4.9.1-16ubuntu6)
  configuration: --prefix=/usr --extra-version='1ubuntu1~utopic1' --build-suffix=-ffmpeg --toolchain=hardened --extra-cflags= --extra-cxxflags= --libdir=/usr/lib/x86_64-linux-gnu --shlibdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --enable-shared --disable-stripping --enable-avresample --enable-avisynth --enable-fontconfig --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-opengl --enable-x11grab --enable-libxvid --enable-libdc1394 --enable-libiec61883 --enable-libzvbi --enable-libzmq --enable-frei0r --enable-libx264 --enable-libsoxr --enable-openal --enable-libopencv
  libavutil      54.  7.100 / 54.  7.100
  libavcodec     56.  1.100 / 56.  1.100
  libavformat    56.  4.101 / 56.  4.101
  libavdevice    56.  0.100 / 56.  0.100
  libavfilter     5.  1.100 /  5.  1.100
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  0.100 /  3.  0.100
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  0.100 / 53.  0.100
Note: See TracTickets for help on using tickets.