A/V drift when reencoding NTSC

[Timothy Pearson]

Summary of the bug:

I have been having significant difficulty with A/V sync drift in ffmpeg (live streaming applications), which unfortunately do not lend themselves well to debugging by most ffmpeg developers. In the process of debugging the live stream, I was able to reproduce the A/V sync problems with the attached input file, which was specifically built to find A/V desync issues.

The input file is very simple. It is a 24000/1001 (NTSC) MPEG2 sync check file, with every 24th frame showing a coinciding tone and white flash. This file was assembled by mencoder, and I have verified that the audio tones are sent exactly once every 1.001 seconds in the output file.

The ffmpeg command below simply overlays the audio waveform back over the video frames. If ffmpeg was processing this input file correctly, each white frame should have the audio tone starting at exactly the same position. Unfortunately, as you can see, the audio tone in fact drifts significantly (in fact it often ends up in the adjacent frame). In addition to this obvious desync with ~4.004s periodicity, there is a very slight continual desync of the audio even when only looking at the output every 4.004s. Interestingly, 4.004s is the normal self-sync period for audio and video frames in NTSC content, indicating something may be wrong with how the underlying A/V frames (not video frames!) are handled / decoded / encoded.

How to reproduce:

% ffmpeg -y -copyts -i sample.mpeg -filter_complex "[0:a] showwaves=s=720x240:colors=Red|Green|Blue|Yellow:split_channels=1:mode=cline [waveview]; [0:v][waveview] overlay=0:H/2" --acodec copy -f dvd output.mpeg

ffmpeg built from GIT master

Note that the attached segment file was trivially reduced with dd, as the original file is far too large to attach. This should not affect the results given that this is a transport stream file.

[Timothy Pearson]

Example output showing audio drift

[Timothy Pearson]

Snippet of sync check file, starting with beginning

[Timothy Pearson]

Another example showing the desync is not container dependent

[Carl Eugen Hoyos]

Please provide the command line you tested together with the complete, uncut console output to make this a valid ticket.

[Timothy Pearson]

I provided the full command above. What console output do you need? There were no errors or warnings on the console; this is very easy to run and see the (incorrect) results.

[pdr0]

check ffmpeg output file's AV sync

[pdr0]

The drift is only affecting either showwaves or overlay.

The file "output.mp4" flash frames and audio waveform are in AV sync with other checks, verified with several tools. eg. You can the output.mp4 audio waveform on a NLE ,or avisynth.

added output.mp4 check

[Timothy Pearson]

Thank you for investigating this. The ffmpeg showwaves issue affects my normal NLE (kdenlive), and I don't use Windows so avisynth is not normally an option for me.

[pdr0]

Yes, showwaves and/or overlay is affected .

But the actual file AV sync is ok (white flashes in the video sync with audio waveform) .

Just watching it in a media player, can you detect desync with the white flashes ? There is a disconnect with showwaves (possibly showwaves is not accurate).

Or are you saying the track audio waveform in kdenlive (not showwaves) also goes out of sync with the white flashes in kdenlive ?

This is not a good diagnostic test for your other livestream AV sync issue - since there appears to be some accuracy issue with showwaves

[Timothy Pearson]

Yes, I understand now that showwaves is not going to be able to help diagnose the other A/V sync issue due to this bug. I'm still searching for an alternate method to view per-frame A/V sync using open source software, without much luck so far.

kdenlive apparently uses ffmpeg for its audio waveform display, and is very much affected by the same bug. This also added some confusion.

The live streaming bug manifests as a desync of several hundred ms over a few hours, and is not practical to debug until a reliable way is found to measure the small (one or two ms) desync that would occur over a much shorter period of time. I have an initial hardware method to detect the output sync within some hundreds of microseconds accuracy, but since I cannot measure the *input* file A/V sync (yet) I am kind of stuck.

[Timothy Pearson]

Regarding just watching it, it's really hard to tell. The perceptual limits on the human visual system are often quoted as ~40ms advanced, ~100ms retarded (audio against video) so this is not a reliable way to detect A/V sync issues until they are quite severe. In fact I am attempting to debug a pipeline where the vendor did exactly this, relied on visible desync with predictably bad results with long-running streams.

[pdr0]

Another crossplatform tool would be Blender's video sequence editor. You can see the flash frames align with the audio waveform there too like other commercial NLE's (at least on that short 30s output.mp4 sample)

The other question is why isn't showwaves producing the expected output ? I tried some of the options like setting n, r and resetting timebase, but couldn't get it to work properly

[Timothy Pearson]

Thanks for the tip about Blender, that worked quite well to verify the input files!

[Timothy Pearson]

With Blender I was able to verify the live A/V stream desync, and that it is shown in this example.

If you pull each 96th flash frame (to avoid the obvious audio drifting), and look really carefully at the overlaid audio, you can see it's very slowly inching to the left (i.e. slowly advancing compared to the video). I can confirm the exact same problem with the live stream, both in magnitude and direction, so this test case seems to show two distinct issues within ffmpeg. For testing the live stream, I used ffmpeg to record the stream and loaded it into Blender, then observed the very slow A/V desync accumulate over time.

[pdr0]

I don't follow you - did you mean on a longer sample, or "output.mp4" ?

What overlaid audio ? Are we looking at the same thing ?

To be clear - I'm looking at the audio track waveform, on the actual track. Make sure view=> waveform drawing => waveforms on is checked. Not the showwaves waveform burned into the video.

Since the audio tone occurs every second, and since the video framerate is 23.976fps (~ 41.7ms / frame) - you cannot express whole seconds perfectly everytime (unlike something like 25.0 fps). So the audio waveform position might be slightly off a few ms each flash - you should expect that since it's not a integer multiple.

But I'm not seeing drifting in that 30s sample. Drifting implies progressively worsening. This means the largest delta should be been first and last. I'm seeing the expected fwe ms variation in relation to the flash frame. If it's a bit to the left, it's a bit to the right a bit later. Progressive drift implies always 1 direction

You need to check longer sample. If it eventually shift 1 frame by a certain time

Another possible complication is blender might not be the best tool for this either; it usually deals with image sequences. It can have problems with long GOP formats with non linear seeks (sometimes frames get mixed up)

[Timothy Pearson]

Sorry, wasn't clear. Yes, much longer sample (hours). Drift was progressive.

I'm currently running a long test to see if it's the encoding side affected or, ironically, if I fixed the encoding side some months back (through a bit of hacking to force proper hardware timestamps, etc. in a different ffmpeg tree than the one I'm testing with here), then got messed up by this bug affecting the recording / waveform extraction.

[Timothy Pearson]

My tests with long running streams are showing minor drift back and forth around proper sync, but no long-term desync using my patched ffmpeg copy that passes the hardware timestamps straight through into the encoders for both video and audio. The drift seems to be up to a frame either side, which at 24P is still a fairly serious issue. VLC seems to be a reliable witness for the A/V sync issues on 1/4 playback speed, as is Blender.

It would be useful to get the showwaves issue corrected; at minimum, it may shed some light on why there is minor sync "wobble" over time with the live streams. Getting this fixed would also repair kdenlive.

[Carl Eugen Hoyos]

I am unable to reproduce A/V sync issues with the attached samples and the minimal needed information is missing, feel free to reopen if you want to add it.

[Timothy Pearson]

Please elaborate, pdr0 was able to reproduce and I have provided enough information to reproduce. What else do you need?

[Carl Eugen Hoyos]

The minimum required information for a valid ticket on this bug tracker is the tested command line including the complete, uncut console output, remember that only current FFmpeg git head is supported.