Crackling when concatenating audio introduced in commit d85c6aba

[jonnyburger]

Summary of the bug:

We found that when upgrading to FFmpeg 6.1, a regression in audio quality has occurred when concatenating PCM-16 audio chunks into AAC.

After bisecting 2000 commits, we find that the behavior has changed in ​https://github.com/FFmpeg/FFmpeg/commit/d85c6aba0cf27db2a6c4dfa3452cfb9c248d1b4a. This commit seems to have changed the test fixtures as well, indicating that it makes inadverted changes to the output.

As of writing, the crackling is still on master.

Important detail: The crackling is only audible on QuickTime on macOS with a relatively high volume. Playing the video for example in a browser does not result in audio artifacts.

Repro: ​https://jonnyburger.s3.eu-central-1.amazonaws.com/crackling.zip
Also uploaded as crackling.zip on the VideoLAN file uploader.

To reproduce, you need to download the PCM chunks and concat them using:

ffmpeg -loglevel verbose -r 30 -f concat -safe 0 -i files.txt -c:v copy -c:a aac -b:a 320k output.mp4

Since the codebase has further evolved and it's not possible to just revert the commit, I derived a hacky fix for anyone else facing this issue (might break something else): ​https://jonnyburger.s3.eu-central-1.amazonaws.com/fix-aac.patch

[jonnyburger]

For those interested, there is an even narrower patch:

diff --git a/fftools/ffmpeg_dec.c b/fftools/ffmpeg_dec.c
index fcee8b65ac..07edcef4f2 100644
--- a/fftools/ffmpeg_dec.c
+++ b/fftools/ffmpeg_dec.c
@@ -214,7 +214,7 @@ static void audio_ts_process(void *logctx, Decoder *d, AVFrame *frame)
     // on samplerate change, choose a new internal timebase for timestamp
     // generation that can represent timestamps from all the samplerates
     // seen so far
-    tb = audio_samplerate_update(logctx, d, frame);
+    tb = frame->time_base;
     pts_pred = d->last_frame_pts == AV_NOPTS_VALUE ? 0 :
                d->last_frame_pts + d->last_frame_duration_est;

Unfortunately I don't have the expertise to understand what is going on and submit a proper patch to FFmpeg.

[Marton Balint]

chunk6 is missing from the sample, so can't really reproduce. Also I suggest try reproducing with sine wave as source, much easier to hear any cracks. What is returned from audio_samplerate_update and what is in frame->time_base?

[Balling]

when concatenating PCM-16 audio chunks into AAC.

Does it happen if you just get wav pcm? No encode to aac.

Marton... You can just copy paste one of the segments into file called chunk_00000006, seriously?

[Balling]

Anyway I cannot reproduce that. Looking it up in Audacityy the segments all got concatenated together correctly.

[jonnyburger]

Thanks a lot for looking into it!
I am working on providing a better reproduction and answering your questions.
Unfortunately chunk000006 had a trailing whitespace.
I find the audio sounds different in Chrome than it does in QuickTime.
It indicates a problem with the PTS, not with the waveform I suppose.

I’ll follow up on this with a better example.

[jonnyburger]

I have now uploaded a second repro!

Filename: reproduction-two.zip
Alternate URL: ​https://jonnyburger.s3.eu-central-1.amazonaws.com/reproduction-two.zip

This one uses a sine sound.
It also only leads to audio artifacts in QuickTime.

I also compiled a version of FFmpeg n6.1 tag with the following patch applied:

diff --git a/fftools/ffmpeg_dec.c b/fftools/ffmpeg_dec.c
index fcee8b65ac..3b27e15279 100644
--- a/fftools/ffmpeg_dec.c
+++ b/fftools/ffmpeg_dec.c
@@ -215,6 +215,9 @@ static void audio_ts_process(void *logctx, Decoder *d, AVFrame *frame)
     // generation that can represent timestamps from all the samplerates
     // seen so far
     tb = audio_samplerate_update(logctx, d, frame);
+    av_log(logctx, AV_LOG_WARNING,
+                "Remotion Repro: tb: %d sr: %d\n", tb, frame->sample_rate);
+
     pts_pred = d->last_frame_pts == AV_NOPTS_VALUE ? 0 :
                d->last_frame_pts + d->last_frame_duration_est;

It prints out:

Remotion Repro: tb: 1 sr: 48000

Is it bad to use %d? Both with this and the fix I got the same log.
I can do more logging if you like me to.

I also added my compiled version of FFmpeg in the ZIP.

[jonnyburger]

I managed to properly log now:

frame->time_base: 1/1000
audio_samplerate_update(logctx, d, frame) -> 1/48000

[Balling]

Can you attach two aac encoded files, after and before the patch?

[jonnyburger]

Comparing output of FFmpeg 6.0 and 6.1

[jonnyburger]

Absolutely, here is the difference between the outputs. 6_0.mp4 sounds good, 6_1 does not. The change in source code is only:

diff --git a/fftools/ffmpeg_dec.c b/fftools/ffmpeg_dec.c
index fcee8b65ac..07edcef4f2 100644
--- a/fftools/ffmpeg_dec.c
+++ b/fftools/ffmpeg_dec.c
@@ -214,7 +214,7 @@ static void audio_ts_process(void *logctx, Decoder *d, AVFrame *frame)
     // on samplerate change, choose a new internal timebase for timestamp
     // generation that can represent timestamps from all the samplerates
     // seen so far
-    tb = audio_samplerate_update(logctx, d, frame);
+    tb = frame->time_base;
     pts_pred = d->last_frame_pts == AV_NOPTS_VALUE ? 0 :
                d->last_frame_pts + d->last_frame_duration_est;