"avfoundation" audio capture had missing samples randomly

[drobinson]

Summary of the bug:

I am using macOS 15.1 (Sequoia) on a 2021 Apple M1 Pro MacBook Pro 14-inch laptop, running FFmpeg version 7.1 installed via Homebrew.

Audio captured using AVFoundation through any audio codec (AAC, MP3, PCM, or -c:a copy) produces audible glitches in the form of missing samples. This issue occurs regardless of the sound device selected (built-in audio, USB audio, etc.).

Originally, I ran into this issue while attempting to capture both video and audio and encode them using libx264 and AAC. However, I found that the issue persists even in the most basic use case—just recording audio without video encoding.

Steps to Reproduce:

1. Record a known waveform with the following command:

ffmpeg -f avfoundation -i ":0" -c:a copy output.wav

2. Listen to output.wav. I notice frequent random audible ticks.

3. Inspect the waveform of output.wav in an audio editing tool (e.g., Reaper). I observed that the waveform exhibits discontinuities, where a sequence of input samples, such as (1, 2, 3, 4, 5, 6, 7, 8, 9), results in an output sequence like (1, 2, 3, 8, 9). In this example, samples 4, 5, 6, and 7 are missing, and sample 8 immediately follows sample 3. These missing samples occur randomly, typically every 3000-10000 samples, with no discernible pattern.

Here is an example of a ramp waveform I used with the glitch circled in red:

And up close:

Expected Behavior:
The output audio should accurately represent the input without any missing or skipped samples.

Actual Behavior:
There are consistent audio dropouts, where specific sections of audio are completely missing, resulting in a glitchy sound. This happens with every audio codec tested (AAC, MP3, PCM) and with different sound devices (built-in, USB, etc.).

Details:

% ffmpeg -f avfoundation -i ":0" -c:a copy output.wav
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with Apple clang version 16.0.0 (clang-1600.0.26.4)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1_4 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100

Input #0, avfoundation, from ':0':
  Duration: N/A, start: 10114.532792, bitrate: 3072 kb/s
  Stream #0:0: Audio: pcm_f32le, 48000 Hz, stereo, flt, 3072 kb/s
File 'output.wav' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Output #0, wav, to 'output.wav':
  Metadata:
    ISFT            : Lavf61.7.100
  Stream #0:0: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, stereo, flt, 3072 kb/s
Press [q] to stop, [?] for help
size=    1792KiB time=00:00:07.71 bitrate=1903.5kbits/s speed=1.28x 

[MasterQuestionable]

͏    Reference:
͏    https://ffmpeg.org/ffmpeg-devices.html#avfoundation

͏    AVFoundation appears to be Apple specific.

[drobinson]

I had a chance to do some more investigation.

It seems like the issue is related to captureOutput:didOutputSampleBuffer:fromConnection callback in the AVFAudioReceiver class being called before the avf_read_packet function and freeing/overwriting the current_audio_frame before it is read.

I am able to avoid the dropouts by creating a FIFO for the received frames.

Not being that familiar with this codebase or AVFoundation, I am not sure if this is a reasonable fix or not (I suppose there could be latency/alignment fallout), but the avfoundation_audio_fifo.patch attached patch seems to resolve the issue for me.

[drobinson]

When the frame list overflows, advance the read index by half the FIFO length

[MasterQuestionable]

͏    Difference:

.patch

@@ -18 +18 @@
 +static void push_frame(AVFFrameList* list, CMSampleBufferRef frame)
 +{
 +    list->frames[list->write_index] = (CMSampleBufferRef)CFRetain(frame);
 +    list->write_index = (list->write_index + 1) % AVF_FRAME_LIST_LENGTH;
 +
 +    if (list->write_index == list->read_index) {
-+        CFRelease(list->frames[list->read_index]);
-+        list->read_index = (list->read_index + 1) % AVF_FRAME_LIST_LENGTH;
++        for (int i = 0; i < (AVF_FRAME_LIST_LENGTH / 2); ++i) {
++            CFRelease(list->frames[list->read_index]);
++            list->read_index = (list->read_index + 1) % AVF_FRAME_LIST_LENGTH;
++        }
 +    }
 +}

[Justin Taylor]

Hi there, also experiencing this issue. The only way to resolve it is to roll back to ffmpeg 4.1 which isn't ideal.

Has this patch been integrated into the main build yet?

[Aman]

drobinson could you submit this patch to the ffmpeg-devel mailing list so it can be reviewed

[Gyan]

@drobinson, can you open a PR at code.ffmpeg.org with avfoundation_audio_fifo-v2.patch​?

I'll do it myself within a week if there's no reply.

[drobinson]

Sorry for the delay, this fell off my radar. I can try to submit a pull request this weekend.

[drobinson]

​https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20871