heap-use-after-free in libavcodec/libaomenc.c found by AddressSanitizer

[Ayose]

AddressSanitizer found a heap-use-after-free in libavcodec/libaomenc.c.

The problem is triggered with a command like the following one, but only if FFmpeg is compiled with ASAN (./configure --toolchain=clang-asan):

$ ffmpeg -f lavfi -i testsrc -to 1 -c:v libaom-av1 /tmp/0.webm

If FFmpeg is executed without ASAN, the previous command does not fail.

Steps to Reproduce

This test is done against ab37c7e49ff98a8c32c90e6df49dd11145bc64bb, clang 19, and libaom 3.12:

$ clang --version
Debian clang version 19.1.7 (3)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-19/bin

$ pkg-config --modversion aom
3.12.1

$ git log --format='%h %cs' -1
ab37c7e49f 2025-05-30

FFmpeg is built with --toolchain=clang-asan:

$ git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg

$ cd ffmpeg

$ ./configure \
    --toolchain=clang-asan \
    --enable-debug \
    --enable-libaom

$ make -j16

Then, a heap-use-after-free is detected in libavcodec/libaomenc.c:1028:28:

$ ./ffmpeg_g -y -v warning -hide_banner -f lavfi -i testsrc -to 1 -c:v libaom-av1 /tmp/0.webm
[libaom-av1 @ 0x519000002380] Neither bitrate nor constrained quality specified, using default CRF of 32
=================================================================
==19780==ERROR: AddressSanitizer: heap-use-after-free on address 0x509000007818 at pc 0x564397e7f93a bp 0x7f27e79ad430 sp 0x7f27e79ad428
WRITE of size 8 at 0x509000007818 thread T10 (vf#0:0)
    #0 0x564397e7f939 in aom_init /tmp/ffmpeg/libavcodec/libaomenc.c:1028:28
    #1 0x564398f08f62 in avcodec_open2 /tmp/ffmpeg/libavcodec/avcodec.c:336:19
    #2 0x564398070117 in enc_open /tmp/ffmpeg/fftools/ffmpeg_enc.c:337:16
    #3 0x5643980b7938 in enc_open /tmp/ffmpeg/fftools/ffmpeg_sched.c:1686:11
    #4 0x5643980b7938 in send_to_enc /tmp/ffmpeg/fftools/ffmpeg_sched.c:1806:19
    #5 0x564398086538 in fg_output_frame /tmp/ffmpeg/fftools/ffmpeg_filter.c:2359:15
    #6 0x564398084fba in fg_output_step /tmp/ffmpeg/fftools/ffmpeg_filter.c:2460:11
    #7 0x564398084fba in read_frames /tmp/ffmpeg/fftools/ffmpeg_filter.c:2521:23
    #8 0x56439807916e in filter_thread /tmp/ffmpeg/fftools/ffmpeg_filter.c:2962:15
    #9 0x5643980b85a9 in task_wrapper /tmp/ffmpeg/fftools/ffmpeg_sched.c:2534:11
    #10 0x564398013b3a in asan_thread_start(void*) asan_interceptors.cpp.o
    #11 0x7f27f5743b7a  (/lib/x86_64-linux-gnu/libc.so.6+0x92b7a) (BuildId: b46a78e7229ed6fe08549e2bc7ca64155cc5cf1e)
    #12 0x7f27f57c15ef in clone (/lib/x86_64-linux-gnu/libc.so.6+0x1105ef) (BuildId: b46a78e7229ed6fe08549e2bc7ca64155cc5cf1e)

0x509000007818 is located 24 bytes inside of 40-byte region [0x509000007800,0x509000007828)
freed by thread T10 (vf#0:0) here:
    #0 0x564398015f4a in free (/tmp/ffmpeg/ffmpeg_g+0x734f4a) (BuildId: fc49802141dc7a2a32e6338567bec09a75fe6f79)
    #1 0x56439996eb44 in av_packet_side_data_free /tmp/ffmpeg/libavcodec/packet.c:751:9

previously allocated by thread T10 (vf#0:0) here:
    #0 0x564398016ceb in posix_memalign (/tmp/ffmpeg/ffmpeg_g+0x735ceb) (BuildId: fc49802141dc7a2a32e6338567bec09a75fe6f79)
    #1 0x56439b47a9cb in av_malloc /tmp/ffmpeg/libavutil/mem.c:107:9
    #2 0x56439b47a9cb in av_mallocz /tmp/ffmpeg/libavutil/mem.c:258:17
    #3 0x564399e2452d in av_cpb_properties_alloc /tmp/ffmpeg/libavcodec/utils.c:956:30
    #4 0x564397e7edf4 in aom_init /tmp/ffmpeg/libavcodec/libaomenc.c:992:17
    #5 0x564398f08f62 in avcodec_open2 /tmp/ffmpeg/libavcodec/avcodec.c:336:19

Thread T10 (vf#0:0) created by T0 here:
    #0 0x564397ffb6f5 in pthread_create (/tmp/ffmpeg/ffmpeg_g+0x71a6f5) (BuildId: fc49802141dc7a2a32e6338567bec09a75fe6f79)
    #1 0x5643980b2738 in task_start /tmp/ffmpeg/fftools/ffmpeg_sched.c:414:11
    #2 0x5643980e79f3 in transcode /tmp/ffmpeg/fftools/ffmpeg.c:881:11
    #3 0x5643980e79f3 in main /tmp/ffmpeg/fftools/ffmpeg.c:1009:11
    #4 0x7f27f56daca7  (/lib/x86_64-linux-gnu/libc.so.6+0x29ca7) (BuildId: b46a78e7229ed6fe08549e2bc7ca64155cc5cf1e)

SUMMARY: AddressSanitizer: heap-use-after-free /tmp/ffmpeg/libavcodec/libaomenc.c:1028:28 in aom_init
Shadow bytes around the buggy address:
  0x509000007580: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
  0x509000007600: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
  0x509000007680: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
  0x509000007700: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x509000007780: 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x509000007800: fd fd fd[fd]fd fa fa fa fa fa fa fa fa fa fa fa
  0x509000007880: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x509000007900: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x509000007980: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
  0x509000007a00: fa fa fa fa fa fa fa fa 00 00 00 00 00 fa fa fa
  0x509000007a80: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==19780==ABORTING

Possible Cause

In libavcodec/libaomenc.c:1028:28, the code tries to write to memory pointed by cpb_props:

cpb_props->buffer_size = avctx->rc_buffer_size;

cpb_props is assigned in line 992, before calling to ff_dovi_configure:

cpb_props = ff_encode_add_cpb_side_data(avctx);
if (!cpb_props)
    return AVERROR(ENOMEM);

ctx->dovi.logctx = avctx;
if ((res = ff_dovi_configure(&ctx->dovi, avctx)) < 0)
    return res;

​`ff_encode_add_cpb_side_data` returns a new array, and its address is stored in avctx->coded_side_data. ​`ff_dovi_configure` calls to ​`avcodec_parameters_to_context`, which replaces the memory of coded_side_data:

int avcodec_parameters_to_context(AVCodecContext *codec,
                                  const AVCodecParameters *par)
{
    // ...

    av_packet_side_data_free(&codec->coded_side_data, &codec->nb_coded_side_data);
    ret = codec_parameters_copy_side_data(&codec->coded_side_data, &codec->nb_coded_side_data,
                                          par->coded_side_data, par->nb_coded_side_data);

    // ...
}

With GDB I can see that the memory stored in the cpb_props variable is freed by av_free:

$ gdb --args ./ffmpeg_g -y -v warning -hide_banner -f lavfi -i testsrc -to 1 -c:v libaom-av1 /tmp/0.webm
[...]

(gdb) break libavcodec/libaomenc.c:993

(gdb) commands
>printf "cpb_props = %p\n", cpb_props
>c
>end

(gdb) break libavcodec/packet.c:751

(gdb) commands
>printf "av_free: %p\n", sd[i].data
>c
>end

(gdb) r
[...]

Thread 18 "vf#0:0" hit Breakpoint 1, aom_init (avctx=0x519000002880, iface=<optimized out>)
    at libavcodec/libaomenc.c:993
993	    if (!cpb_props)
cpb_props = 0x509000007800

Thread 18 "vf#0:0" hit Breakpoint 2, av_packet_side_data_free (psd=psd@entry=0x519000002b88,
    pnb_sd=pnb_sd@entry=0x519000002b90) at libavcodec/packet.c:751
751	        av_free(sd[i].data);
av_free: 0x509000007800

Thread 18 "vf#0:0" hit Breakpoint 2, av_packet_side_data_free (psd=psd@entry=0x51100000d9e0,
    pnb_sd=pnb_sd@entry=0x51100000d9e8) at libavcodec/packet.c:751
751	        av_free(sd[i].data);
av_free: 0x509000007900

=================================================================
==165799==ERROR: AddressSanitizer: heap-use-after-free on address 0x509000007818 at pc 0x555555b3d998 bp 0x7fffdfe67090 sp 0x7fffdfe67088

[...]

Possible Fix

I modified libavcodec/libaomenc.c to initialize cpb_props after the call to ff_dovi_configure:

libavcodec/libaomenc.c

@@ -989 +989 @@
     if (codec_caps & AOM_CODEC_CAP_HIGHBITDEPTH)
         ctx->rawimg.bit_depth = enccfg.g_bit_depth;
 
-    cpb_props = ff_encode_add_cpb_side_data(avctx);
-    if (!cpb_props)
-        return AVERROR(ENOMEM);
-
     ctx->dovi.logctx = avctx;
     if ((res = ff_dovi_configure(&ctx->dovi, avctx)) < 0)
         return res;
 
+    cpb_props = ff_encode_add_cpb_side_data(avctx);
+    if (!cpb_props)
+        return AVERROR(ENOMEM);
+
     if (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER) {
         const AVBitStreamFilter *filter = av_bsf_get_by_name("extract_extradata");
         int ret;

With the change, the command works with no issues:

$ ./ffmpeg_g -y -v warning -hide_banner -f lavfi -i testsrc -to 1 -c:v libaom-av1 /tmp/0.webm
[libaom-av1 @ 0x519000002380] Neither bitrate nor constrained quality specified, using default CRF of 32

$ ./ffprobe -hide_banner -i /tmp/0.webm
Input #0, matroska,webm, from '/tmp/0.webm':
  Metadata:
    ENCODER         : Lavf62.0.102
  Duration: 00:00:01.00, start: 0.000000, bitrate: 33 kb/s
  Stream #0:0: Video: av1 (libaom-av1) (High), gbrp(pc, gbr/bt709/iec61966-2-1, progressive), 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 1k tbn
    Metadata:
      ENCODER         : Lavc62.3.101 libaom-av1
      DURATION        : 00:00:01.000000000

$

Though I don't know if this change may break something else.

[mkver]

Fixed in 8d45dc858ecfcd615e0c517b81034f300e3f9832.