AAC encoder crash/invalid read

[Kieran Kunhya]

Sometimes this causes a segfault, sometimes just invalid reads - haven't managed to get a backtrace of the segfault

​http://paste.ubuntu.com/12817025/

File: ​http://obe.tv/Downloads/fuzz/fuzz2.wav

[Rostislav Pehlivanov]

I can't replicate, seems the .wav file gets recognized as an aac file and ffmpeg fails to decode it:

[aac @ 0x3758cc0] Error decoding AAC frame header

The encoder doesn't even seem get a single frame to segfault on.

[Carl Eugen Hoyos]

With 4ffdba24, no encoder options:

(gdb) bt
#0  quantize_and_encode_band_cost_template (ROUNDING=0.405400008, BT_STEREO=0, BT_NOISE=0,
    BT_ESC=1, BT_PAIR=1, BT_UNSIGNED=1, BT_ZERO=0, energy=0x3ac6564, bits=0x3ac6568,
    uplim=inf, lambda=1, cb=11, scale_idx=127, size=8, scaled=<optimized out>, out=0x0,
    in=<optimized out>, pb=0x0, s=0x3a7c280) at libavcodec/aacenc_quantization.h:96
#1  quantize_and_encode_band_cost_ESC (s=0x3a7c280, pb=0x0, in=<optimized out>, quant=0x0,
    scaled=<optimized out>, size=8, scale_idx=127, cb=11, lambda=1, uplim=inf,
    bits=0x3ac6568, energy=0x3ac6564) at libavcodec/aacenc_quantization.h:190
#2  0x0000000000f1f5c4 in quantize_band_cost (rtz=0, energy=0x3ac6564, bits=0x3ac6568,
    lambda=1, cb=11, scale_idx=<optimized out>, size=<optimized out>, scaled=0x3a86090,
    in=0x3c2b290, s=0x3a7c280, uplim=inf) at libavcodec/aacenc_quantization.h:255
#3  quantize_band_cost_cached (energy=<synthetic pointer>, bits=<synthetic pointer>, cb=11,
    scale_idx=<optimized out>, size=<optimized out>, scaled=0x3a86090, in=0x3c2b290,
    g=<optimized out>, w=<optimized out>, s=0x3a7c280, lambda=<optimized out>,
    uplim=<optimized out>, rtz=<optimized out>) at libavcodec/aacenc_quantization_misc.h:40
#4  search_for_quantizers_twoloop (avctx=0x30, s=0x3a7c280, sce=<optimized out>,
    lambda=<optimized out>) at ./libavcodec/aaccoder_twoloop.h:384
#5  0x0000000000dedeb0 in aac_encode_frame (avctx=0x3a72e00, avpkt=0x7fff5653cbb0,
    frame=0x0, got_packet_ptr=0x7fff5653c8ec) at libavcodec/aacenc.c:655
#6  0x0000000000b445bc in avcodec_encode_audio2 (avctx=avctx@entry=0x3a72e00,
    avpkt=avpkt@entry=0x7fff5653cbb0, frame=frame@entry=0x0,
    got_packet_ptr=got_packet_ptr@entry=0x7fff5653c8ec) at libavcodec/utils.c:1750
#7  0x0000000000495099 in flush_encoders () at ffmpeg.c:1741
#8  transcode () at ffmpeg.c:4100
#9  0x00000000004787cb in main (argc=<optimized out>, argv=0x7fff5653cdf8) at ffmpeg.c:4256
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0xf236ee to 0xf2372e:
   0x0000000000f236ee <quantize_and_encode_band_cost_ESC+478>:  clc
   0x0000000000f236ef <quantize_and_encode_band_cost_ESC+479>:  add    %eax,%r8d
   0x0000000000f236f2 <quantize_and_encode_band_cost_ESC+482>:  add    0x95c4(%rdi,%rbp,4),%r8d
   0x0000000000f236fa <quantize_and_encode_band_cost_ESC+490>:  movslq %r8d,%rcx
   0x0000000000f236fd <quantize_and_encode_band_cost_ESC+493>:  add    %r8d,%r8d
   0x0000000000f23700 <quantize_and_encode_band_cost_ESC+496>:  movslq %r8d,%rsi
   0x0000000000f23703 <quantize_and_encode_band_cost_ESC+499>:  add    %rcx,%rdx
   0x0000000000f23706 <quantize_and_encode_band_cost_ESC+502>:  lea    0x0(,%rsi,4),%r10
=> 0x0000000000f2370e <quantize_and_encode_band_cost_ESC+510>:  movzbl (%rdx),%eax
   0x0000000000f23711 <quantize_and_encode_band_cost_ESC+513>:  lea    0x0(%r13,%r10,1),%r12
   0x0000000000f23716 <quantize_and_encode_band_cost_ESC+518>:  movss  (%r12),%xmm2
   0x0000000000f2371c <quantize_and_encode_band_cost_ESC+524>:  ucomiss 0x147db1(%rip),%xmm2        # 0x106b4d4
   0x0000000000f23723 <quantize_and_encode_band_cost_ESC+531>:  jp     0xf2372b <quantize_and_encode_band_cost_ESC+539>
   0x0000000000f23725 <quantize_and_encode_band_cost_ESC+533>:  je     0xf23b00 <quantize_and_encode_band_cost_ESC+1520>
   0x0000000000f2372b <quantize_and_encode_band_cost_ESC+539>:  movaps %xmm12,%xmm7
End of assembler dump.
(gdb) info register
rax            0x80000000       2147483648
rbx            0x0      0
rcx            0xffffffff80000010       -2147483632
rdx            0xffffffff812a1d90       -2127946352
rsi            0x20     32
rdi            0x3a7c280        61325952
rbp            0x0      0x0
rsp            0x7fff5653af10   0x7fff5653af10
r8             0x20     32
r9             0x8      8
r10            0x80     128
r11            0x1      1
r12            0x0      0
r13            0x12a1030        19533872
r14            0x3c2b290        63091344
r15            0x0      0
rip            0xf2370e 0xf2370e <quantize_and_encode_band_cost_ESC+510>
eflags         0x10286  [ PF SF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

Allow me to take the opportunity to express how very rude these kind of reports are.

[Carl Eugen Hoyos]

Still (rarely) reproducible with 7303962

[Kieran Kunhya]

@carl

I was not able to get a backtrace of the segfault (because it is a nondeterministic crash) as I said in the report.

[Rostislav Pehlivanov]

I still can't reproduce, the file doesn't even get recognized as a valid .wav file by any decoder I have and even if it did, after all the changes that have been made since October it's doubtful this bug remains. If someone does have a problem with the git master of the encoder they could reopen this bug.

[Carl Eugen Hoyos]

Reproducible with 2.8 but not 3.0.