crashing libavcodec when encoding to mp3 on Mac

[dfuhrmann]

VLC recently updated to lame 3.100 in its contrib system, and it uses lame over libavcodec to encode mp3 audio for its chromecast streaming. We get several crashes in lame, which are only visible with our macOS port, not on windows.
This can be seen with the latest stable version VLC 3.0.8.

Here is the relevant stack trace we have, the full trace is attached as a file:

Crash reason: EXC_SOFTWARE / SIGABRT
Crash address: 0x7fff7b50c23e
Process uptime: 78 seconds
Thread 14 (crashed)
0 libsystem_kernel.dylib!pthread_kill + 0xa
rax = 0x0000000000000000 rdx = 0x0000000000000000
rcx = 0x000070000cdd62d8 rbx = 0x000070000cde0000
rsi = 0x0000000000000006 rdi = 0x000000000000d90b
rbp = 0x000070000cdd6310 rsp = 0x000070000cdd62d8
r8 = 0x0000000000000240 r9 = 0xcccccccccccccccd
r10 = 0x0000000000000000 r11 = 0x0000000000000206
r12 = 0x000000000000d90b r13 = 0x000000010c525000
r14 = 0x0000000000000006 r15 = 0x000000000000002d
rip = 0x00007fff7b50c23e
Found by: given as instruction pointer in context
1 libsystem_c.dylib!abort + 0x7f
rbp = 0x000070000cdd6340 rsp = 0x000070000cdd6320
rip = 0x00007fff7b4751c9
Found by: previous frame's frame pointer
2 libsystem_c.dylib!assert_rtn + 0x140
rbp = 0x000070000cdd6380 rsp = 0x000070000cdd6350
rip = 0x00007fff7b43d868
Found by: previous frame's frame pointer
3 libavcodec_plugin.dylib!L3psycho_anal_vbr [psymodel.c : 576 + 0x1f]
rbp = 0x000070000cddb9f0 rsp = 0x000070000cdd6390
rip = 0x000000010f007617
Found by: previous frame's frame pointer
4 libavcodec_plugin.dylib!lame_encode_mp3_frame [encoder.c : 374 + 0x30]
rbx = 0x0000000000000010 rbp = 0x000070000cddfa20
rsp = 0x000070000cddba00 r12 = 0x00007fc5f5a73518
r13 = 0x00007fc5f5a6f800 r14 = 0x0000000000000001
r15 = 0x00007fc5f5a772c8 rip = 0x000000010eff6abf
Found by: call frame info
5 libavcodec_plugin.dylib!lame_encode_buffer_template [lame.c : 1793 + 0xb]
rbx = 0x00007fc5f5a41c00 rbp = 0x000070000cddfad0
rsp = 0x000070000cddfa30 r12 = 0x00007fc5f5a6f800
r13 = 0x0000000000000000 r14 = 0x00007fc5f5a13600
r15 = 0x0000000000001200 rip = 0x000000010f000038
Found by: call frame info
6 libavcodec_plugin.dylib!lame_encode_buffer_float [lame.c : 1925 + 0x11]
rbx = 0x00007fc5f4ca98c0 rbp = 0x000070000cddfaf0
rsp = 0x000070000cddfae0 r12 = 0x00007fc5f5bcda00
r13 = 0x000070000cddfc7c r14 = 0x00007fc5f4ca9e20
r15 = 0x00007fc5f4ca8f80 rip = 0x000000010f000255
Found by: call frame info
7 libavcodec_plugin.dylib!mp3lame_encode_frame [libmp3lame.c : 216 + 0x22]
rbx = 0x00007fc5f4ca98c0 rbp = 0x0000000000000002
rsp = 0x000070000cddfb00 r12 = 0x00007fc5f5bcda00
r13 = 0x000070000cddfc7c r14 = 0x00007fc5f4ca9e20
r15 = 0x00007fc5f4ca8f80 rip = 0x000000010e99a525
Found by: call frame info

This is in the following code part:

FLOAT const el = fftenergy[j];
assert(el >= 0);

Unfortunately, so far this cannot be reproduced locally. Any hint how to further debug this would be very helpful.

[dfuhrmann]

Attached is one example stack trace, more traces are available in Videolans crash system.