FFmpeg crashes on decoding H.264 MP4 file

[mbradshaw]

Summary of the bug:
Simply running: ffmpeg -i hunger.mp4 -vn out.wav
results in: Segmentation fault: 11

Sample source file (33MB):
​https://googledrive.com/host/0BxWx_dIBnyRoN2cxT1ZOaEhOUnc/hunger.mp4

Operating System:
OS X 10.8.5

How to reproduce (here's the result of running it in gdb):

$ gdb ./ffmpeg
GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin12.4.0".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /Users/mjbshaw/Projects/ffmpeg/ffmpeg...done.
(gdb) run -i hunger.mp4 -vn out.wav
Starting program: /Users/mjbshaw/Projects/ffmpeg/./ffmpeg -i hunger.mp4 -vn out.wav
BFD: /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork(i386:x86-64): unknown load command 0x20
ffmpeg version N-56663-g851a6e2 Copyright (c) 2000-2013 the FFmpeg developers
  built on Sep 25 2013 18:57:30 with llvm-gcc 4.2.1 (LLVM build 2336.11.00)
  configuration: --enable-debug=gdb --disable-optimizations --disable-stripping --enable-libopenjpeg --disable-decoder=jpeg2000
  libavutil      52. 46.100 / 52. 46.100
  libavcodec     55. 33.100 / 55. 33.100
  libavformat    55. 18.102 / 55. 18.102
  libavdevice    55.  3.100 / 55.  3.100
  libavfilter     3. 87.100 /  3. 87.100
  libswscale      2.  5.100 /  2.  5.100
  libswresample   0. 17.103 /  0. 17.103

Program received signal SIGSEGV, Segmentation fault.
0x00000001004d8b3b in refill2 (c=0x101cf5ed0) at /Users/mjbshaw/Projects/ffmpeg/libavcodec/cabac_functions.h:73
73	        x+= (c->bytestream[0]<<9) + (c->bytestream[1]<<1);
(gdb) bt
#0  0x00000001004d8b3b in refill2 (c=0x101cf5ed0) at /Users/mjbshaw/Projects/ffmpeg/libavcodec/cabac_functions.h:73
#1  0x00000001004d8ce7 in get_cabac_inline (c=0x101cf5ed0, state=0x101cf5f6a "\027%\004\v#\025-\001#\035\031==?A*\001\n%\001\037\023\031\021-\037;\034\023\001/#\t\017\002\016\004")
    at /Users/mjbshaw/Projects/ffmpeg/libavcodec/cabac_functions.h:101
#2  get_cabac_noinline (c=0x101cf5ed0, state=0x101cf5f6a "\027%\004\v#\025-\001#\035\031==?A*\001\n%\001\037\023\031\021-\037;\034\023\001/#\t\017\002\016\004") at h264_cabac.c:107
#3  0x00000001004e0a22 in decode_cabac_mb_cbp_luma (h=0x101ca0000) at h264_cabac.c:1403
#4  0x00000001004f28e4 in ff_h264_decode_mb_cabac (h=0x101ca0000) at h264_cabac.c:2286
#5  0x00000001004d5e1a in decode_slice (avctx=0x102013800, arg=0x7fff5fbfe7c8) at h264.c:4434
#6  0x00000001004d691d in execute_decode_slices (h=0x101ca0000, context_count=1) at h264.c:4590
#7  0x00000001004d7cd5 in decode_nal_units (h=0x101ca0000, buf=0x1019138b0 "", buf_size=214, parse_extradata=0) at h264.c:4942
#8  0x00000001004d84c0 in decode_frame (avctx=0x102013800, data=0x101913a40, got_frame=0x7fff5fbfec44, avpkt=0x7fff5fbfeac0) at h264.c:5079
#9  0x00000001008f4d43 in avcodec_decode_video2 (avctx=0x102013800, picture=0x101913a40, got_picture_ptr=0x7fff5fbfec44, avpkt=0x7fff5fbfebb8) at utils.c:1994
#10 0x0000000100238ff9 in try_decode_frame (st=0x1019115c0, avpkt=0x1019139a0, options=0x101911be0) at utils.c:2484
#11 0x000000010023b623 in avformat_find_stream_info (ic=0x10200f200, options=0x101911be0) at utils.c:2930
#12 0x0000000100004ad2 in open_input_file (o=0x7fff5fbff558, filename=0x7fff5fbffbf3 "hunger.mp4") at ffmpeg_opt.c:809
#13 0x000000010000e3dd in open_files (l=0x101910958, inout=0x100d9935f "input", open_file=0x100004160 <open_input_file>) at ffmpeg_opt.c:2494
#14 0x000000010000e619 in ffmpeg_parse_options (argc=5, argv=0x7fff5fbffa78) at ffmpeg_opt.c:2531
#15 0x00000001000235cd in main (argc=5, argv=0x7fff5fbffa78) at ffmpeg.c:3393
(gdb) p c
$1 = (CABACContext *) 0x101cf5ed0
(gdb) p c->bytestream
$2 = (uint8_t *) 0x1ff0a <Address 0x1ff0a out of bounds>
(gdb) p *c
$3 = {low = -1643708416, range = 292, outstanding_count = 0, bytestream_start = 0x101d0f003 "\343\205\035Y", bytestream = 0x1ff0a <Address 0x1ff0a out of bounds>, 
  bytestream_end = 0x101d0f0a4 "", pb = {bit_buf = 0, bit_left = 0, buf = 0x0, buf_ptr = 0x0, buf_end = 0x0, size_in_bits = 0}}

[Carl Eugen Hoyos]

(I cannot reproduce)

Is this also reproducible with older versions of FFmpeg? If not, and if you use such an older version to extract the h264 core, is that enough to crash current FFmpeg?

[mbradshaw]

I have to configure with --enable-debug=gdb --disable-optimizations for it to crash. Enabling optimizations results in a successful run.

git bisect shows the first bad commit is b3ab2810277decc2c0bfbaa08414a432e4774f34

I think this is a duplicate of #2156. Building with clang and gcc-4.8 works fine. Feel free to reopen if you think it's not a duplicate.

[Carl Eugen Hoyos]

Sounds important.

[Michael Niedermayer]

Is this also reproducable with --disable-asm ?
also is this specific to this file ? or does the binary crash with any cabac h264 file ?

[mbradshaw]

Replying to michael:

Is this also reproducable with --disable-asm ?

No, disabling asm does not allow me to reproduce, and seems to fix the crash.

also is this specific to this file ? or does the binary crash with any cabac h264 file ?

It is not just this file. It crashes with this file: ​https://googledrive.com/host/0BxWx_dIBnyRoN2cxT1ZOaEhOUnc/starcrafts.mp4
but not with this file (though there are some errors logged, but it decodes fine as far as I can tell): ​https://googledrive.com/host/0BxWx_dIBnyRoN2cxT1ZOaEhOUnc/photonn.mp4

So far all of these samples I've obtained from YouTube via youtube-dl

[Michael Niedermayer]

Replying to mbradshaw:

Replying to michael:

Is this also reproducable with --disable-asm ?

No, disabling asm does not allow me to reproduce, and seems to fix the crash.

then the next question is which asm function needs to be disabled to fix this ?
once its found either just put it under #if !BROKEN_COMPILER or if you belive its not a compiler bug then the functions disassmbly would need to be compared against the source

[Michael Niedermayer]

can you try to disable get_cabac_bypass_sign_x86() that is comment "#define get_cabac_bypass_sign get_cabac_bypass_sign_x86" out ?

[mbradshaw]

Replying to michael:

can you try to disable get_cabac_bypass_sign_x86() that is comment "#define get_cabac_bypass_sign get_cabac_bypass_sign_x86" out ?

I will when I get the chance. I don't have access to an OS X 10.8 machine with Xcode 4 at the moment. I've since upgraded my machine and Apple completely removed GCC. I'll try to find a machine...

[Michael Niedermayer]

change made, lets see if it fixes some of the fate h264 failures there are on mac/darwin

[Michael Niedermayer]

noone in 6 weeks could reproduce the ticket thus assume its fixed

[Carl Eugen Hoyos]

I tested with llvm-gcc 4.2, the last non-clang compiler published by Apple (XCode 4.6.3).
The crash was fixed in 41efb8d9
fate passes with default compilation, h264 (and snow) decoding is broken with --disable-optimizations, works fine with --disable-optimizations --disable-asm