Undefined behaviors in ffmpeg

[Dingbao Xie]

Summary of the bug:
found some undefined behaviors in ffmpeg
How to reproduce:

% ffmpeg -i input -y test.avi[]
ffmpeg version git master
built on clang-3.4 with flag -fsanitize=undefined

You can download the malformed inputs from ​https://www.dropbox.com/s/yfflpbb07jz3eh0/inputs.tar.gz?dl=0

libavcodec/mpegvideo_enc.c:142:21: runtime error: left shift of negative value -64
libavcodec/h264.c:259:17: runtime error: load of misaligned address 0x000008ec0929 for type 'const uint64_t' (aka 'const unsigned long'), which requires 8 byte alignment
libavcodec/h264.c:258:17: runtime error: member access within misaligned address 0x000008ec0929 for type 'const av_alias64', which requires 8 byte alignment
libavcodec/h264_slice.c:533:54: runtime error: index -2011593 out of bounds for type 'uint32_t [88][16]'
libavcodec/h264_slice.c:537:54: runtime error: index -503031 out of bounds for type 'uint32_t [88][64]'
libavcodec/golomb.h:75:13: runtime error: shift exponent -1 is negative
libavcodec/h264_slice.c:2017:61: runtime error: index -1 out of bounds for type 'int [64]'
libavcodec/h264_slice.c:2018:38: runtime error: index -1 out of bounds for type 'int [64]'
libavcodec/h264_slice.c:1989:38: runtime error: index -1 out of bounds for type 'int [64]'
libavcodec/cabac_functions.h:70:13: runtime error: left shift of negative value -921043036
/slibavformat/dump.c:423:37: runtime error: signed integer overflow: 48 * 2139225729 cannot be represented in type 'int'
libavformat/dump.c:424:37: runtime error: signed integer overflow: 144 * 16712191 cannot be represented in type 'int'
          ^ 
libavcodec/h264_cavlc.c:586:54: runtime error: index -1 out of bounds for type 'VLC [15]'
libavcodec/h264_cavlc.c:627:9: runtime error: index -1 out of bounds for type 'VLC [6]'
libavcodec/h264_cavlc.c:580:69: runtime error: index -1 out of bounds for type 'VLC [3]'
libavcodec/cabac_functions.h:70:13: runtime error: left shift of negative value -1842012160
./libavutil/rational.h:81:18: runtime error: division by zero
./libavutil/rational.h:81:18: runtime error: division by zero
libavformat/mov.c:2442:25: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
libavformat/mov.c:2482:13: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
libavcodec/ac3enc.c:1186:14: runtime error: left shift of negative value -9
libavcodec/h264_slice.c:537:54: runtime error: index -489998 out of bounds for type 'uint32_t [88][64]'
libavcodec/h264_cavlc.c:627:9: runtime error: index -1 out of bounds for type 'VLC [6]'

[Dingbao Xie]

malformed input to reproduce undefined behavior

[Ganesh Ajjanagadde]

Could you please respin on the latest git master and post an updated log- many of the line numbers have changed? On that note, please give a more verbose log so that the context is retained for future reference even after the code moves around.

Also, your command line invocation does not work - I get after extracting (e.g)
ffmpeg -i id:000000,src:000000,op:flip1,pos:4 -y test.avi
...
id:000000,src:000000,op:flip1,pos:4: Protocol not found

[Ganesh Ajjanagadde]

Issues with libavformat/dump should be fixed in: d1bdaf3fb2c45020f72a378bb64eab1bf136581c.

[Ronald S. Bultje]

Ganesh, to workaround protocol parsing, use -i file:$filename, e.g. -i file:id:000000,src:000000,op:flip1,pos:4, I think that should fix it.

[Ganesh Ajjanagadde]

@Ronald: Thanks for the tip, can now reproduce.

mpegvideoenc issues fixed in:
3be27e07d3c5239f6d53b86aebcd201f722df4d0

[Ganesh Ajjanagadde]

file id 14 still triggers on gcc-usan. This is likely a bug in GCC.
After all, the commit 053e80f6eaf8d87521fe58ea96886b6ee0bbe59d addresses this.
Basically, gcc-usan can't use the FFNABS solution:

#include <limits.h>
#define FFNABS(a) ((a) <= 0 ? (a) : (-(a)))
int main(void) {

int d = INT_MIN;
return (FFNABS(d) < -(1<<28));

}

fails on gcc-usan while clang-usan is fine with it.

Same goes for file id 15.

[Ganesh Ajjanagadde]

Many files seem to be missing:
id 8, 9, 10, 17, 20, 21, 23 (assuming 25 is the last one).

@xiedingbao:
Can you confirm this and reupload the missing ones? Thanks.

[Ganesh Ajjanagadde]

ac3enc issue fixed in:
979572365f2133f969f3f49ec6a99cc8739d2eba.

[Ganesh Ajjanagadde]

One issue with libavformat/mov.c should have been fixed in:
053e80f6eaf8d87521fe58ea96886b6ee0bbe59d via the FFNABS.

The second issue is a genuine one that still remains with master; a patch has been submitted just now:
https://ffmpeg.org/pipermail/ffmpeg-devel/2015-October/180727.html.

I will update with the commit ID when it gets pushed.

@Michael: could you please have a look at the h264 ones? I am not at all familiar with it.

[Ganesh Ajjanagadde]

Other issue with libavformat/mov.c should be fixed in: d59bfcd11229300182c672ca734568919a85f773.

All remaining issues are with h264.

[Carl Eugen Hoyos]

I don't think the h264 issues were fixed.