x264: x264_cqm_init access beyond array causes undefined behavior in gcc48

[Safari]

Summary of the bug:
can't x264 bugs be submitted with trac..? anyways...

i can get to 7, whereas quant4_mf quant4_bias0 quant4_mf dequant4_mf and unquant4_mf are defined as array [4], therefore undefined behavior happens with gcc-4.8-0.14.fc19 (infinite loop).

How to reproduce:

run x264 and it segfaults in x264_cqm_init.

small test case which causes infinite loop

#include <stdio.h>
#include <string.h>
#include <inttypes.h>
#include <stdlib.h>

uint32_t (*quant4_mf[4])[16];

int main(int argc, char *argv[])
{

int i;

for(i = 0; i < 8; i++ )
{

fprintf(stderr, "i=%d\n", i);
quant4_mf[i] = (void*)0;

}
return 0;

}

% ffmpeg -i input ... output
ffmpeg version
built on ...

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Carl Eugen Hoyos]

Please consider posting on ​http://mailman.videolan.org/listinfo/x264-devel

[Roger Pack]

I think I may have just run into this on windows (cross compiling) with 4.8 (it crashes right after the line [libx264 @ 037bd3a0] using SAR=32/27).
Was this ever reported to x264-devel does anybody know?

Program received signal SIGSEGV, Segmentation fault.
0x759c79c8 in msvcrt!memcmp () from C:\Windows\syswow64\msvcrt.dll
(gdb) bt
#0 0x759c79c8 in msvcrt!memcmp () from C:\Windows\syswow64\msvcrt.dll
#1 0x004a303c in x264_cqm_init ()

Also related: ​http://forum.doom9.org/showthread.php?p=1623567 which seems to have some fix.