Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#2675 closed defect (fixed)

postproc crashes with -fstack-protector-all

Reported by: Carl Eugen Hoyos Owned by: Michael Niedermayer
Priority: normal Component: postproc
Version: git-master Keywords: crash SIGSEGV gcc
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

As reported in http://thread.gmane.org/gmane.comp.video.ffmpeg.user/46215/focus=46220
Only crashes with MMX2 and SSE2, works fine for C-only and MMX

$ ffmpeg -f lavfi -i testsrc -vf pp=dr -f null -
ffmpeg version N-54036-g6c4516d Copyright (c) 2000-2013 the FFmpeg developers
  built on Jun 16 2013 15:48:02 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack --extra-cflags=-fstack-protector-all
  libavutil      52. 35.101 / 52. 35.101
  libavcodec     55. 16.100 / 55. 16.100
  libavformat    55.  8.102 / 55.  8.102
  libavdevice    55.  2.100 / 55.  2.100
  libavfilter     3. 77.101 /  3. 77.101
  libswscale      2.  3.100 /  2.  3.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf55.8.102
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
Segmentation fault
(gdb) r -f lavfi -i testsrc -vf pp=dr -f null -
Starting program: ffmpeg_g -f lavfi -i testsrc -vf pp=dr -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-54036-g6c4516d Copyright (c) 2000-2013 the FFmpeg developers
  built on Jun 16 2013 15:48:02 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack --extra-cflags=-fstack-protector-all
  libavutil      52. 35.101 / 52. 35.101
  libavcodec     55. 16.100 / 55. 16.100
  libavformat    55.  8.102 / 55.  8.102
  libavdevice    55.  2.100 / 55.  2.100
  libavfilter     3. 77.101 /  3. 77.101
  libswscale      2.  3.100 /  2.  3.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
[New Thread 0x7ffff59e7700 (LWP 15836)]
[New Thread 0x7ffff51e6700 (LWP 15837)]
[New Thread 0x7ffff49e5700 (LWP 15838)]
[New Thread 0x7ffff41e4700 (LWP 15839)]
[New Thread 0x7ffff39e3700 (LWP 15840)]
[New Thread 0x7ffff31e2700 (LWP 15841)]
[New Thread 0x7ffff29e1700 (LWP 15842)]
[New Thread 0x7ffff21e0700 (LWP 15843)]
[New Thread 0x7ffff19df700 (LWP 15844)]
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
[New Thread 0x7ffff11de700 (LWP 15845)]
[New Thread 0x7ffff09dd700 (LWP 15846)]
[New Thread 0x7ffff01dc700 (LWP 15847)]
[New Thread 0x7fffef9db700 (LWP 15848)]
[New Thread 0x7fffef1da700 (LWP 15849)]
[New Thread 0x7fffee9d9700 (LWP 15850)]
[New Thread 0x7fffee1d8700 (LWP 15851)]
[New Thread 0x7fffed9d7700 (LWP 15852)]
[New Thread 0x7fffed1d6700 (LWP 15853)]
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf55.8.102
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
0x0000000000bd4cfd in dering_SSE2 (
    src=0x17c8ec0 '\020' <repeats 40 times>, 'Q' <repeats 38 times>"\252, \252", 'j' <repeats 40 times>, ')' <repeats 40 times>"\322, \322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322", <incomplete sequence \322>...,
    src@entry=<error reading variable: Cannot access memory at address 0x10001028>, stride=320,
    stride@entry=<error reading variable: Cannot access memory at address 0x10001028>, c=0x17c9500,
    c@entry=<error reading variable: Cannot access memory at address 0x10001028>) at libpostproc/postprocess_template.c:1094
1094        __asm__ volatile(
(gdb) bt
#0  0x0000000000bd4cfd in dering_SSE2 (
    src=0x17c8ec0 '\020' <repeats 40 times>, 'Q' <repeats 38 times>"\252, \252", 'j' <repeats 40 times>, ')' <repeats 40 times>"\322, \322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322\322", <incomplete sequence \322>...,
    src@entry=<error reading variable: Cannot access memory at address 0x10001028>, stride=320,
    stride@entry=<error reading variable: Cannot access memory at address 0x10001028>, c=0x17c9500,
    c@entry=<error reading variable: Cannot access memory at address 0x10001028>) at libpostproc/postprocess_template.c:1094
Cannot access memory at address 0x10001028
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0xbd4cdd to 0xbd4d1d:
   0x0000000000bd4cdd <dering_SSE2+173>:        pshufw $0xf9,%mm6,%mm4
   0x0000000000bd4ce1 <dering_SSE2+177>:        pmaxub %mm4,%mm6
   0x0000000000bd4ce4 <dering_SSE2+180>:        pshufw $0xfe,%mm6,%mm4
   0x0000000000bd4ce8 <dering_SSE2+184>:        pmaxub %mm4,%mm6
   0x0000000000bd4ceb <dering_SSE2+187>:        movq   %mm6,%mm0
   0x0000000000bd4cee <dering_SSE2+190>:        psubb  %mm7,%mm6
   0x0000000000bd4cf1 <dering_SSE2+193>:        push   %rsp
   0x0000000000bd4cf2 <dering_SSE2+194>:        movd   %mm6,%esp
   0x0000000000bd4cf5 <dering_SSE2+197>:        cmp    0xe3d260,%spl
=> 0x0000000000bd4cfd <dering_SSE2+205>:        pop    %rsp
   0x0000000000bd4cfe <dering_SSE2+206>:        jb     0xbd5291 <dering_SSE2+1633>
   0x0000000000bd4d04 <dering_SSE2+212>:        pavgb  %mm0,%mm7
   0x0000000000bd4d07 <dering_SSE2+215>:        punpcklbw %mm7,%mm7
   0x0000000000bd4d0a <dering_SSE2+218>:        punpcklbw %mm7,%mm7
   0x0000000000bd4d0d <dering_SSE2+221>:        punpcklbw %mm7,%mm7
   0x0000000000bd4d10 <dering_SSE2+224>:        movq   %mm7,(%rsp)
   0x0000000000bd4d14 <dering_SSE2+228>:        movq   (%rdi),%mm0
   0x0000000000bd4d17 <dering_SSE2+231>:        movq   %mm0,%mm1
   0x0000000000bd4d1a <dering_SSE2+234>:        movq   %mm0,%mm2
End of assembler dump.
(gdb) info all-register
rax            0x17c9000        24940544
rbx            0x17c9008        24940552
rcx            0x7fffffffc670   140737488340592
rdx            0x17c9500        24941824
rsi            0x140    320
rdi            0x17c8ec0        24940224
rbp            0x8      0x8
rsp            0x10001000       0x10001000
r8             0x1721c40        24255552
r9             0x0      0
r10            0x140    320
r11            0x0      0
r12            0x178cf88        24694664
r13            0x1721c48        24255560
r14            0x8      8
r15            0x140    320
rip            0xbd4cfd 0xbd4cfd <dering_SSE2+205>
eflags         0x10293  [ CF AF SF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
st0            -nan(0x1010101010101010) (raw 0xffff1010101010101010)
st1            -nan(0x010101010)        (raw 0xffff0000000010101010)
st2            -nan(0x010101010)        (raw 0xffff0000000010101010)
st3            -nan(0x010101010)        (raw 0xffff0000000010101010)
st4            -nan(0x1010101010101010) (raw 0xffff1010101010101010)
st5            0        (raw 0x00000000000000000000)
st6            -nan(0x1000100010001000) (raw 0xffff1000100010001000)
st7            -nan(0x10001000100010)   (raw 0xffff0010001000100010)
fctrl          0x37f    895
fstat          0x0      0
ftag           0xa6aa   42666
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
mxcsr          0x1fa0   [ PE IM DM ZM OM UM PM ]
ymm0           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x10 <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0x1010, 0x1010, 0x1010, 0x1010, 0x1010, 0x1010, 0x1010, 0x1010, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x10101010, 0x10101010, 0x10101010, 0x10101010, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x1010101010101010, 0x1010101010101010, 0x0, 0x0}, v2_int128 = {0x10101010101010101010101010101010, 0x00000000000000000000000000000000}}
ymm1           {v8_float = {0x96970000, 0x96970000, 0x96970000, 0x96970000, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x8000000000000000, 0x8000000000000000, 0x0, 0x0}, v32_int8 = {0xd2 <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0xd2d2d2d2, 0xd2d2d2d2, 0xd2d2d2d2, 0xd2d2d2d2, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0xd2d2d2d2d2d2d2d2, 0xd2d2d2d2d2d2d2d2, 0x0, 0x0}, v2_int128 = {0xd2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2, 0x00000000000000000000000000000000}}
ymm2           {v8_float = {0x96970000, 0x96970000, 0x96970000, 0x96970000, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x8000000000000000, 0x8000000000000000, 0x0, 0x0}, v32_int8 = {0xd2 <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0xd2d2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0xd2d2d2d2, 0xd2d2d2d2, 0xd2d2d2d2, 0xd2d2d2d2, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0xd2d2d2d2d2d2d2d2, 0xd2d2d2d2d2d2d2d2, 0x0, 0x0}, v2_int128 = {0xd2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2, 0x00000000000000000000000000000000}}
ymm3           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x29 <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x29292929, 0x29292929, 0x29292929, 0x29292929, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x2929292929292929, 0x2929292929292929, 0x0, 0x0}, v2_int128 = {0x29292929292929292929292929292929, 0x00000000000000000000000000000000}}
ymm4           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x29 <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x2929, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x29292929, 0x29292929, 0x29292929, 0x29292929, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x2929292929292929, 0x2929292929292929, 0x0, 0x0}, v2_int128 = {0x29292929292929292929292929292929, 0x00000000000000000000000000000000}}
ymm5           {v8_float = {0x15151000, 0x15151000, 0x15151000, 0x15151000, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x8000000000000000, 0x8000000000000000, 0x0, 0x0}, v32_int8 = {0x51 <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0x5151, 0x5151, 0x5151, 0x5151, 0x5151, 0x5151, 0x5151, 0x5151, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x51515151, 0x51515151, 0x51515151, 0x51515151, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x5151515151515151, 0x5151515151515151, 0x0, 0x0}, v2_int128 = {0x51515151515151515151515151515151, 0x00000000000000000000000000000000}}
ymm6           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0x1, 0x40, 0x0, 0x0, 0x1, 0x40, 0x0, 0x0, 0x1, 0x40, 0x0, 0x0, 0x1, 0x40, 0x0 <repeats 17 times>}, v16_int16 = {0x100, 0x40, 0x100, 0x40, 0x100, 0x40, 0x100, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x400100, 0x400100, 0x400100, 0x400100, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x40010000400100, 0x40010000400100, 0x0, 0x0}, v2_int128 = {0x00400100004001000040010000400100, 0x00000000000000000000000000000000}}
ymm7           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0x80, 0x1, 0x80, 0x2, 0x80, 0x3, 0x80, 0x6, 0x80, 0x7, 0x80, 0x8, 0x80, 0x9, 0x80, 0x0 <repeats 16 times>}, v16_int16 = {0x8000, 0x8001, 0x8002, 0x8003, 0x8006, 0x8007, 0x8008, 0x8009, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x80018000, 0x80038002, 0x80078006, 0x80098008, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x8003800280018000, 0x8009800880078006, 0x0, 0x0}, v2_int128 = {0x80098008800780068003800280018000, 0x00000000000000000000000000000000}}
ymm8           {v8_float = {0x80000000, 0x0, 0x80000000, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x8000000000000000, 0x8000000000000000, 0x0, 0x0}, v32_int8 = {0xff, 0xec, 0xc8, 0xda, 0x0, 0x0, 0xff, 0xec, 0xff, 0xec, 0xc8, 0xda, 0x0, 0x0, 0xff, 0xec, 0x0 <repeats 16 times>}, v16_int16 = {0xecff, 0xdac8, 0x0, 0xecff, 0xecff, 0xdac8, 0x0, 0xecff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0xdac8ecff, 0xecff0000, 0xdac8ecff, 0xecff0000, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0xecff0000dac8ecff, 0xecff0000dac8ecff, 0x0, 0x0}, v2_int128 = {0xecff0000dac8ecffecff0000dac8ecff, 0x00000000000000000000000000000000}}
ymm9           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x38, 0x38, 0x0, 0x0, 0xc8, 0xda, 0x38, 0x38, 0x38, 0x38, 0x0, 0x0, 0xc8, 0xda, 0x38, 0x38, 0x0 <repeats 16 times>}, v16_int16 = {0x3838, 0x0, 0xdac8, 0x3838, 0x3838, 0x0, 0xdac8, 0x3838, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x3838, 0x3838dac8, 0x3838, 0x3838dac8, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x3838dac800003838, 0x3838dac800003838, 0x0, 0x0}, v2_int128 = {0x3838dac8000038383838dac800003838, 0x00000000000000000000000000000000}}
ymm10          {v8_float = {0xe63e4000, 0x0, 0xe63e4000, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x38, 0x38, 0xe3, 0xd0, 0x0, 0x0, 0x38, 0x38, 0x38, 0x38, 0xe3, 0xd0, 0x0, 0x0, 0x38, 0x38, 0x0 <repeats 16 times>}, v16_int16 = {0x3838, 0xd0e3, 0x0, 0x3838, 0x3838, 0xd0e3, 0x0, 0x3838, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0xd0e33838, 0x38380000, 0xd0e33838, 0x38380000, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x38380000d0e33838, 0x38380000d0e33838, 0x0, 0x0}, v2_int128 = {0x38380000d0e3383838380000d0e33838, 0x00000000000000000000000000000000}}
ymm11          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x8000000000000000, 0x8000000000000000, 0x0, 0x0}, v32_int8 = {0xe4, 0xf6, 0x0, 0x0, 0xe3, 0xd0, 0xe4, 0xf6, 0xe4, 0xf6, 0x0, 0x0, 0xe3, 0xd0, 0xe4, 0xf6, 0x0 <repeats 16 times>}, v16_int16 = {0xf6e4, 0x0, 0xd0e3, 0xf6e4, 0xf6e4, 0x0, 0xd0e3, 0xf6e4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0xf6e4, 0xf6e4d0e3, 0xf6e4, 0xf6e4d0e3, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0xf6e4d0e30000f6e4, 0xf6e4d0e30000f6e4, 0x0, 0x0}, v2_int128 = {0xf6e4d0e30000f6e4f6e4d0e30000f6e4, 0x00000000000000000000000000000000}}
ymm12          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x2, 0x80, 0x3, 0x80, 0x4, 0x80, 0x5, 0x80, 0x8, 0x80, 0x9, 0x80, 0xa, 0x80, 0xb, 0x80, 0x0 <repeats 16 times>}, v16_int16 = {0x8002, 0x8003, 0x8004, 0x8005, 0x8008, 0x8009, 0x800a, 0x800b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x80038002, 0x80058004, 0x80098008, 0x800b800a, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x8005800480038002, 0x800b800a80098008, 0x0, 0x0}, v2_int128 = {0x800b800a800980088005800480038002, 0x00000000000000000000000000000000}}
ymm13          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x42, 0xbc, 0x0 <repeats 24 times>}, v16_int16 = {0x0, 0x0, 0x0, 0xbc42, 0x0 <repeats 12 times>}, v8_int32 = {0x0, 0xbc420000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0xbc42000000000000, 0x0, 0x0, 0x0}, v2_int128 = {0x0000000000000000bc42000000000000, 0x00000000000000000000000000000000}}
ymm14          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0xcb, 0x1a, 0xf2, 0x64, 0xae, 0xaa, 0x6c, 0xbc, 0x0 <repeats 24 times>}, v16_int16 = {0x1acb, 0x64f2, 0xaaae, 0xbc6c, 0x0 <repeats 12 times>}, v8_int32 = {0x64f21acb, 0xbc6caaae, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0xbc6caaae64f21acb, 0x0, 0x0, 0x0}, v2_int128 = {0x0000000000000000bc6caaae64f21acb, 0x00000000000000000000000000000000}}
ymm15          {v8_float = {0x0, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x3, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0xc6, 0xb8, 0xbe, 0xd3, 0xb9, 0xb, 0x40, 0x0 <repeats 24 times>}, v16_int16 = {0xc600, 0xbeb8, 0xb9d3, 0x400b, 0x0 <repeats 12 times>}, v8_int32 = {0xbeb8c600, 0x400bb9d3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x400bb9d3beb8c600, 0x0, 0x0, 0x0}, v2_int128 = {0x0000000000000000400bb9d3beb8c600, 0x00000000000000000000000000000000}}
$ valgrind ./ffmpeg_g -f lavfi -i testsrc -vf pp=dr -f null -
==15918== Memcheck, a memory error detector
==15918== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==15918== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==15918== Command: ./ffmpeg_g -f lavfi -i testsrc -vf pp=dr -f null -
==15918==
ffmpeg version N-54036-g6c4516d Copyright (c) 2000-2013 the FFmpeg developers
  built on Jun 16 2013 15:48:02 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack --extra-cflags=-fstack-protector-all
  libavutil      52. 35.101 / 52. 35.101
  libavcodec     55. 16.100 / 55. 16.100
  libavformat    55.  8.102 / 55.  8.102
  libavdevice    55.  2.100 / 55.  2.100
  libavfilter     3. 77.101 /  3. 77.101
  libswscale      2.  3.100 /  2.  3.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
==15918== Invalid read of size 8
==15918==    at 0x68DB59F: __GI___strncasecmp_l (in /lib64/libc-2.15.so)
==15918==    by 0x688F8A5: ____strtod_l_internal (in /lib64/libc-2.15.so)
==15918==    by 0xC5A83E: av_strtod (eval.c:96)
==15918==    by 0xC5B0E4: parse_primary (eval.c:322)
==15918==    by 0xC5BBA0: parse_factor (eval.c:481)
==15918==    by 0xC5BDAB: parse_term (eval.c:530)
==15918==    by 0xC5AE0E: parse_expr (eval.c:554)
==15918==    by 0xC5BFED: av_expr_parse (eval.c:671)
==15918==    by 0xC5C180: av_expr_parse_and_eval (eval.c:708)
==15918==    by 0x4C3AAF: config_props (vf_scale.c:209)
==15918==    by 0x48CC3E: avfilter_config_links (avfilter.c:242)
==15918==    by 0x48CC21: avfilter_config_links (avfilter.c:231)
==15918==  Address 0x746a780 is 0 bytes inside a block of size 3 alloc'd
==15918==    at 0x4C290FE: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==15918==    by 0x4C291A7: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==15918==    by 0xC6443F: av_malloc (mem.c:93)
==15918==    by 0xC5BF3C: av_expr_parse (eval.c:648)
==15918==    by 0xC5C180: av_expr_parse_and_eval (eval.c:708)
==15918==    by 0x4C3AAF: config_props (vf_scale.c:209)
==15918==    by 0x48CC3E: avfilter_config_links (avfilter.c:242)
==15918==    by 0x48CC21: avfilter_config_links (avfilter.c:231)
==15918==    by 0x490AEE: avfilter_graph_config (avfiltergraph.c:267)
==15918==    by 0x47211E: configure_filtergraph (ffmpeg_filter.c:862)
==15918==    by 0x47BA00: transcode_init (ffmpeg.c:2283)
==15918==    by 0x466D3A: main (ffmpeg.c:3176)
==15918==
==15918== Invalid read of size 8
==15918==    at 0x68DB5A7: __GI___strncasecmp_l (in /lib64/libc-2.15.so)
==15918==    by 0x688F8A5: ____strtod_l_internal (in /lib64/libc-2.15.so)
==15918==    by 0xC5A83E: av_strtod (eval.c:96)
==15918==    by 0xC5B0E4: parse_primary (eval.c:322)
==15918==    by 0xC5BBA0: parse_factor (eval.c:481)
==15918==    by 0xC5BDAB: parse_term (eval.c:530)
==15918==    by 0xC5AE0E: parse_expr (eval.c:554)
==15918==    by 0xC5BFED: av_expr_parse (eval.c:671)
==15918==    by 0xC5C180: av_expr_parse_and_eval (eval.c:708)
==15918==    by 0x4C3AAF: config_props (vf_scale.c:209)
==15918==    by 0x48CC3E: avfilter_config_links (avfilter.c:242)
==15918==    by 0x48CC21: avfilter_config_links (avfilter.c:231)
==15918==  Address 0x746a788 is 5 bytes after a block of size 3 alloc'd
==15918==    at 0x4C290FE: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==15918==    by 0x4C291A7: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==15918==    by 0xC6443F: av_malloc (mem.c:93)
==15918==    by 0xC5BF3C: av_expr_parse (eval.c:648)
==15918==    by 0xC5C180: av_expr_parse_and_eval (eval.c:708)
==15918==    by 0x4C3AAF: config_props (vf_scale.c:209)
==15918==    by 0x48CC3E: avfilter_config_links (avfilter.c:242)
==15918==    by 0x48CC21: avfilter_config_links (avfilter.c:231)
==15918==    by 0x490AEE: avfilter_graph_config (avfiltergraph.c:267)
==15918==    by 0x47211E: configure_filtergraph (ffmpeg_filter.c:862)
==15918==    by 0x47BA00: transcode_init (ffmpeg.c:2283)
==15918==    by 0x466D3A: main (ffmpeg.c:3176)
==15918==
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf55.8.102
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
==15918== Warning: client switching stacks?  SP change: 0x10001000 --> 0x0
==15918==          to suppress, use: --max-stackframe=268439552 or greater
==15918==
==15918== Process terminating with default action of signal 11 (SIGSEGV)
==15918==  Access not within mapped region at address 0x29
==15918==    at 0xBD5291: dering_SSE2 (postprocess_template.c:1437)
==15918==  If you believe this happened as a result of a stack
==15918==  overflow in your program's main thread (unlikely but
==15918==  possible), you can try to increase the size of the
==15918==  main thread stack using the --main-stacksize= flag.
==15918==  The main thread stack size used in this run was 8388608.
==15918==
==15918== HEAP SUMMARY:
==15918==     in use at exit: 1,340,134 bytes in 237 blocks
==15918==   total heap usage: 3,219 allocs, 2,982 frees, 3,375,434 bytes allocated
==15918==
==15918== LEAK SUMMARY:
==15918==    definitely lost: 0 bytes in 0 blocks
==15918==    indirectly lost: 0 bytes in 0 blocks
==15918==      possibly lost: 4,896 bytes in 18 blocks
==15918==    still reachable: 1,335,238 bytes in 219 blocks
==15918==         suppressed: 0 bytes in 0 blocks
==15918== Rerun with --leak-check=full to see details of leaked memory
==15918==
==15918== For counts of detected and suppressed errors, rerun with: -v
==15918== ERROR SUMMARY: 6 errors from 2 contexts (suppressed: 2 from 2)
Killed

Change History (3)

comment:1 by Carl Eugen Hoyos, 11 years ago

The reason is apparently that gcc miscompiles dering_SSE2() and dering_MMX2() in libpostproc/postproc_template.c:

1182        "movq %%mm6, %%mm0                      \n\t" // max
1183        "psubb %%mm7, %%mm6                     \n\t" // max - min
1184        "push %4                              \n\t"
1184        "movd %%mm6, %k4                        \n\t"
1185        "cmpb "MANGLE(deringThreshold)", %b4    \n\t"
1186        "pop %4                               \n\t"

...

        : : "r" (src), "r" ((x86_reg)stride), "m" (c->pQPb), "m"(c->pQPb2), "q"(tmp)
        : "%"REG_a, "%"REG_d
   0x0000000000bd40eb <dering_SSE2+187>:        movq   %mm6,%mm0
   0x0000000000bd40ee <dering_SSE2+190>:        psubb  %mm7,%mm6
   0x0000000000bd40f1 <dering_SSE2+193>:        push   %rsp
   0x0000000000bd40f2 <dering_SSE2+194>:        movd   %mm6,%esp
   0x0000000000bd40f5 <dering_SSE2+197>:        cmp    0xe3c6c0,%spl
=> 0x0000000000bd40fd <dering_SSE2+205>:        pop    %rsp

comment:2 by Carl Eugen Hoyos, 11 years ago

Resolution: fixed
Status: newclosed

Fixed / worked-around by Michael.

comment:3 by Carl Eugen Hoyos, 11 years ago

Keywords: SIGSEGV gcc added
Note: See TracTickets for help on using tickets.