segmentation fault while using vf_deshake

[taeuber]

Segmentaiton fault while using deshaker video filter

[taeuber]

log file of gdb output

[Carl Eugen Hoyos]

For future reports (it made absolutely no difference for this one):
Please consider recompiling FFmpeg without --enable-shared to get a useful backtrace.
Please do not use external libraries (like libx264) in your command line if they are not necessary to trigger the problem.

Afaiu, the crash happens because the variable pos reaches with attached sample a value of 1348, overwriting the stack in "angles[pos++] = ..." (making the backtrace useless).

To reproduce the crash, vf_deshake.o has to be compiled with at least -O2.

ffmpeg -i out.avi -vf deshake -f null -
ffmpeg version N-33271-g171a5b5, Copyright (c) 2000-2011 the FFmpeg developers
  built on Oct  4 2011 15:28:26 with gcc 4.5.3
  configuration: --cc=/usr/local/gcc-4.5.3/bin/gcc --disable-optimizations
  libavutil    51. 19. 0 / 51. 19. 0
  libavcodec   53. 19. 0 / 53. 19. 0
  libavformat  53. 13. 0 / 53. 13. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 43. 6 /  2. 43. 6
  libswscale    2.  1. 0 /  2.  1. 0

Seems stream 0 codec frame rate differs from container frame rate: 5994.00 (5994/1) -> 0.08 (1/12)
Input #0, avi, from 'out.avi':
  Metadata:
    encoder         : Lavf53.13.0
  Duration: 00:00:00.47, start: 0.000000, bitrate: 36494 kb/s
    Stream #0:0: Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x800, 0.08 tbr, 2997 tbn, 5994 tbc
[buffer @ 0x145e200] w:1920 h:800 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[deshake @ 0x1452000] rx: 16, ry: 16, edge: 3 blocksize: 16 contrast: 125 search: 0
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf53.13.0
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x800, q=2-31, 200 kb/s, 90k tbn, 0.08 tbc
Stream mapping:
  Stream #0.0 -> #0.0 (h264 -> rawvideo)
Press [q] to stop, [?] for help
Segmentation fault1 q=0.0 size=      -0kB time=00:00:24.00 bitrate=  -0.0kbits/s dup=0 drop=10

[Michael Niedermayer]

Locally fixed, will be pushed in a moment