crash in ff_rgb24ToY_ssse3()

[ami_stuff]

​https://files.fm/u/xq5gewcc

(gdb) r -i utv.avi out.avi
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: D:\MinGW\msys\1.0\ffmpeg\ffmpeg_g.exe -i utv.avi out.avi
[New Thread 5956.0x138c]
ffmpeg version 3.3.git Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 4.6.2 (GCC)
  configuration: --disable-pthreads --disable-ffprobe --enable-gpl
  libavutil      55. 61.100 / 55. 61.100
  libavcodec     57. 92.100 / 57. 92.100
  libavformat    57. 72.101 / 57. 72.101
  libavdevice    57.  7.100 / 57.  7.100
  libavfilter     6. 84.101 /  6. 84.101
  libswscale      4.  7.101 /  4.  7.101
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
[utvideo @ 0686cf90] Warning: not compiled with thread support, using thread emu
lation
Input #0, avi, from 'utv.avi':
  Duration: 00:00:03.20, start: 0.000000, bitrate: 11583 kb/s
    Stream #0:0: Video: utvideo (ULRG / 0x47524C55), rgb24, 500x500, 11933 kb/s,
 10 fps, 10 tbr, 10 tbn, 10 tbc
File 'out.avi' already exists. Overwrite ? [y/N] y
[utvideo @ 06c1e228] Warning: not compiled with thread support, using thread emu
lation
Stream mapping:
  Stream #0:0 -> #0:0 (utvideo (native) -> mpeg4 (native))
Press [q] to stop, [?] for help
[swscaler @ 06c282c0] Warning: data is not aligned! This can lead to a speedloss


Program received signal SIGSEGV, Segmentation fault.
0x01006773 in ff_rgb24ToY_ssse3 ()
(gdb) bt
#0  0x01006773 in ff_rgb24ToY_ssse3 ()
#1  0x00ffdc0d in lum_convert (c=0x6c282c0, desc=0x686c3c0, sliceY=0,
    sliceH=7) at libswscale/hscale.c:108
#2  0x00fa49d2 in swscale (c=0x6c282c0, src=0x22ef80, srcStride=0x22efa0,
    srcSliceY=0, srcSliceH=500, dst=0x22ef90, dstStride=0x22efb0)
    at libswscale/swscale.c:460
#3  0x00fa5e5e in sws_scale (c=0x6c282c0, srcSlice=0x22f010,
    srcStride=0x22f030, srcSliceY=0, srcSliceH=500, dst=0x22f020,
    dstStride=0x22f040) at libswscale/swscale.c:980
#4  0x004f5b44 in scale_slice (out_buf=<optimized out>, cur_pic=0x6c4c950,
    sws=0x6c282c0, y=0, h=500, mul=1, field=0, link=<optimized out>)
    at libavfilter/vf_scale.c:399
#5  0x004f61f8 in filter_frame (link=0x6869930, in=0x6c4c950)
    at libavfilter/vf_scale.c:520
#6  0x0042b560 in ff_filter_frame_framed (frame=0x6c4c950, link=0x6869930)
    at libavfilter/avfilter.c:1116
#7  ff_filter_frame_to_filter (link=0x6869930) at libavfilter/avfilter.c:1264
#8  ff_filter_activate_default (filter=<optimized out>)
    at libavfilter/avfilter.c:1315
#9  ff_filter_activate (filter=0x686a130) at libavfilter/avfilter.c:1476
#10 0x004300fc in push_frame (graph=0x6869190) at libavfilter/buffersrc.c:181
#11 av_buffersrc_add_frame_internal (ctx=<optimized out>, frame=0x6c1df38,
    flags=<optimized out>) at libavfilter/buffersrc.c:262
#12 0x00430467 in av_buffersrc_add_frame_flags (ctx=0x6869270,
    frame=0x6c1df38, flags=4) at libavfilter/buffersrc.c:164
#13 0x0041ddb8 in ifilter_send_frame (frame=0x6c1df38, ifilter=0x6c1fe68)
    at ffmpeg.c:2206
#14 send_frame_to_filters (ist=0x6c1e0c0, decoded_frame=0x6c1df38)
    at ffmpeg.c:2284
#15 0x0041e9b5 in decode_video (decode_failed=<optimized out>,
    eof=<optimized out>, got_output=<optimized out>, pkt=<optimized out>,
    ist=<optimized out>) at ffmpeg.c:2484
#16 process_input_packet (ist=0x6c1e0c0, pkt=0x22f6f8, no_eof=0)
    at ffmpeg.c:2633
#17 0x00421335 in process_input (file_index=1) at ffmpeg.c:4390
#18 0x00423d8f in transcode_step () at ffmpeg.c:4501
#19 transcode () at ffmpeg.c:4555
#20 0x01049919 in main (argc=4, argv=<optimized out>) at ffmpeg.c:4760
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0x1006753 to 0x1006793:
   0x01006753 <ff_rgb24ToY_ssse3+115>:  pop    %edi
   0x01006754 <ff_rgb24ToY_ssse3+116>:  jo     0x10067bc <ff_bgr24ToY_ssse3+44>
   0x01006756 <ff_rgb24ToY_ssse3+118>:  paddd  %mm1,%mm0
   0x01006759 <ff_rgb24ToY_ssse3+121>:  paddd  %xmm3,%xmm2
   0x0100675d <ff_rgb24ToY_ssse3+125>:  paddd  %xmm4,%xmm0
   0x01006761 <ff_rgb24ToY_ssse3+129>:  paddd  %xmm4,%xmm2
   0x01006765 <ff_rgb24ToY_ssse3+133>:  psrad  $0x9,%xmm0
   0x0100676a <ff_rgb24ToY_ssse3+138>:  psrad  $0x9,%xmm2
   0x0100676f <ff_rgb24ToY_ssse3+143>:  packssdw %xmm2,%xmm0
=> 0x01006773 <ff_rgb24ToY_ssse3+147>:  movdqa %xmm0,(%eax,%esi,1)
   0x01006778 <ff_rgb24ToY_ssse3+152>:  add    $0x10,%esi
   0x0100677b <ff_rgb24ToY_ssse3+155>:  jl     0x1006711 <ff_rgb24ToY_ssse3+49>
   0x0100677d <ff_rgb24ToY_ssse3+157>:  pop    %edi
   0x0100677e <ff_rgb24ToY_ssse3+158>:  pop    %esi
   0x0100677f <ff_rgb24ToY_ssse3+159>:  pop    %ebx
   0x01006780 <ff_rgb24ToY_ssse3+160>:  ret
   0x01006781 <ff_rgb24ToY_ssse3+161>:  nopl   0x0(%eax)
   0x01006788 <ff_rgb24ToY_ssse3+168>:  nopl   0x0(%eax,%eax,1)
   0x01006790 <ff_bgr24ToY_ssse3+0>:    push   %ebx
   0x01006791 <ff_bgr24ToY_ssse3+1>:    push   %esi
   0x01006792 <ff_bgr24ToY_ssse3+2>:    push   %edi
End of assembler dump.
(gdb) info all-registers
eax            0x6c37bb0        113474480
ecx            0x6cf0638        114230840
edx            0x6cf0620        114230816
ebx            0x6cf0620        114230816
esp            0x22ed80 0x22ed80
ebp            0x4      0x4
esi            0xfffffc18       -1000
edi            0x6c2dc30        113433648
eip            0x1006773        0x1006773 <ff_rgb24ToY_ssse3+147>
eflags         0x10202  [ IF RF ]
cs             0x1b     27
ss             0x23     35
ds             0x23     35
es             0x23     35
fs             0x3b     59
gs             0x0      0
st0            -513     (raw 0xc0088040000000000000)
st1            -513     (raw 0xc0088040000000000000)
st2            128      (raw 0x40068000000000000000)
st3            128      (raw 0x40068000000000000000)
st4            128      (raw 0x40068000000000000000)
st5            1519964783640576 (raw 0x4031accccc2000000000)
st6            18014398509481.984       (raw 0x402b83126e978d4fdf3b)
st7            0.002    (raw 0x3ff683126e978d4fdf3b)
fctrl          0xffff037f       -64641
fstat          0xffff0020       -65504
ftag           0xffffffff       -1
fiseg          0x1b     27
fioff          0x10c0103        17563907
foseg          0xffff0023       -65501
fooff          0x22ec50 2288720
fop            0x0      0
xmm0           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x31, 0x2d, 0x31, 0x2d, 0x31, 0x2d, 0x31, 0x2d, 0x31, 0x2d,
    0x31, 0x2d, 0x31, 0x2d, 0x31, 0x2d}, v8_int16 = {0x2d31, 0x2d31, 0x2d31,
    0x2d31, 0x2d31, 0x2d31, 0x2d31, 0x2d31}, v4_int32 = {0x2d312d31,
    0x2d312d31, 0x2d312d31, 0x2d312d31}, v2_int64 = {0x2d312d312d312d31,
    0x2d312d312d312d31}, uint128 = 0x2d312d312d312d312d312d312d312d31}
xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x98, 0x1c, 0xb, 0x0, 0xf9, 0x4d, 0x41, 0x0, 0x98, 0x1c, 0xb,
    0x0, 0xf9, 0x4d, 0x41, 0x0}, v8_int16 = {0x1c98, 0xb, 0x4df9, 0x41,
    0x1c98, 0xb, 0x4df9, 0x41}, v4_int32 = {0xb1c98, 0x414df9, 0xb1c98,
    0x414df9}, v2_int64 = {0x414df9000b1c98, 0x414df9000b1c98},
  uint128 = 0x00414df9000b1c9800414df9000b1c98}
xmm2           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x31, 0x2d, 0x0, 0x0, 0x31, 0x2d, 0x0, 0x0, 0x31, 0x2d, 0x0,
    0x0, 0x31, 0x2d, 0x0, 0x0}, v8_int16 = {0x2d31, 0x0, 0x2d31, 0x0, 0x2d31,
    0x0, 0x2d31, 0x0}, v4_int32 = {0x2d31, 0x2d31, 0x2d31, 0x2d31},
  v2_int64 = {0x2d3100002d31, 0x2d3100002d31},
  uint128 = 0x00002d3100002d3100002d3100002d31}
xmm3           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x98, 0x1c, 0xb, 0x0, 0xf9, 0x4d, 0x41, 0x0, 0x98, 0x1c, 0xb,
    0x0, 0xf9, 0x4d, 0x41, 0x0}, v8_int16 = {0x1c98, 0xb, 0x4df9, 0x41,
    0x1c98, 0xb, 0x4df9, 0x41}, v4_int32 = {0xb1c98, 0x414df9, 0xb1c98,
    0x414df9}, v2_int64 = {0x414df9000b1c98, 0x414df9000b1c98},
  uint128 = 0x00414df9000b1c9800414df9000b1c98}
xmm4           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x0, 0x1, 0x8, 0x0, 0x0, 0x1, 0x8, 0x0, 0x0, 0x1, 0x8, 0x0,
    0x0, 0x1, 0x8, 0x0}, v8_int16 = {0x100, 0x8, 0x100, 0x8, 0x100, 0x8,
    0x100, 0x8}, v4_int32 = {0x80100, 0x80100, 0x80100, 0x80100}, v2_int64 = {
    0x8010000080100, 0x8010000080100},
  uint128 = 0x00080100000801000008010000080100}
xmm5           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
  uint128 = 0x00000000000000000000000000000000}
xmm6           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
  uint128 = 0x00000000000000000000000000000000}
xmm7           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
  v16_int8 = {0x0, 0x80, 0x1, 0x80, 0x2, 0x80, 0x3, 0x80, 0x6, 0x80, 0x7,
    0x80, 0x8, 0x80, 0x9, 0x80}, v8_int16 = {0x8000, 0x8001, 0x8002, 0x8003,
    0x8006, 0x8007, 0x8008, 0x8009}, v4_int32 = {0x80018000, 0x80038002,
    0x80078006, 0x80098008}, v2_int64 = {0x8003800280018000,
    0x8009800880078006}, uint128 = 0x80098008800780068003800280018000}
mxcsr          0x1f80   [ IM DM ZM OM UM PM ]
mm0            {uint64 = 0x8040000000000000, v2_int32 = {0x0, 0x80400000},
  v4_int16 = {0x0, 0x0, 0x0, 0x8040}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x40, 0x80}}
mm1            {uint64 = 0x8040000000000000, v2_int32 = {0x0, 0x80400000},
  v4_int16 = {0x0, 0x0, 0x0, 0x8040}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x40, 0x80}}
mm2            {uint64 = 0x8000000000000000, v2_int32 = {0x0, 0x80000000},
  v4_int16 = {0x0, 0x0, 0x0, 0x8000}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x0, 0x80}}
mm3            {uint64 = 0x8000000000000000, v2_int32 = {0x0, 0x80000000},
  v4_int16 = {0x0, 0x0, 0x0, 0x8000}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x0, 0x80}}
mm4            {uint64 = 0x8000000000000000, v2_int32 = {0x0, 0x80000000},
  v4_int16 = {0x0, 0x0, 0x0, 0x8000}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x0, 0x80}}
mm5            {uint64 = 0xaccccc2000000000, v2_int32 = {0x0, 0xaccccc20},
  v4_int16 = {0x0, 0x0, 0xcc20, 0xaccc}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x20,
    0xcc, 0xcc, 0xac}}
mm6            {uint64 = 0x83126e978d4fdf3b, v2_int32 = {0x8d4fdf3b,
    0x83126e97}, v4_int16 = {0xdf3b, 0x8d4f, 0x6e97, 0x8312}, v8_int8 = {
    0x3b, 0xdf, 0x4f, 0x8d, 0x97, 0x6e, 0x12, 0x83}}
mm7            {uint64 = 0x83126e978d4fdf3b, v2_int32 = {0x8d4fdf3b,
    0x83126e97}, v4_int16 = {0xdf3b, 0x8d4f, 0x6e97, 0x8312}, v8_int8 = {
    0x3b, 0xdf, 0x4f, 0x8d, 0x97, 0x6e, 0x12, 0x83}}
(gdb)

[Carl Eugen Hoyos]

I cannot reproduce but I was unable to get the alignment warning.

[ami_stuff]

This is likely only reproducible with builds made with original mingw32. There is also no alignment warning (and no crash) when running FFmpeg with "-cpuflags 0".

[ami_stuff]

does not happen with current snapshot