Scaling bayer crashes libswscale

[Carl Eugen Hoyos]

$ valgrind ./ffmpeg_g -cpuflags 0 -f rawvideo -s pal -pix_fmt bayer_rggb16le -i /dev/zero -s cif -f null -
==3875== Memcheck, a memory error detector
==3875== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3875== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3875== Command: ./ffmpeg_g -cpuflags 0 -f rawvideo -s pal -pix_fmt bayer_rggb16le -i /dev/zero -s cif -f null -
==3875==
ffmpeg version N-67086-gdd3f156 Copyright (c) 2000-2014 the FFmpeg developers
  built on Oct 22 2014 00:56:03 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      54. 10.100 / 54. 10.100
  libavcodec     56.  8.102 / 56.  8.102
  libavformat    56.  9.101 / 56.  9.101
  libavdevice    56.  1.100 / 56.  1.100
  libavfilter     5.  2.100 /  5.  2.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, rawvideo, from '/dev/zero':
  Duration: N/A, start: 0.000000, bitrate: 165888 kb/s
    Stream #0:0: Video: rawvideo ([186]RG[16] / 0x104752BA), bayer_rggb16le, 720x576, 165888 kb/s, 25 tbr, 25 tbn, 25 tbc
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf56.9.101
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 352x288, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.8.102 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Press [q] to stop, [?] for help
==3875== Invalid read of size 2
==3875==    at 0xDDE610: hScale16To15_c (swscale.c:111)
==3875==    by 0xDDFA0C: swscale (swscale.c:287)
==3875==    by 0xDE12C1: sws_scale (swscale.c:1088)
==3875==    by 0x4EDA84: filter_frame (vf_scale.c:429)
==3875==    by 0x4981BD: ff_filter_frame_framed (avfilter.c:1098)
==3875==    by 0x4986C0: default_filter_frame (avfilter.c:1178)
==3875==    by 0x4981BD: ff_filter_frame_framed (avfilter.c:1098)
==3875==    by 0x499308: ff_filter_frame (avfilter.c:1178)
==3875==    by 0x49D5B1: request_frame (buffersrc.c:499)
==3875==    by 0x49D84A: av_buffersrc_add_frame_internal (buffersrc.c:181)
==3875==    by 0x49DBDC: av_buffersrc_add_frame_flags (buffersrc.c:106)
==3875==    by 0x483301: decode_video (ffmpeg.c:1989)
==3875==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==3875==
==3875==
==3875== Process terminating with default action of signal 11 (SIGSEGV)
==3875==  Access not within mapped region at address 0x0
==3875==    at 0xDDE610: hScale16To15_c (swscale.c:111)
==3875==    by 0xDDFA0C: swscale (swscale.c:287)
==3875==    by 0xDE12C1: sws_scale (swscale.c:1088)
==3875==    by 0x4EDA84: filter_frame (vf_scale.c:429)
==3875==    by 0x4981BD: ff_filter_frame_framed (avfilter.c:1098)
==3875==    by 0x4986C0: default_filter_frame (avfilter.c:1178)
==3875==    by 0x4981BD: ff_filter_frame_framed (avfilter.c:1098)
==3875==    by 0x499308: ff_filter_frame (avfilter.c:1178)
==3875==    by 0x49D5B1: request_frame (buffersrc.c:499)
==3875==    by 0x49D84A: av_buffersrc_add_frame_internal (buffersrc.c:181)
==3875==    by 0x49DBDC: av_buffersrc_add_frame_flags (buffersrc.c:106)
==3875==    by 0x483301: decode_video (ffmpeg.c:1989)
==3875==  If you believe this happened as a result of a stack
==3875==  overflow in your program's main thread (unlikely but
==3875==  possible), you can try to increase the size of the
==3875==  main thread stack using the --main-stacksize= flag.
==3875==  The main thread stack size used in this run was 8388608.
==3875==
==3875== HEAP SUMMARY:
==3875==     in use at exit: 1,272,590 bytes in 161 blocks
==3875==   total heap usage: 1,521 allocs, 1,360 frees, 1,913,121 bytes allocated
==3875==
==3875== LEAK SUMMARY:
==3875==    definitely lost: 0 bytes in 0 blocks
==3875==    indirectly lost: 0 bytes in 0 blocks
==3875==      possibly lost: 2,736 bytes in 9 blocks
==3875==    still reachable: 1,269,854 bytes in 152 blocks
==3875==         suppressed: 0 bytes in 0 blocks
==3875== Rerun with --leak-check=full to see details of leaked memory
==3875==
==3875== For counts of detected and suppressed errors, rerun with: -v
==3875== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
Killed

(gdb) r -f rawvideo -s pal -pix_fmt bayer_rggb16le -i /dev/zero -s cif -f null -
Starting program: ffmpeg_g -f rawvideo -s pal -pix_fmt bayer_rggb16le -i /dev/zero -s cif -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-67086-gdd3f156 Copyright (c) 2000-2014 the FFmpeg developers
  built on Oct 22 2014 00:56:03 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      54. 10.100 / 54. 10.100
  libavcodec     56.  8.102 / 56.  8.102
  libavformat    56.  9.101 / 56.  9.101
  libavdevice    56.  1.100 / 56.  1.100
  libavfilter     5.  2.100 /  5.  2.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, rawvideo, from '/dev/zero':
  Duration: N/A, start: 0.000000, bitrate: 165888 kb/s
    Stream #0:0: Video: rawvideo ([186]RG[16] / 0x104752BA), bayer_rggb16le, 720x576, 165888 kb/s, 25 tbr, 25 tbn, 25 tbc
[New Thread 0x7ffff1afe700 (LWP 3909)]
[New Thread 0x7ffff12fd700 (LWP 3910)]
[New Thread 0x7ffff0afc700 (LWP 3911)]
[New Thread 0x7ffff02fb700 (LWP 3912)]
[New Thread 0x7fffefafa700 (LWP 3913)]
[New Thread 0x7fffef2f9700 (LWP 3914)]
[New Thread 0x7fffeeaf8700 (LWP 3915)]
[New Thread 0x7fffee2f7700 (LWP 3916)]
[New Thread 0x7fffedaf6700 (LWP 3917)]
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf56.9.101
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 352x288, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.8.102 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
ff_hscale14to15_8_ssse3.loop () at libswscale/x86/scale.asm:429
429     SCALE_FUNCS2 6, 6, 8
(gdb) bt
#0  ff_hscale14to15_8_ssse3.loop () at libswscale/x86/scale.asm:429
#1  0x0000000000ddfa0d in hcscale (pal=0x1a43660, formatConvBuffer=0x1a48e80 "P",
    hChrFilterSize=8, hChrFilterPos=0x1a4c200, hChrFilter=0x1a3c960, xInc=67025, srcW=360,
    src_in=0x7fffffffd050, dstWidth=352, dst2=0x1a3a1c0, dst1=0x1a39ea0, c=0x1a3f4e0)
    at libswscale/swscale.c:287
#2  swscale (c=0x1a3f4e0, src=0x7fffffffd130, srcStride=0x7fffffffd110, srcSliceY=0,
    srcSliceH=576, dst=0x7fffffffd150, dstStride=0x7fffffffd120) at libswscale/swscale.c:508
#3  0x0000000000de12c2 in sws_scale (c=<optimized out>,
    srcSlice=srcSlice@entry=0x7fffffffd240, srcStride=srcStride@entry=0x7fffffffd200,
    srcSliceY=srcSliceY@entry=0, srcSliceH=576, dst=dst@entry=0x7fffffffd260,
    dstStride=0x7fffffffd210) at libswscale/swscale.c:1088
#4  0x00000000004eda85 in scale_slice (field=<optimized out>, mul=<optimized out>,
    h=<optimized out>, sws=<optimized out>, cur_pic=<optimized out>, out_buf=<optimized out>,
    link=<optimized out>, y=<optimized out>) at libavfilter/vf_scale.c:429
#5  filter_frame (link=link@entry=0x1a487c0, in=0x1a5c240) at libavfilter/vf_scale.c:526
#6  0x00000000004981be in ff_filter_frame_framed (link=link@entry=0x1a487c0, frame=0x1a3a160,
    frame@entry=0x1a5c240) at libavfilter/avfilter.c:1098
#7  0x00000000004986c1 in ff_filter_frame (frame=0x1a5c240, link=0x1a487c0)
    at libavfilter/avfilter.c:1178
#8  default_filter_frame (link=link@entry=0x1a3eba0, frame=0x1a5c240)
    at libavfilter/avfilter.c:1009
#9  0x00000000004981be in ff_filter_frame_framed (link=link@entry=0x1a3eba0, frame=0x1a3a160,
    frame@entry=0x1a5c240) at libavfilter/avfilter.c:1098
#10 0x0000000000499309 in ff_filter_frame (link=link@entry=0x1a3eba0, frame=0x1a5c240)
    at libavfilter/avfilter.c:1178
#11 0x000000000049d5b2 in request_frame (link=0x1a3eba0) at libavfilter/buffersrc.c:499
#12 0x000000000049d84b in av_buffersrc_add_frame_internal (ctx=ctx@entry=0x1a46e80,
    frame=frame@entry=0x1a5bc80, flags=flags@entry=4) at libavfilter/buffersrc.c:181
#13 0x000000000049dbdd in av_buffersrc_add_frame_flags (ctx=0x1a46e80,
    frame=frame@entry=0x1a5bc80, flags=flags@entry=4) at libavfilter/buffersrc.c:106
#14 0x0000000000483302 in decode_video (ist=ist@entry=0x1a58d00, pkt=pkt@entry=0x7fffffffda10,
    got_output=got_output@entry=0x7fffffffd78c) at ffmpeg.c:1989
#15 0x0000000000486adc in process_input_packet (pkt=0x7fffffffd9b0, ist=0x1a58d00)
    at ffmpeg.c:2123
#16 process_input (file_index=27587328) at ffmpeg.c:3541
#17 0x000000000046c351 in transcode_step () at ffmpeg.c:3635
#18 transcode () at ffmpeg.c:3687
#19 main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3863
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0xe3243e to 0xe3247e:
   0x0000000000e3243e <ff_hscale14to15_4_ssse3.loop+76>:        retq
   0x0000000000e3243f <ff_hscale14to15_4_ssse3.loop+77>:        nop
   0x0000000000e32440 <ff_hscale14to15_8_ssse3+0>:      movslq %edx,%rdx
   0x0000000000e32443 <ff_hscale14to15_8_ssse3+3>:      shl    %rdx
   0x0000000000e32446 <ff_hscale14to15_8_ssse3+6>:      lea    (%r8,%rdx,8),%r8
   0x0000000000e3244a <ff_hscale14to15_8_ssse3+10>:     lea    (%rsi,%rdx,1),%rsi
   0x0000000000e3244e <ff_hscale14to15_8_ssse3+14>:     lea    (%r9,%rdx,2),%r9
   0x0000000000e32452 <ff_hscale14to15_8_ssse3+18>:     neg    %rdx
   0x0000000000e32455 <ff_hscale14to15_8_ssse3.loop+0>: movslq (%r9,%rdx,2),%rdi
   0x0000000000e32459 <ff_hscale14to15_8_ssse3.loop+4>: movslq 0x4(%r9,%rdx,2),%rax
=> 0x0000000000e3245e <ff_hscale14to15_8_ssse3.loop+9>: movdqu (%rcx,%rdi,2),%xmm0
   0x0000000000e32463 <ff_hscale14to15_8_ssse3.loop+14>:        movdqu (%rcx,%rax,2),%xmm1
   0x0000000000e32468 <ff_hscale14to15_8_ssse3.loop+19>:        movslq 0x8(%r9,%rdx,2),%rdi
   0x0000000000e3246d <ff_hscale14to15_8_ssse3.loop+24>:        movslq 0xc(%r9,%rdx,2),%rax
   0x0000000000e32472 <ff_hscale14to15_8_ssse3.loop+29>:        movdqu (%rcx,%rdi,2),%xmm4
   0x0000000000e32477 <ff_hscale14to15_8_ssse3.loop+34>:        movdqu (%rcx,%rax,2),%xmm5
   0x0000000000e3247c <ff_hscale14to15_8_ssse3.loop+39>:        pmaddwd (%r8,%rdx,8),%xmm0
End of assembler dump.
(gdb) info register
rax            0x0      0
rbx            0x0      0
rcx            0x0      0
rdx            0xfffffffffffffd40       -704
rsi            0x1a3a160        27500896
rdi            0x0      0
rbp            0x1a3f4e0        0x1a3f4e0
rsp            0x7fffffffce18   0x7fffffffce18
r8             0x1a3df60        27516768
r9             0x1a4c780        27576192
r10            0x2b8    696
r11            0x0      0
r12            0x7fffffffd130   140737488343344
r13            0x7fffffffd110   140737488343312
r14            0x0      0
r15            0x0      0
rip            0xe3245e 0xe3245e <ff_hscale14to15_8_ssse3.loop+9>
eflags         0x10283  [ CF SF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

[Michael Niedermayer]

Fixed in 2f6bb86f85886a7fb36e8a10e4dd8cc3a1849377