swscale: Out-of-bounds memory accesses

[Gramner]

Many assembly functions in swscale will read past the end of their input buffers which causes segfaults and/or bus errors if the buffer happens to be located near the end of a memory page and the next page is invalid.

Aligning input buffers isn't even enough for formats like RGB24 (and requiring alignment would be a bad idea anyway since it wouldn't work with memory-mapped input files for example).

Using swscale with x264 CLI seems to be a fairly consistent way to trigger such out-of-bounds crashes. This command line for example will cause segfaults in ff_rgb24ToY_avx():

./x264 -o /dev/null --input-csp rgb --input-res 512x512 <any_input_file>

If asm is disabled in swscale the problem goes away.

[Carl Eugen Hoyos]

(Version information, backtrace, disassembly and register dump missing.)
Could you confirm that your input buffers are sufficiently padded?

[Gramner]

Not sure which disassembly you're interested in, but the source shows the entire row loop done using 128-bit loads with no special handling for the tail: ​http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libswscale/x86/input.asm;h=af9afcaa53a74f51eaa1257bafd1052524046074;hb=HEAD#l243

With width=512 as an example, on the last loop iteration "movu m4, [srcq+12]" reads 16 bytes from offset 1524 which results in an overflow of 4 bytes.

The input buffer is the exact size of the input data with zero padding as no such requirement is documented.