Opened 6 years ago

Last modified 6 years ago

#6763 new defect

swscale: Out-of-bounds memory accesses

Reported by: Gramner Owned by:
Priority: important Component: swscale
Version: git-master Keywords: crash
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Many assembly functions in swscale will read past the end of their input buffers which causes segfaults and/or bus errors if the buffer happens to be located near the end of a memory page and the next page is invalid.

Aligning input buffers isn't even enough for formats like RGB24 (and requiring alignment would be a bad idea anyway since it wouldn't work with memory-mapped input files for example).

Using swscale with x264 CLI seems to be a fairly consistent way to trigger such out-of-bounds crashes. This command line for example will cause segfaults in ff_rgb24ToY_avx():

./x264 -o /dev/null --input-csp rgb --input-res 512x512 <any_input_file>

If asm is disabled in swscale the problem goes away.

Change History (2)

comment:1 by Carl Eugen Hoyos, 6 years ago

Keywords: crash added
Priority: normalimportant

(Version information, backtrace, disassembly and register dump missing.)
Could you confirm that your input buffers are sufficiently padded?

comment:2 by Gramner, 6 years ago

Version: unspecifiedgit-master

Not sure which disassembly you're interested in, but the source shows the entire row loop done using 128-bit loads with no special handling for the tail:;a=blob;f=libswscale/x86/input.asm;h=af9afcaa53a74f51eaa1257bafd1052524046074;hb=HEAD#l243

With width=512 as an example, on the last loop iteration "movu m4, [srcq+12]" reads 16 bytes from offset 1524 which results in an overflow of 4 bytes.

The input buffer is the exact size of the input data with zero padding as no such requirement is documented.

Note: See TracTickets for help on using tickets.