swscale: Out-of-bounds memory accesses
|Reported by:||Gramner||Owned by:|
|Blocking:||Reproduced by developer:||no|
|Analyzed by developer:||no|
Many assembly functions in swscale will read past the end of their input buffers which causes segfaults and/or bus errors if the buffer happens to be located near the end of a memory page and the next page is invalid.
Aligning input buffers isn't even enough for formats like RGB24 (and requiring alignment would be a bad idea anyway since it wouldn't work with memory-mapped input files for example).
Using swscale with x264 CLI seems to be a fairly consistent way to trigger such out-of-bounds crashes. This command line for example will cause segfaults in ff_rgb24ToY_avx():
./x264 -o /dev/null --input-csp rgb --input-res 512x512 <any_input_file>
If asm is disabled in swscale the problem goes away.