SEGV bug at /fftools/ffmpeg_enc.c in set_encoder_id in FFmpeg

[ZengYunxiang]

Summary of the bug:

Dear developers,
I found the following SEGV bug on FFmpeg6.1 when using both transpose and gradfun filters, please confirm.

The poc file(poc6ffmpeg) will be attached to this ticket.

How to reproduce:

git clone https://github.com/FFmpeg/FFmpeg.git ffmpeg6-1
cd ffmpeg6-1
git checkout 466799d
./configure --cc=clang --cxx=clang++ --ld=clang --enable-debug --toolchain=clang-asan
make -j30

./ffmpeg_g -y -i poc6ffmpeg -filter_complex transpose,gradfun tmp.mp4

ASAN Log:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==324330==ERROR: AddressSanitizer: SEGV on unknown address 0x631000050000 (pc 0x000000dfa6df bp 0x7fffffffc450 sp 0x7fffffffc238 T0)
==324330==The signal is caused by a READ memory access.
    #0 0xdfa6de in ff_gradfun_blur_line_movdqa_sse2 (/ffmpeg6-1/ffmpeg_g+0xdfa6de)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/ffmpeg6-1/ffmpeg_g+0xdfa6de) in ff_gradfun_blur_line_movdqa_sse2
==324330==ABORTING

ffmpeg version:

# ./ffmpeg -version
ffmpeg version n6.1-3-g466799d4f5 Copyright (c) 2000-2023 the FFmpeg developers
built with clang version 9.0.0 (https://github.com/llvm-mirror/llvm c62b24f070c9a4bb1a76409e623042a740cac4cd)
configuration: --cc=clang --cxx=clang++ --ld=clang --enable-debug --toolchain=clang-asan
libavutil      58. 29.100 / 58. 29.100
libavcodec     60. 31.102 / 60. 31.102
libavformat    60. 16.100 / 60. 16.100
libavdevice    60.  3.100 / 60.  3.100
libavfilter     9. 12.100 /  9. 12.100
libswscale      7.  5.100 /  7.  5.100
libswresample   4. 12.100 /  4. 12.100

Credit:

Zeng Yunxiang

Thanks for your time!

[ZengYunxiang]

POC file

[ZengYunxiang]

The title misdescribes the location of the vulnerability.The trigger location of the vulnerability is in function ff_gradfun_blur_line_movdqa_sse2.

[Michael Niedermayer]

fixed by e4d2666bdc3dbd177a81bbf428654a5f2fa3787a