ffmpeg crashed when libx264rgb and preset is used

[Patrick Dung]

Summary of the bug:
How to reproduce:
This command runs:
ffmpeg -i P1160144.MOV -vcodec libx264rgb -an test.mov

This command would crash:
ffmpeg -i P1160144.MOV -vcodec libx264rgb -an -preset ultrafast test.mov

cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1

ffmpeg: encoder/slicetype.c:1993: x264_rc_analyse_slice: Assertion `cost >= 0' failed.
Aborted (core dumped)

(gdb) where
#0 0x00007f4e6519a8df in raise () from /lib64/libc.so.6
#1 0x00007f4e6519c4da in abort () from /lib64/libc.so.6
#2 0x00007f4e65192d67 in assert_fail_base () from /lib64/libc.so.6
#3 0x00007f4e65192e12 in assert_fail () from /lib64/libc.so.6
#4 0x00007f4e6a56f144 in x264_rc_analyse_slice () from /usr/local/lib/libx264.so.152
#5 0x00007f4e6a57c224 in ?? () from /usr/local/lib/libx264.so.152
#6 0x00007f4e6a581dc8 in x264_ratecontrol_start () from /usr/local/lib/libx264.so.152
#7 0x00007f4e6a5acb19 in x264_encoder_encode () from /usr/local/lib/libx264.so.152
#8 0x00000000009172b3 in ?? ()
#9 0x00000000007953fe in ?? ()
#10 0x00000000007956db in ?? ()
#11 0x0000000000795668 in ?? ()
#12 0x000000000042b736 in ?? ()
#13 0x000000000042a67f in ?? ()
#14 0x0000000000423522 in ?? ()
#15 0x0000000000420134 in main ()

ffmpeg version N-87359-g67da268 Copyright (c) 2000-2017 the FFmpeg developers

built with clang version 3.9.1 (tags/RELEASE_391/final)

[Carl Eugen Hoyos]

Please provide backtrace, disassembly and register dump as explained on https://ffmpeg.org/bugreports.html

[Patrick Dung]

output of gdb debug

[Patrick Dung]

output of valgrind debug

[Patrick Dung]

Please note the related debug message is attached.

[Carl Eugen Hoyos]

==16876== Process terminating with default action of signal 6 (SIGABRT): dumping core
==16876==    at 0xD8F58DF: raise (in /usr/lib64/libc-2.24.so)
==16876==    by 0xD8F74D9: abort (in /usr/lib64/libc-2.24.so)
==16876==    by 0xD8EDD66: __assert_fail_base (in /usr/lib64/libc-2.24.so)
==16876==    by 0xD8EDE11: __assert_fail (in /usr/lib64/libc-2.24.so)
==16876==    by 0x85E4143: x264_rc_analyse_slice (in /usr/local/lib/libx264.so.152)
==16876==    by 0x85F1223: ??? (in /usr/local/lib/libx264.so.152)
==16876==    by 0x85F6DC7: x264_ratecontrol_start (in /usr/local/lib/libx264.so.152)
==16876==    by 0x8621B18: x264_encoder_encode (in /usr/local/lib/libx264.so.152)
==16876==    by 0x9172B2: X264_frame (in /usr/local/src/ffmpeg/ffmpeg_g)
==16876==    by 0x7953FD: avcodec_encode_video2 (in /usr/local/src/ffmpeg/ffmpeg_g)
==16876==    by 0x7956DA: do_encode (in /usr/local/src/ffmpeg/ffmpeg_g)
==16876==    by 0x795667: avcodec_send_frame (in /usr/local/src/ffmpeg/ffmpeg_g)

Even if there is a bug in FFmpeg, libx264 should not abort because of bad input.
Feel free to provide a sample, but please recompile with (only) --enable-gpl --enable-libx264 and report to libx264 if the issue is still reproducible.