#6692 closed defect (invalid)

ffmpeg crashed when libx264rgb and preset is used

Reported by: dkt Owned by:
Priority: important Component: undetermined
Version: unspecified Keywords: libx264
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Summary of the bug:
How to reproduce:
This command runs:
ffmpeg -i P1160144.MOV -vcodec libx264rgb -an test.mov

This command would crash:
ffmpeg -i P1160144.MOV -vcodec libx264rgb -an -preset ultrafast test.mov

cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1

ffmpeg: encoder/slicetype.c:1993: x264_rc_analyse_slice: Assertion `cost >= 0' failed.
Aborted (core dumped)

(gdb) where
#0 0x00007f4e6519a8df in raise () from /lib64/libc.so.6
#1 0x00007f4e6519c4da in abort () from /lib64/libc.so.6
#2 0x00007f4e65192d67 in assert_fail_base () from /lib64/libc.so.6
#3 0x00007f4e65192e12 in
assert_fail () from /lib64/libc.so.6
#4 0x00007f4e6a56f144 in x264_rc_analyse_slice () from /usr/local/lib/libx264.so.152
#5 0x00007f4e6a57c224 in ?? () from /usr/local/lib/libx264.so.152
#6 0x00007f4e6a581dc8 in x264_ratecontrol_start () from /usr/local/lib/libx264.so.152
#7 0x00007f4e6a5acb19 in x264_encoder_encode () from /usr/local/lib/libx264.so.152
#8 0x00000000009172b3 in ?? ()
#9 0x00000000007953fe in ?? ()
#10 0x00000000007956db in ?? ()
#11 0x0000000000795668 in ?? ()
#12 0x000000000042b736 in ?? ()
#13 0x000000000042a67f in ?? ()
#14 0x0000000000423522 in ?? ()
#15 0x0000000000420134 in main ()

ffmpeg version N-87359-g67da268 Copyright (c) 2000-2017 the FFmpeg developers

built with clang version 3.9.1 (tags/RELEASE_391/final)

Attachments (2)

gdb debug.txt (12.3 KB) - added by dkt 10 months ago.
output of gdb debug
valgrind debug.txt (6.2 KB) - added by dkt 10 months ago.
output of valgrind debug

Download all attachments as: .zip

Change History (5)

comment:1 Changed 10 months ago by cehoyos

  • Component changed from ffmpeg to undetermined
  • Keywords libx264 added
  • Priority changed from normal to important

Please provide backtrace, disassembly and register dump as explained on https://ffmpeg.org/bugreports.html

Changed 10 months ago by dkt

output of gdb debug

Changed 10 months ago by dkt

output of valgrind debug

comment:2 Changed 10 months ago by dkt

Please note the related debug message is attached.

comment:3 Changed 10 months ago by cehoyos

  • Resolution set to invalid
  • Status changed from new to closed
==16876== Process terminating with default action of signal 6 (SIGABRT): dumping core
==16876==    at 0xD8F58DF: raise (in /usr/lib64/libc-2.24.so)
==16876==    by 0xD8F74D9: abort (in /usr/lib64/libc-2.24.so)
==16876==    by 0xD8EDD66: __assert_fail_base (in /usr/lib64/libc-2.24.so)
==16876==    by 0xD8EDE11: __assert_fail (in /usr/lib64/libc-2.24.so)
==16876==    by 0x85E4143: x264_rc_analyse_slice (in /usr/local/lib/libx264.so.152)
==16876==    by 0x85F1223: ??? (in /usr/local/lib/libx264.so.152)
==16876==    by 0x85F6DC7: x264_ratecontrol_start (in /usr/local/lib/libx264.so.152)
==16876==    by 0x8621B18: x264_encoder_encode (in /usr/local/lib/libx264.so.152)
==16876==    by 0x9172B2: X264_frame (in /usr/local/src/ffmpeg/ffmpeg_g)
==16876==    by 0x7953FD: avcodec_encode_video2 (in /usr/local/src/ffmpeg/ffmpeg_g)
==16876==    by 0x7956DA: do_encode (in /usr/local/src/ffmpeg/ffmpeg_g)
==16876==    by 0x795667: avcodec_send_frame (in /usr/local/src/ffmpeg/ffmpeg_g)

Even if there is a bug in FFmpeg, libx264 should not abort because of bad input.
Feel free to provide a sample, but please recompile with (only) --enable-gpl --enable-libx264 and report to libx264 if the issue is still reproducible.

Note: See TracTickets for help on using tickets.