Opened 7 years ago

Last modified 4 months ago

#7177 open defect

"libx264" encoding crash with 10bpc (10 bit-per-component)

Reported by: Rollinnn Owned by:
Priority: important Component: avcodec
Version: git-master Keywords: libx264 yuv420p10le crash abort
Cc: MasterQuestionable Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:I was trying to convert from HEVC 10 bit to x264 8 bit and found that with -vf "scale=1280:534:flags=lanczos:dst_format=yuv420p" -vcodec libx264 -crf 18 -preset ultrafast ffmpeg crashes.
ffmpeg is static 32 bit windows build from Zeranoe.
I guess that using dst_format in such way can be wrong, but with some files it works aithout any warnings.
File that causes crash is attached.

d:\ffmpeg.exe -i "D:\0\hevc10bit -dst_format yuv420p crash.mkv" -vf "scale=1280:534:flags=lanczos:dst_format=yuv420p" -vcodec libx264 -crf 18 -preset ultrafast -an "D:\0\output.mkv"
ffmpeg version N-90884-g19c3df0cd6 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 17.100 / 56. 17.100
  libavcodec     58. 19.100 / 58. 19.100
  libavformat    58. 13.100 / 58. 13.100
  libavdevice    58.  4.100 / 58.  4.100
  libavfilter     7. 20.100 /  7. 20.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
  libpostproc    55.  2.100 / 55.  2.100
Input #0, matroska,webm, from 'D:\0\hevc10bit -dst_format yuv420p crash.mkv':
  Metadata:
    ENCODER         : Lavf58.10.100
  Duration: 00:00:10.09, start: 0.000000, bitrate: 2473 kb/s
    Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt709/unknown/unknown), 1920x800, SAR 1:1 DAR 12:5, 23.98 fps, 23.98 tbr, 1k tbn, 23.98 tbc (default) (forced)
    Metadata:
      BPS             : 7373812
      BPS-eng         : 7373812
      DURATION-eng    : 02:09:07.240000000
      NUMBER_OF_FRAMES: 185748
      NUMBER_OF_FRAMES-eng: 185748
      NUMBER_OF_BYTES : 7140836609
      NUMBER_OF_BYTES-eng: 7140836609
      _STATISTICS_WRITING_APP: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-09-14 16:29:54
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-09-14 16:29:54
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      DURATION        : 00:00:10.093000000
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 04c56380] using SAR=801/800
[libx264 @ 04c56380] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 04c56380] profile High 10, level 3.1, 4:2:0 10-bit
[libx264 @ 04c56380] 264 - core 155 r2901 7d0ff22 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=23 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=18.0 qcomp=0.60 qpmin=0 qpmax=81 qpstep=4 ip_ratio=1.40 aq=0
Output #0, matroska, to 'D:\0\output.mkv':
  Metadata:
    encoder         : Lavf58.13.100
    Stream #0:0: Video: h264 (libx264) (H264 / 0x34363248), yuv420p10le, 1280x534 [SAR 801:800 DAR 12:5], q=-1--1, 23.98 fps, 1k tbn, 23.98 tbc (default) (forced)
    Metadata:
      BPS             : 7373812
      BPS-eng         : 7373812
      DURATION-eng    : 02:09:07.240000000
      NUMBER_OF_FRAMES: 185748
      NUMBER_OF_FRAMES-eng: 185748
      NUMBER_OF_BYTES : 7140836609
      NUMBER_OF_BYTES-eng: 7140836609
      _STATISTICS_WRITING_APP: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-09-14 16:29:54
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-09-14 16:29:54
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      DURATION        : 00:00:10.093000000
      encoder         : Lavc58.19.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
Assertion failed!

Program: d:\ffmpeg.exe
File: ../src/x264-20180118-7d0ff22/encoder/slicetype.c, Line 1988

Expression: cost >= 0

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Attachments (2)

hevc10bit -dst_format yuv420p crash.mkv​ (994.7 KB ) - added by Rollinnn 7 years ago.
x264assert.nut​ (2.0 MB ) - added by Carl Eugen Hoyos 6 years ago.

Change History (13)

by Rollinnn, 7 years ago

Attachment: hevc10bit -dst_format yuv420p crash.mkv​ added

in reply to:  description comment:1 by Carl Eugen Hoyos, 7 years ago

Keywords: libx264 added
Resolution: → invalid
Status: new → closed

Replying to Rollinnn:

Summary of the bug: I was trying to convert from HEVC 10 bit to x264 8 bit and found that with -vf "scale=1280:534:flags=lanczos:dst_format=yuv420p" -vcodec libx264 -crf 18 -preset ultrafast ffmpeg crashes.
ffmpeg is static 32 bit windows build from Zeranoe.
I guess that using dst_format in such way can be wrong, but with some files it works aithout any warnings.

Are you indicating that the following command line does not cause the crash?

$ ffmpeg -i "hevc10bit -dst_format yuv420p crash.mkv" -vf "scale=1280:534:flags=lanczos" -pix_fmt yuv420p -vcodec libx264 -f null -

Assertion failed!

Program: d:\ffmpeg.exe
File: ../src/x264-20180118-7d0ff22/encoder/slicetype.c, Line 1988

Expression: cost >= 0

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

This indicates that the bug cannot be fixed in FFmpeg.

comment:2 by Rollinnn, 7 years ago

So, this is x264 bug and i should report it to x264 developers?

comment:3 by Carl Eugen Hoyos, 7 years ago

Yes.

comment:4 by Gramner, 7 years ago

convert from HEVC 10 bit to x264 8 bit
-vf "scale=1280:534:flags=lanczos:dst_format=yuv420p"

This does not seem to do what you think it does (no idea if it's the intended behavior or not though. Actually I'm not really sure what the output from that even is). Use -pix_fmt yuv420p instead of dst_format=yuv420p and it works. Just dropping the dst_format produces 10-bit H.264.

*edit* this is not an x264 bug as far as I can see, it's FFmpeg that feeds x264 with invalid data (out-of-range pixels) which causes overflows.

Last edited 7 years ago by Gramner (previous) (diff)

comment:5 by Gramner, 7 years ago

Resolution: invalid
Status: closed → reopened

comment:6 by Carl Eugen Hoyos, 6 years ago

Keywords: crash assert added
Priority: normal → important

The issue is reproducible with current FFmpeg and current libx264:

$ ffmpeg -i x264assert.nut -vcodec libx264 -preset ultrafast -f null -
ffmpeg version N-91983-gcd732ac Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.4.0 (GCC)
  configuration: --enable-gpl --enable-gnutls --enable-libxml2 --enable-libx264
  libavutil      56. 19.101 / 56. 19.101
  libavcodec     58. 30.100 / 58. 30.100
  libavformat    58. 18.101 / 58. 18.101
  libavdevice    58.  4.103 / 58.  4.103
  libavfilter     7. 32.100 /  7. 32.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
  libpostproc    55.  2.100 / 55.  2.100
Input #0, nut, from 'x264assert.nut':
  Metadata:
    encoder         : Lavf58.18.101
  Duration: 00:00:00.00, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le, 1280x534, SAR 801:800 DAR 12:5, 23.98 tbr, 48k tbn, 48k tbc (default)
    Metadata:
      BPS             : 7373812
      BPS-eng         : 7373812
      DURATION-eng    : 02:09:07.240000000
      NUMBER_OF_FRAMES: 185748
      NUMBER_OF_FRAMES-eng: 185748
      NUMBER_OF_BYTES : 7140836609
      NUMBER_OF_BYTES-eng: 7140836609
      _STATISTICS_WRITING_APP: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-09-14 16:29:54
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-09-14 16:29:54
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      DURATION        : 00:00:06.130000000
      encoder         : Lavc58.30.100 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x27b7c00] using SAR=801/800
[libx264 @ 0x27b7c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x27b7c00] profile High 10, level 3.1, 4:2:0, 10-bit
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.18.101
    Stream #0:0: Video: h264 (libx264), yuv420p10le, 1280x534 [SAR 801:800 DAR 12:5], q=-1--1, 23.98 fps, 23.98 tbn, 23.98 tbc (default)
    Metadata:
      BPS             : 7373812
      BPS-eng         : 7373812
      DURATION-eng    : 02:09:07.240000000
      NUMBER_OF_FRAMES: 185748
      NUMBER_OF_FRAMES-eng: 185748
      NUMBER_OF_BYTES : 7140836609
      NUMBER_OF_BYTES-eng: 7140836609
      _STATISTICS_WRITING_APP: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.9.0 ('Pick Up') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-09-14 16:29:54
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-09-14 16:29:54
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      DURATION        : 00:00:06.130000000
      encoder         : Lavc58.30.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
ffmpeg: encoder/slicetype.c:1989: x264_10_rc_analyse_slice: Assertion `cost >= 0' failed.
Aborted

by Carl Eugen Hoyos, 6 years ago

Attachment: x264assert.nut​ added

comment:7 by Carl Eugen Hoyos, 5 years ago

Keywords: abort added; assert removed

comment:8 by Balling, 2 years ago

Status: reopened → open

Assertion failed: cost >= 0, file encoder/slicetype.c,

still happens.

comment:9 by Balling, 4 months ago

It still happens with ffmpeg -i x264assert.nut -vcodec libx264 -preset ultrafast -an fernfwe.mp4

So... Reporting this to x264? Would be nice to know what is the value of cost at assert.

Last edited 4 months ago by Balling (previous) (diff)

comment:10 by Balling, 4 months ago

THIS IS a bug not just in ultrafast. If you losslessly encode with libx264 using 3 presets:

ffmpeg -i x264assert.nut -vcodec libx264 -qp 0 -an somepreset.mp4

ffmpeg -i x264assert.nut -vcodec libx264 -qp 0 -preset ultrafast -an ultrafast .mp4

ffmpeg -i x264assert.nut -vcodec libx264 -qp 0 -preset placebo -an fernfweplacebo.mp4

(no, cfr 0 is not lossless for 10 bit, as High 10 profile is not lossless)
the resultant in all cases is different and different with original, though placebo is rather close.

comment:11 by MasterQuestionable, 4 months ago

Cc: MasterQuestionable added
Component: undetermined → avcodec
Keywords: yuv420p10le added
Summary: crash when using scale with dst_format on 10 bit HEVC source file → "libx264" encoding crash with 10bpc (10 bit-per-component)

͏    The description is misleading: per command output the output shall be "yuv420p10le".
͏    (so not 10bpc to 8bpc)

Note: See TracTickets for help on using tickets.