Integer overflow in bitrate / libx264 encoding

[Jon Frydensbjerg]

Summary of the bug:

There seems to be a 32 bit integer overflow / conversion issue in how the video bitrate parameter is being handled by libx264 (I have tested a few other encoders and they seem to be fine).

Notice how a bitrate of 3 gb/s gets interpreted as "-1294967 kb/s" in the example below. FFmpeg also displays a warning "The bitrate parameter is set too low" which is nice and how I initially detected the problem.

Bonus info: the same example works fine in version 4.1.

How to reproduce:

% ffmpeg -f lavfi -i life -c:v libx264 -vb 3G test.mov
ffmpeg version 4.1.4-0york3~18.04 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
  configuration: --prefix=/usr --extra-version='0york3~18.04' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-nonfree --enable-libfdk-aac --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, lavfi, from 'life':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (B0W1 / 0x31573042), monob, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[swscaler @ 0x556dfb947740] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0x556dfb936e00] using SAR=1/1
[libx264 @ 0x556dfb936e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x556dfb936e00] profile High 4:4:4 Predictive, level 6.2, 4:4:4 8-bit
[libx264 @ 0x556dfb936e00] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=7 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=2000000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
The bitrate parameter is set too low. It takes bits/s as argument, not kbits/s
Output #0, mov, to 'test.mov':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuvj444p(pc), 320x240 [SAR 1:1 DAR 4:3], q=-1--1, -1294967 kb/s, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.35.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/-1294967296 buffer size: 0 vbv_delay: -1
frame=  841 fps=575 q=-1.0 Lsize=   16868kB time=00:00:33.52 bitrate=4122.5kbits/s speed=22.9x    

[Carl Eugen Hoyos]

Patch sent.

[Carl Eugen Hoyos]

Replying to jonfryd:

    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuvj444p(pc), 320x240 [SAR 1:1 DAR 4:3], q=-1--1, -1294967 kb/s, 25 fps, 12800 tbn, 25 tbc

Fixed in 4b1687f23c133599f38e5f47433619b3db2e368d

      cpb: bitrate max/min/avg: 0/0/-1294967296 buffer size: 0 vbv_delay: -1

This will be fixed after the next libavcodec version bump, see 2e8b0446c6798947dac77fee4a06f9c4e8131ab5, 098ed8a73e68e3aa8d6c75460daed1e38bf66e45 and 690cab32326eb76cf923a4ca5eea5cc8be3a8600