outside the range of unsigned long at libavcodec/ratecontrol.c

[Suhwan]

Summary of the bug:
There is outside the range of unsigned long at libavcodec/ratecontrol.c:967:37

libavcodec/ratecontrol.c:967:37: runtime error: -4.52978e+06 is outside the range of representable values of type 'unsigned long'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior libavcodec/ratecontrol.c:967:37 in

#0  ff_rate_estimate_qscale (s=0x626000000100, dry_run=0)
    at libavcodec/ratecontrol.c:967
#1  0x000000000437e887 in estimate_qp (s=0x626000000100, dry_run=0)
    at libavcodec/mpegvideo_enc.c:3607
#2  0x000000000436ae4e in encode_picture (s=0x626000000100, picture_number=6)
    at libavcodec/mpegvideo_enc.c:3807
#3  0x000000000433e284 in ff_mpv_encode_picture (avctx=0x619000002380, 
    pkt=0x60e000010780, pic_arg=<optimized out>, got_packet=0x7fffffffae00)
    at libavcodec/mpegvideo_enc.c:1868
#4  0x000000000313a8d1 in avcodec_encode_video2 (avctx=0x619000002380, 
    avpkt=<optimized out>, frame=0x616000050480, got_packet_ptr=<optimized out>)
    at libavcodec/encode.c:302
#5  0x000000000313da5f in do_encode (avctx=<optimized out>, frame=0x616000050480, 
    got_packet=0x7fffffffae00) at libavcodec/encode.c:371
#6  0x000000000313d13f in avcodec_send_frame (avctx=<optimized out>, 
    frame=0x616000050480) at libavcodec/encode.c:420
#7  0x000000000065b93d in do_video_out (of=0x60a0000002c0, ost=0x616000001e80, 
    next_picture=0x616000050480, sync_ipts=4.9406564584124654e-324)
    at fftools/ffmpeg.c:1287
#8  0x00000000006536ee in reap_filters (flush=0) at fftools/ffmpeg.c:1504
#9  0x00000000005e7433 in transcode_step () at fftools/ffmpeg.c:4648
#10 transcode () at fftools/ffmpeg.c:4692
#11 0x00000000005db6ec in main (argc=<optimized out>, argv=<optimized out>)
    at fftools/ffmpeg.c:4894

How to reproduce:

% ./ffmpeg_g -stream_loop 23 -y -i flower-palette-02.tif -target vcd -loglevel 0 -map 0 -c:a:48 pcm_f64le -c:v:0 h263p -ab 420k -ac 3 -strict 3 tmp.x11grab

ffmpeg version N-94982-gea673a0edb Copyright (c) 2000-2019 the FFmpeg developers
  built with clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
  configuration: --cc=clang --cxx=clang++ --ld=clang --enable-debug --toolchain=clang-usan

[Suhwan]

poc

[Elon Musk]

Fixed in 996797f8ed06e6ccb5287e18a5de88b2b0db31ec