outside the range of representable values of type 'unsigned long' at libavcodec/ratecontrol.c:966

[Suhwan]

Summary of the bug:
There is an outside the range of representable values of type 'unsigned long' at libavcodec/ratecontrol.c:966

I compiled ffmpeg with "--toolchain=clang-usan" to check the undefined-behaviours and attached log file.
How to reproduce:

% ffmpeg_g -y -i $PoC -filter_complex framerate -target vcd -loglevel 0 -c wmv1 tmp.wsd

ffmpeg version N-95517-g0b8956b25c Copyright (c) 2000-2019 the FFmpeg developers
built with clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
configuration: --cc=clang --cxx=clang++ --ld=clang --enable-debug --toolchain=clang-usan

Here's UBSAN log

libavcodec/ratecontrol.c:966:37: runtime error: -443679 is outside the range of representable values of type 'unsigned long'

Thread 1 "ffmpeg_g" hit Breakpoint 1, 0x00000000004288b0 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x00000000004288b0 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042a4e6 in handleFloatCastOverflow(void*, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042cd11 in __ubsan_handle_float_cast_overflow ()
#3  0x0000000002fa96f3 in ff_rate_estimate_qscale (s=0x93d8940, dry_run=0) at libavcodec/ratecontrol.c:966
#4  0x0000000002be940f in estimate_qp (s=0x93d8940, dry_run=0) at libavcodec/mpegvideo_enc.c:3607
#5  0x0000000002bde6a1 in encode_picture (s=0x93d8940, picture_number=30) at libavcodec/mpegvideo_enc.c:3807
#6  0x0000000002bc347a in ff_mpv_encode_picture (avctx=0x93ceb40, pkt=0x9410d40, pic_arg=<optimized out>, 
    got_packet=0x7fffffffc2a4) at libavcodec/mpegvideo_enc.c:1868
#7  0x0000000001fe242f in avcodec_encode_video2 (avctx=0x93ceb40, avpkt=<optimized out>, frame=<optimized out>, 
    got_packet_ptr=0x7fffffffc2a4) at libavcodec/encode.c:302
#8  0x0000000001fe4160 in do_encode (avctx=0x93ceb40, frame=0x9449f80, got_packet=0x7fffffffc2a4) at libavcodec/encode.c:371
#9  0x0000000001fe3cda in avcodec_send_frame (avctx=0x93ceb40, frame=0x9449f80) at libavcodec/encode.c:420
#10 0x00000000004c51f8 in do_video_out (of=0x93cce80, ost=<optimized out>, next_picture=<optimized out>, 
    sync_ipts=28.500007629394535) at fftools/ffmpeg.c:1287
#11 0x00000000004c0f2b in reap_filters (flush=0) at fftools/ffmpeg.c:1504
#12 0x000000000048c3d5 in transcode_from_filter (graph=0x93c7700, best_ist=<optimized out>) at fftools/ffmpeg.c:4533
#13 transcode_step () at fftools/ffmpeg.c:4606
#14 transcode () at fftools/ffmpeg.c:4682
#15 0x0000000000487dc4 in main (argc=13, argv=<optimized out>) at fftools/ffmpeg.c:4884

Please confirm.
Thanks

[Suhwan]

poc

[Michael Niedermayer]

Does not reproduce with git master for me