left shift of negative value at libavcodec/dvenc.c

[Suhwan]

Summary of the bug:
There're 4 left shift of negative value at libavcodec/dvenc.c.

I compiled ffmpeg with "--toolchain=clang-usan" to check the undefined-behaviours and attached log file.

How to reproduce:

% ffmpeg_g -y -i $PoC -filter_complex vflip -target dv -loglevel 0 -map 0  tmp.cavsvideo

ffmpeg version N-95385-ge1b89c76f6 Copyright (c) 2000-2019 the FFmpeg developers
built with clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
configuration: --cc=clang --cxx=clang++ --ld=clang --enable-debug --toolchain=clang-usan

Here's UBSAN log

libavcodec/dvenc.c:452:46: runtime error: left shift of negative value -768
[Switching to Thread 0x7fffd0fd1700 (LWP 6449)]

Thread 66 "ffmpeg_g" hit Breakpoint 1, 0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042a950 in handleShiftOutOfBoundsImpl(__ubsan::ShiftOutOfBoundsData*, unsigned long, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042caf1 in __ubsan_handle_shift_out_of_bounds ()
#3  0x0000000001f1c019 in dv_encode_video_segment (avctx=0x9247080, arg=0x92475d8)
    at libavcodec/dvenc.c:452
#4  0x0000000003449441 in avcodec_default_execute (c=0x9247080, 
    func=0x1f186c0 <dv_encode_video_segment>, arg=0x92475d8, ret=<optimized out>, 
    count=<optimized out>, size=12) at libavcodec/utils.c:446
#5  0x0000000001f16899 in dvvideo_encode_frame (c=0x9247080, pkt=0x7fffc8000900, frame=<optimized out>, 
    got_packet=0x7fffd0fd0ea4) at libavcodec/dvenc.c:743
#6  0x0000000001fb303f in avcodec_encode_video2 (avctx=0x9247080, avpkt=<optimized out>, 
    frame=<optimized out>, got_packet_ptr=0x7fffd0fd0ea4) at libavcodec/encode.c:302
#7  0x0000000002155181 in worker (v=<optimized out>) at libavcodec/frame_thread_encoder.c:89
#8  0x00007ffff668e6db in start_thread (arg=0x7fffd0fd1700) at pthread_create.c:463
#9  0x00007ffff5d9388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) c
Continuing.
libavcodec/dvenc.c:457:59: runtime error: left shift of negative value -9180

Thread 66 "ffmpeg_g" hit Breakpoint 1, 0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042a950 in handleShiftOutOfBoundsImpl(__ubsan::ShiftOutOfBoundsData*, unsigned long, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042caf1 in __ubsan_handle_shift_out_of_bounds ()
#3  0x0000000001f1be6b in dv_encode_video_segment (avctx=0x9247080, arg=0x92475d8)
    at libavcodec/dvenc.c:457
#4  0x0000000003449441 in avcodec_default_execute (c=0x9247080, 
    func=0x1f186c0 <dv_encode_video_segment>, arg=0x92475d8, ret=<optimized out>, 
    count=<optimized out>, size=12) at libavcodec/utils.c:446
#5  0x0000000001f16899 in dvvideo_encode_frame (c=0x9247080, pkt=0x7fffc8000900, frame=<optimized out>, 
    got_packet=0x7fffd0fd0ea4) at libavcodec/dvenc.c:743
#6  0x0000000001fb303f in avcodec_encode_video2 (avctx=0x9247080, avpkt=<optimized out>, 
    frame=<optimized out>, got_packet_ptr=0x7fffd0fd0ea4) at libavcodec/encode.c:302
#7  0x0000000002155181 in worker (v=<optimized out>) at libavcodec/frame_thread_encoder.c:89
#8  0x00007ffff668e6db in start_thread (arg=0x7fffd0fd1700) at pthread_create.c:463
#9  0x00007ffff5d9388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) c
Continuing.
libavcodec/dvenc.c:477:83: runtime error: left shift of negative value -2286

Thread 66 "ffmpeg_g" hit Breakpoint 1, 0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042a950 in handleShiftOutOfBoundsImpl(__ubsan::ShiftOutOfBoundsData*, unsigned long, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042caf1 in __ubsan_handle_shift_out_of_bounds ()
#3  0x0000000001f18a4d in dv_encode_video_segment (avctx=0x9247080, arg=0x92475d8)
    at libavcodec/dvenc.c:477
#4  0x0000000003449441 in avcodec_default_execute (c=0x9247080, 
    func=0x1f186c0 <dv_encode_video_segment>, arg=0x92475d8, ret=<optimized out>, 
    count=<optimized out>, size=12) at libavcodec/utils.c:446
#5  0x0000000001f16899 in dvvideo_encode_frame (c=0x9247080, pkt=0x7fffc8000900, frame=<optimized out>, 
    got_packet=0x7fffd0fd0ea4) at libavcodec/dvenc.c:743
#6  0x0000000001fb303f in avcodec_encode_video2 (avctx=0x9247080, avpkt=<optimized out>, 
    frame=<optimized out>, got_packet_ptr=0x7fffd0fd0ea4) at libavcodec/encode.c:302
#7  0x0000000002155181 in worker (v=<optimized out>) at libavcodec/frame_thread_encoder.c:89
#8  0x00007ffff668e6db in start_thread (arg=0x7fffd0fd1700) at pthread_create.c:463
#9  0x00007ffff5d9388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) c
Continuing.
libavcodec/dvenc.c:481:67: runtime error: left shift of negative value -384

Thread 66 "ffmpeg_g" hit Breakpoint 1, 0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042a950 in handleShiftOutOfBoundsImpl(__ubsan::ShiftOutOfBoundsData*, unsigned long, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042caf1 in __ubsan_handle_shift_out_of_bounds ()
#3  0x0000000001f30eab in dv_encode_video_segment (avctx=0x9247080, arg=0x92475d8)
    at libavcodec/dvenc.c:481
#4  0x0000000003449441 in avcodec_default_execute (c=0x9247080, 
    func=0x1f186c0 <dv_encode_video_segment>, arg=0x92475d8, ret=<optimized out>, 
    count=<optimized out>, size=12) at libavcodec/utils.c:446
#5  0x0000000001f16899 in dvvideo_encode_frame (c=0x9247080, pkt=0x7fffc8000900, frame=<optimized out>, 
    got_packet=0x7fffd0fd0ea4) at libavcodec/dvenc.c:743
#6  0x0000000001fb303f in avcodec_encode_video2 (avctx=0x9247080, avpkt=<optimized out>, 
    frame=<optimized out>, got_packet_ptr=0x7fffd0fd0ea4) at libavcodec/encode.c:302
#7  0x0000000002155181 in worker (v=<optimized out>) at libavcodec/frame_thread_encoder.c:89
#8  0x00007ffff668e6db in start_thread (arg=0x7fffd0fd1700) at pthread_create.c:463
#9  0x00007ffff5d9388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Please confirm.
Thanks

[Suhwan]

poc

[mkver]

Fixed in 6770057ac97e78c799f06eb3769fecdb0833d44f.