Opened 3 years ago

Closed 3 years ago

#8278 closed defect (fixed)

outside the range of representable values of type 'int' and division by zero at libavfilter/vf_bm3d.c:663

Reported by: Suhwan Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords: ubsan
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
There is an outside the range of representable values of type 'int' and a division by zero at libavfilter/vf_bm3d.c:663

I compiled ffmpeg with "--toolchain=clang-usan" to check the undefined-behaviours and attached log file.
How to reproduce:

% ffmpeg_g -y -i $PoC -filter_complex bm3d -target dv -map 0 tmp.smjpeg

ffmpeg version N-95385-ge1b89c76f6 Copyright (c) 2000-2019 the FFmpeg developers
built with clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
configuration: --cc=clang --cxx=clang++ --ld=clang --enable-debug --toolchain=clang-usan

Here's UBSAN log

libavfilter/vf_bm3d.c:663:45: runtime error: division by zero

Thread 1 "ffmpeg_g" hit Breakpoint 1, 0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x0000000000429afe in handleDivremOverflowImpl(__ubsan::OverflowData*, unsigned long, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042ca81 in __ubsan_handle_divrem_overflow ()
#3  0x000000000075afc4 in do_output (s=0x915ff80, dst=0x93067c0 "", dst_linesize=32, plane=<optimized out>, nb_jobs=0) at libavfilter/vf_bm3d.c:663
#4  0x000000000074a51e in filter_frame (ctx=<optimized out>, out=0x7fffffffc138, in=0x9305340, ref=0x9305340) at libavfilter/vf_bm3d.c:768
#5  0x00000000007474c2 in activate (ctx=0x915e080) at libavfilter/vf_bm3d.c:861
#6  0x00000000005ce29c in ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1442
#7  0x00000000005f426f in push_frame (graph=0x9158b40) at libavfilter/buffersrc.c:187
#8  av_buffersrc_add_frame_internal (ctx=<optimized out>, frame=0x9158140, flags=<optimized out>) at libavfilter/buffersrc.c:261
#9  0x00000000005f2e2e in av_buffersrc_add_frame_flags (ctx=0x910a900, frame=0x9158140, flags=4) at libavfilter/buffersrc.c:170
#10 0x00000000004cad10 in ifilter_send_frame (ifilter=<optimized out>, frame=<optimized out>) at fftools/ffmpeg.c:2186
#11 send_frame_to_filters (ist=0x910f380, decoded_frame=0x9158140) at fftools/ffmpeg.c:2260
#12 0x00000000004a076c in decode_video (ist=<optimized out>, pkt=<optimized out>, got_output=<optimized out>, duration_pts=<optimized out>, 
    eof=<optimized out>, decode_failed=<optimized out>) at fftools/ffmpeg.c:2459
#13 process_input_packet (ist=<optimized out>, pkt=<optimized out>, no_eof=<optimized out>) at fftools/ffmpeg.c:2613
#14 0x00000000004bf0a0 in process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4508
#15 0x000000000048d59b in transcode_step () at fftools/ffmpeg.c:4628
#16 transcode () at fftools/ffmpeg.c:4682
#17 0x0000000000487d54 in main (argc=37, argv=<optimized out>) at fftools/ffmpeg.c:4884
(gdb) c
Continuing.
libavfilter/vf_bm3d.c:663:37: runtime error: -nan is outside the range of representable values of type 'int'

Thread 1 "ffmpeg_g" hit Breakpoint 1, 0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428860 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042a496 in handleFloatCastOverflow(void*, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042ccc1 in __ubsan_handle_float_cast_overflow ()
#3  0x000000000075b01f in do_output (s=0x915ff80, dst=0x93067c0 "", dst_linesize=32, plane=<optimized out>, nb_jobs=0) at libavfilter/vf_bm3d.c:663
#4  0x000000000074a51e in filter_frame (ctx=<optimized out>, out=0x7fffffffc138, in=0x9305340, ref=0x9305340) at libavfilter/vf_bm3d.c:768
#5  0x00000000007474c2 in activate (ctx=0x915e080) at libavfilter/vf_bm3d.c:861
#6  0x00000000005ce29c in ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1442
#7  0x00000000005f426f in push_frame (graph=0x9158b40) at libavfilter/buffersrc.c:187
#8  av_buffersrc_add_frame_internal (ctx=<optimized out>, frame=0x9158140, flags=<optimized out>) at libavfilter/buffersrc.c:261
#9  0x00000000005f2e2e in av_buffersrc_add_frame_flags (ctx=0x910a900, frame=0x9158140, flags=4) at libavfilter/buffersrc.c:170
#10 0x00000000004cad10 in ifilter_send_frame (ifilter=<optimized out>, frame=<optimized out>) at fftools/ffmpeg.c:2186
#11 send_frame_to_filters (ist=0x910f380, decoded_frame=0x9158140) at fftools/ffmpeg.c:2260
#12 0x00000000004a076c in decode_video (ist=<optimized out>, pkt=<optimized out>, got_output=<optimized out>, duration_pts=<optimized out>, 
    eof=<optimized out>, decode_failed=<optimized out>) at fftools/ffmpeg.c:2459
#13 process_input_packet (ist=<optimized out>, pkt=<optimized out>, no_eof=<optimized out>) at fftools/ffmpeg.c:2613
#14 0x00000000004bf0a0 in process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4508
#15 0x000000000048d59b in transcode_step () at fftools/ffmpeg.c:4628
#16 transcode () at fftools/ffmpeg.c:4682
#17 0x0000000000487d54 in main (argc=37, argv=<optimized out>) at fftools/ffmpeg.c:4884

Please confirm.
Thanks

Attachments (1)

PoC_vf_bm3d_663.bmp3 (58 bytes ) - added by Suhwan 3 years ago.
poc

Download all attachments as: .zip

Change History (2)

by Suhwan, 3 years ago

Attachment: PoC_vf_bm3d_663.bmp3 added

poc

comment:1 by Elon Musk, 3 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.