Opened 4 years ago

Closed 4 years ago

#4301 closed defect (fixed)

-vf scale=1280:720,hqdn3d=5:4:7:6 and -pix_fmt yuv420p10le cause crash

Reported by: Selur Owned by:
Priority: important Component: avfilter
Version: git-master Keywords: hqdn3d crash SIGSEGV regression
Cc: michael Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

Summary of the bug:
when resizing and using hqdn3d ffmpeg crashes
How to reproduce:
call:

ffmpeg -y -threads 8 -r 24000/1001 -analyzeduration 100M -probesize 100M -i "C:\Users\Selur\Desktop\Untitled.split.2_new.split.1.m2ts" -map 0:0 -an -sn -vf scale=1280:720,hqdn3d=5:4:7:6 -vsync 0 -r 24000/1001 -pix_fmt yuv420p10le -sws_flags spline -f rawvideo NUL

uploaded the sample that I used: http://www.embedupload.com/?d=4TAEQREXZO

output I got was:

ffmpeg version git-2015-02-07-5647286 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.9.2 (Rev2, Built by MSYS2 project)
  configuration: --arch=x86_64 --disable-debug --disable-shared --disable-doc --disable-w32threads --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avfilter --enable-bzlib --enable-zlib --enable-decklink --enable-librtmp --enable-gnutls --enable-avisynth --enable-frei0r --enable-filter=frei0r --enable-libbluray --enable-libcaca --enable-libopenjpeg --enable-fontconfig --enable-libfreetype --enable-libass --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libschroedinger --enable-libsoxr --enable-libtwolame --enable-libspeex --enable-libtheora --enable-libutvideo --enable-libvorbis --enable-libvo-aacenc --enable-libopus --enable-libvidstab --enable-libvpx --enable-libwavpack --enable-libxavs --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzvbi
  libavutil      54. 18.100 / 54. 18.100
  libavcodec     56. 21.102 / 56. 21.102
  libavformat    56. 19.100 / 56. 19.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5.  9.103 /  5.  9.103
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
[mpegts @ 0000000000420820] DTS discontinuity in stream 13: packet 15 with DTS 378392194, packet 16 with DTS 378653022
[mpegts @ 0000000000420820] DTS discontinuity in stream 14: packet 7 with DTS 378395914, packet 8 with DTS 378660563
[mpegts @ 0000000000420820] DTS discontinuity in stream 17: packet 31 with DTS 378959007, packet 32 with DTS 379234855
[mpegts @ 0000000000420820] DTS discontinuity in stream 15: packet 39 with DTS 379221797, packet 40 with DTS 379535128
[mpegts @ 0000000000420820] DTS discontinuity in stream 8: packet 39 with DTS 379221784, packet 40 with DTS 379535141
[mpegts @ 0000000000420820] DTS discontinuity in stream 10: packet 39 with DTS 379221781, packet 40 with DTS 379535145
[mpegts @ 0000000000420820] DTS discontinuity in stream 16: packet 39 with DTS 379221757, packet 40 with DTS 379535169
[mpegts @ 0000000000420820] DTS discontinuity in stream 11: packet 39 with DTS 379221744, packet 40 with DTS 379535181
[mpegts @ 0000000000420820] DTS discontinuity in stream 13: packet 39 with DTS 379221731, packet 40 with DTS 379535195
[mpegts @ 0000000000420820] DTS discontinuity in stream 12: packet 39 with DTS 379221728, packet 40 with DTS 379535198
[mpegts @ 0000000000420820] DTS discontinuity in stream 9: packet 39 with DTS 379221726, packet 40 with DTS 379535199
[mpegts @ 0000000000420820] DTS discontinuity in stream 18: packet 39 with DTS 379221717, packet 40 with DTS 379535208
[mpegts @ 0000000000420820] DTS discontinuity in stream 14: packet 31 with DTS 379544612, packet 32 with DTS 379865466
[mpegts @ 0000000000420820] DTS discontinuity in stream 10: packet 47 with DTS 379544624, packet 48 with DTS 379880469
[mpegts @ 0000000000420820] DTS discontinuity in stream 15: packet 71 with DTS 380621842, packet 72 with DTS 380721418
[mpegts @ 0000000000420820] DTS discontinuity in stream 10: packet 71 with DTS 380621831, packet 72 with DTS 380721429
[mpegts @ 0000000000420820] DTS discontinuity in stream 16: packet 71 with DTS 380621820, packet 72 with DTS 380721440
[mpegts @ 0000000000420820] DTS discontinuity in stream 18: packet 71 with DTS 380621820, packet 72 with DTS 380721440
[mpegts @ 0000000000420820] DTS discontinuity in stream 9: packet 71 with DTS 380621813, packet 72 with DTS 380721446
Input #0, mpegts, from 'C:\Users\Selur\Desktop\Untitled.split.2_new.split.1.m2ts':
  Duration: 00:00:30.72, start: 4199.916589, bitrate: 25642 kb/s
  Program 1 
    Stream #0:0[0x1011]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc
    Stream #0:1[0x1100]: Audio: dts (DTS-HD MA) ([134][0][0][0] / 0x0086), 48000 Hz, 5.1(side), fltp, 1536 kb/s
    Stream #0:2[0x1101]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:3[0x1102]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:4[0x1103]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:5[0x1104]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:6[0x1105]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:7[0x1106]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:8[0x1200]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:9[0x1201]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:10[0x1202]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:11[0x1203]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:12[0x1204]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:13[0x1205]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:14[0x1206]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:15[0x1207]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:16[0x1208]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:17[0x1209]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
    Stream #0:18[0x120a]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
Using -vsync 0 and -r can produce invalid output files
Output #0, rawvideo, to 'NUL':
  Metadata:
    encoder         : Lavf56.19.100
    Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc
    Metadata:
      encoder         : Lavc56.21.102 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
frame=   45 fps=0.0 q=0.0 size=  121500kB time=00:00:01.87 bitrate=530311.3kbits/s    
frame=   95 fps= 94 q=0.0 size=  256500kB time=00:00:03.96 bitrate=530311.2kbits/s    
frame=  146 fps= 96 q=0.0 size=  394200kB time=00:00:06.08 bitrate=530311.3kbits/s    

then I got an appcrash:

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	ffmpeg.exe
  Application Version:	0.0.0.0
  Application Timestamp:	00100158
  Fault Module Name:	ffmpeg.exe
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	00100158
  Exception Code:	c0000005
  Exception Offset:	00000000000c6e4e
  OS Version:	6.1.7601.2.1.0.256.48
  Locale ID:	1031
  Additional Information 1:	d31d
  Additional Information 2:	d31de3b7a454222685af38a384600e19
  Additional Information 3:	d352
  Additional Information 4:	d3527b96c9f2cb9a04f87813dde5e3d1

Read our privacy statement online:
  http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409

If the online privacy statement is not available, please read our privacy statement offline:
  C:\Windows\system32\en-US\erofflps.txt

(analog call crashed for me on Mac OS X too)

Attachments (1)

untitled.split.2_new.split.1.cut.avi (1.1 MB) - added by cehoyos 4 years ago.

Download all attachments as: .zip

Change History (5)

comment:1 Changed 4 years ago by michael

input sample is no longer available, please never upload to random places use our ftp or attach it if its small enough or use a reliable place to upload samples
also no gdb backtrace with line numbers / register dump

comment:2 Changed 4 years ago by michael

  • Cc michael added

comment:3 Changed 4 years ago by cehoyos

  • Component changed from undetermined to avfilter
  • Keywords hqdn3d crash SIGSEGV regression added
  • Priority changed from normal to important
  • Reproduced by developer set
  • Status changed from new to open

Could be considered a regression since 1ad715db

(gdb) r -i out.avi -vf hqdn3d=5:4:7:6 -f null -
Starting program: ffmpeg_g -i out.avi -vf hqdn3d=5:4:7:6 -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-72360-g816005d Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      54. 25.100 / 54. 25.100
  libavcodec     56. 41.100 / 56. 41.100
  libavformat    56. 33.101 / 56. 33.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 16.101 /  5. 16.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
out.avi: No such file or directory
[Inferior 1 (process 14801) exited with code 01]
(gdb) r -i untitled.split.2_new.split.1.cut.avi -vf hqdn3d=5:4:7:6 -f null -
Starting program: /home/cehoyos/test/cehoyos/FFmpeg/ffmpeg_g -i untitled.split.2_new.split.1.cut.avi -vf hqdn3d=5:4:7:6 -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-72360-g816005d Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      54. 25.100 / 54. 25.100
  libavcodec     56. 41.100 / 56. 41.100
  libavformat    56. 33.101 / 56. 33.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 16.101 /  5. 16.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, avi, from 'untitled.split.2_new.split.1.cut.avi':
  Metadata:
    encoder         : Lavf56.19.100
  Duration: 00:00:00.08, start: 0.000000, bitrate: 112165 kb/s
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p10le, 1280x720, SAR 1:1 DAR 16:9, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc
[New Thread 0x7ffff14f0700 (LWP 14803)]
[New Thread 0x7ffff0cef700 (LWP 14804)]
[New Thread 0x7ffff04ee700 (LWP 14805)]
[New Thread 0x7fffefced700 (LWP 14806)]
[New Thread 0x7fffef4ec700 (LWP 14807)]
[New Thread 0x7fffeeceb700 (LWP 14808)]
[New Thread 0x7fffee4ea700 (LWP 14809)]
[New Thread 0x7fffedce9700 (LWP 14810)]
[New Thread 0x7fffed4e8700 (LWP 14811)]
[New Thread 0x7fffecce7700 (LWP 14812)]
[New Thread 0x7fffec4e6700 (LWP 14813)]
[New Thread 0x7fffebce5700 (LWP 14814)]
[New Thread 0x7fffeb4e4700 (LWP 14815)]
[New Thread 0x7fffeace3700 (LWP 14816)]
[New Thread 0x7fffea4e2700 (LWP 14817)]
[New Thread 0x7fffe9ce1700 (LWP 14818)]
[New Thread 0x7fffe94e0700 (LWP 14819)]
[New Thread 0x7fffe8cdf700 (LWP 14820)]
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf56.33.101
    Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc
    Metadata:
      encoder         : Lavc56.41.100 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (ffv1 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
[null @ 0x1b9f120] Encoder did not produce proper pts, making some up.

Program received signal SIGSEGV, Segmentation fault.
ff_hqdn3d_row_10_x86.loop2 () at libavfilter/x86/vf_hqdn3d.asm:105
105     HQDN3D_ROW 10
(gdb) bt
#0  ff_hqdn3d_row_10_x86.loop2 () at libavfilter/x86/vf_hqdn3d.asm:105
#1  0x000000000247b740 in ?? ()
#2  0x00000000004da245 in denoise_spatial (temporal=0x1b994a0, spatial=0x1b95460,
    depth=10, dstride=2560, sstride=2560, h=720, w=1280, frame_ant=0xfffffffd,
    line_ant=0x1ba10c0, dst=<optimized out>, src=<optimized out>, s=0x1ba0cc0)
    at libavfilter/vf_hqdn3d.c:107
#3  denoise_depth (depth=10, temporal=<optimized out>, spatial=0x1b93460, dstride=2560,
    sstride=2560, h=<optimized out>, w=<optimized out>, frame_ant_ptr=0x1ba0cf0,
    line_ant=0x1ba10c0, dst=<optimized out>, src=<optimized out>, s=0x1ba0cc0)
    at libavfilter/vf_hqdn3d.c:147
#4  filter_frame (inlink=inlink@entry=0x1b9cde0, in=0x25dc060)
    at libavfilter/vf_hqdn3d.c:312
#5  0x00000000004a1ece in ff_filter_frame_framed (link=link@entry=0x1b9cde0,
    frame=0x271f81e, frame@entry=0x25dc060) at libavfilter/avfilter.c:1091
#6  0x00000000004a2f59 in ff_filter_frame (link=link@entry=0x1b9cde0, frame=0x25dc060)
    at libavfilter/avfilter.c:1172
#7  0x00000000004a72f2 in request_frame (link=0x1b9cde0) at libavfilter/buffersrc.c:500
#8  0x00000000004a758b in av_buffersrc_add_frame_internal (ctx=ctx@entry=0x1b92740,
    frame=frame@entry=0x1ff6840, flags=flags@entry=4) at libavfilter/buffersrc.c:181
#9  0x00000000004a791d in av_buffersrc_add_frame_flags (ctx=0x1b92740,
    frame=frame@entry=0x1ff6840, flags=flags@entry=4) at libavfilter/buffersrc.c:106
#10 0x0000000000489832 in decode_video (ist=ist@entry=0x1b9e780,
    pkt=pkt@entry=0x7fffffffd670, got_output=got_output@entry=0x7fffffffd66c)
    at ffmpeg.c:2098
#11 0x000000000048b56a in process_input_packet (ist=0x1b9e780, pkt=0x0) at ffmpeg.c:2232
#12 0x000000000048efbd in process_input (file_index=0) at ffmpeg.c:3567
#13 transcode_step () at ffmpeg.c:3847
#14 transcode () at ffmpeg.c:3900
#15 0x00000000004726fb in main (argc=<optimized out>, argv=0x7fffffffdce8)
    at ffmpeg.c:4082
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0x548652 to 0x548692:
   0x0000000000548652 <ff_hqdn3d_row_10_x86.loop2+20>:  mov    %bx,(%rdx,%r8,2)
   0x0000000000548657 <ff_hqdn3d_row_10_x86.loop2+25>:  sub    %r11,%r10
   0x000000000054865a <ff_hqdn3d_row_10_x86.loop2+28>:  sar    $0x4,%r10
   0x000000000054865e <ff_hqdn3d_row_10_x86.loop2+32>:  movswl (%r9,%r10,2),%r10d
   0x0000000000548663 <ff_hqdn3d_row_10_x86.loop2+37>:  add    %r11d,%r10d
   0x0000000000548666 <ff_hqdn3d_row_10_x86.loop2+40>:  movzwl (%rcx,%r8,2),%r11d
   0x000000000054866b <ff_hqdn3d_row_10_x86.loop2+45>:  sub    %rbx,%r11
   0x000000000054866e <ff_hqdn3d_row_10_x86.loop2+48>:  sar    $0x4,%r11
=> 0x0000000000548672 <ff_hqdn3d_row_10_x86.loop2+52>:  movswl (%rax,%r11,2),%r11d
   0x0000000000548677 <ff_hqdn3d_row_10_x86.loop2+57>:  add    %ebx,%r11d
   0x000000000054867a <ff_hqdn3d_row_10_x86.loop2+60>:  mov    %r11w,(%rcx,%r8,2)
   0x000000000054867f <ff_hqdn3d_row_10_x86.loop2+65>:  shr    $0x6,%r11d
   0x0000000000548683 <ff_hqdn3d_row_10_x86.loop2+69>:  mov    %r11w,(%rsi,%r8,2)
   0x0000000000548688 <ff_hqdn3d_row_10_x86.loop2+74>:  inc    %r8
   0x000000000054868b <ff_hqdn3d_row_10_x86.loop2+77>:  jl     0x548630 <ff_hqdn3d_row_10_x86.loop>
   0x000000000054868d <ff_hqdn3d_row_10_x86.loop2+79>:  je     0x54863e <ff_hqdn3d_row_10_x86.loop2>
   0x000000000054868f <ff_hqdn3d_row_10_x86.loop2+81>:  pop    %rbx
   0x0000000000548690 <ff_hqdn3d_row_10_x86.loop2+82>:  retq
   0x0000000000548691 <ff_hqdn3d_row_10_x86.loop2+83>:  data32 data32 data32 data32 data32 nopw %cs:0x0(%rax,%rax,1)
End of assembler dump.
(gdb) info register
rax            0x1b994a0        28939424
rbx            0xfffffffd       4294967293
rcx            0x247c13e        38256958
rdx            0x1ba1abe        28973758
rsi            0x271f81e        41023518
rdi            0x7fffdc143c1e   140736885701662
rbp            0x4ff    0x4ff
rsp            0x7fffffffd240   0x7fffffffd240
r8             0xffffffffffffff61       -159
r9             0x1b95460        28922976
r10            0x6a     106
r11            0xfffffffff0000002       -268435454
r12            0x203    515
r13            0x1ba10c0        28971200
r14            0x1b95460        28922976
r15            0x1b994a0        28939424
rip            0x548672 0x548672 <ff_hqdn3d_row_10_x86.loop2+52>
eflags         0x10282  [ SF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

Not reproducible without yasm / nasm.

Changed 4 years ago by cehoyos

comment:4 Changed 4 years ago by Timothy_Gu

  • Resolution set to fixed
  • Status changed from open to closed
Note: See TracTickets for help on using tickets.