Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#3277 closed defect (fixed)

Crash in libavcodec with vlc

Reported by: SevereOverfl0w Owned by:
Priority: important Component: undetermined
Version: unspecified Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

Summary of the bug: A segfault is caused at 9 seconds into this sample. But only when using vlc, not when using ffmplay.
How to reproduce:

% vlc sample.avi
ffmpeg version 2.1.1
VLC version 2.1.2 Rincewind (2.1.2-0-ga4c4876)

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

GDB'd stack trace http://bpaste.net/show/ouFRojkk41iqIBSTqDgS/

Attachments (1)

out.avi (1.1 MB) - added by SevereOverfl0w 5 years ago.
Sample file.

Download all attachments as: .zip

Change History (18)

Changed 5 years ago by SevereOverfl0w

Sample file.

comment:1 Changed 5 years ago by michael

cant reproduce with 2.2.0-git Weatherwax (revision 2.1.0-git-1286-g3dd6aff) and ffmpeg git
nor a really old vlc i had laying around

comment:2 Changed 5 years ago by SevereOverfl0w

This issue has been discovered on Arch Linux, I perhaps should have specified, just in case it makes a difference.

comment:3 Changed 5 years ago by cehoyos

Crashes here sometimes with vlc 2.1.2 and libavcodec 2.1.1

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd08c3700 (LWP 13936)]
ff_emu_edge_vfix3_mmx.body_loop () at libavcodec/x86/videodsp.asm:333
333     libavcodec/x86/videodsp.asm: No such file or directory.
(gdb) bt
#0  ff_emu_edge_vfix3_mmx.body_loop () at libavcodec/x86/videodsp.asm:333
#1  0x00007fffd2b8e66c in emulated_edge_mc (h_extend_var=<optimized out>, hfix_tbl=
    0x7fffd2fa4380 <hfixtbl_sse2>, v_extend_var=<optimized out>, vfix_tbl=
    0x7fffd2fa42c0 <vfixtbl_sse>, h=128, w=3, src_y=<optimized out>, src_x=<optimized out>,
    block_h=9, block_w=<optimized out>, src_stride=<optimized out>, src=<optimized out>,
    dst_stride=304, dst=<optimized out>) at libavcodec/x86/videodsp_init.c:175
#2  emulated_edge_mc_sse2 (buf=0x7fffc4071d70 "~~~~~~~~~", buf_stride=304,
    src=<optimized out>, src_stride=<optimized out>, block_w=<optimized out>, block_h=9,
    src_x=301, src_y=120, w=304, h=128) at libavcodec/x86/videodsp_init.c:232
#3  0x00007fffd2955663 in mpeg_motion_internal (mb_y=15, is_mpeg12=0, h=16, motion_y=2,
    motion_x=21, pix_op=0x7fffc8d1ad68, ref_picture=<optimized out>, field_select=0,
    bottom_field=0, field_based=0, dest_cr=
    0x7fffc4083ae0 "\177\177\177\177\177\177\177\177", dest_cb=
    0x7fffc40814e0 "\215\215\215\215\215\215\215\215", dest_y=
    0x7fffc407eee0 "========<<<<<<<<", s=0x7fffc8d186e0) at libavcodec/mpegvideo_motion.c:333
#4  mpeg_motion (s=0x7fffc8d186e0, dest_y=0x7fffc407eee0 "========<<<<<<<<", dest_cb=
    0x7fffc40814e0 "\215\215\215\215\215\215\215\215", dest_cr=
    0x7fffc4083ae0 "\177\177\177\177\177\177\177\177", field_select=0,
    ref_picture=<optimized out>, pix_op=0x7fffc8d1ad68, motion_x=21, motion_y=2, h=16, mb_y=
    15) at libavcodec/mpegvideo_motion.c:384
#5  0x00007fffd2956147 in MPV_motion_internal (is_mpeg12=<optimized out>,
    qpix_op=<optimized out>, pix_op=<optimized out>, ref_picture=<optimized out>,
    dir=<optimized out>, dest_cr=<optimized out>, dest_cb=<optimized out>,
    dest_y=<optimized out>, s=<optimized out>) at libavcodec/mpegvideo_motion.c:958
#6  ff_MPV_motion (s=s@entry=0x7fffc8d186e0, dest_y=dest_y@entry=
    0x7fffc407eee0 "========<<<<<<<<", dest_cb=dest_cb@entry=
    0x7fffc40814e0 "\215\215\215\215\215\215\215\215", dest_cr=dest_cr@entry=
    0x7fffc4083ae0 "\177\177\177\177\177\177\177\177", dir=dir@entry=1,
    ref_picture=ref_picture@entry=0x7fffc8d190f8, pix_op=0x7fffc8d1ad68, qpix_op=
    0x7fffc8d1a8f8) at libavcodec/mpegvideo_motion.c:992
#7  0x00007fffd293e0f3 in MPV_decode_mb_internal (is_mpeg12=0, lowres_flag=0,
    block=<optimized out>, s=0x7fffc8d186e0) at libavcodec/mpegvideo.c:2796
#8  ff_MPV_decode_mb (s=s@entry=0x7fffc8d186e0, block=<optimized out>)
    at libavcodec/mpegvideo.c:2928
#9  0x00007fffd26cf037 in decode_slice (s=s@entry=0x7fffc8d186e0) at libavcodec/h263dec.c:243
#10 0x00007fffd26cfda3 in ff_h263_decode_frame (avctx=0x7fffc8d18040, data=0x7fffc8d17980,
    got_frame=0x7fffd08c2cdc, avpkt=<optimized out>) at libavcodec/h263dec.c:701
#11 0x00007fffd2a42022 in avcodec_decode_video2 (avctx=0x7fffc8d18040, picture=
    0x7fffc8d17980, got_picture_ptr=0x7fffd08c2cdc, avpkt=0x7fffd08c2ce0)
    at libavcodec/utils.c:2062
#12 0x00007fffd35b8219 in ?? () from /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so
#13 0x00007ffff714a620 in ?? () from /usr/lib64/libvlccore.so.7
#14 0x00007ffff714ba40 in ?? () from /usr/lib64/libvlccore.so.7
#15 0x00007ffff79aae0e in start_thread () from /lib64/libpthread.so.0
#16 0x00007ffff74de2cd in clone () from /lib64/libc.so.6
(gdb) disass $pc-23,$pc+22
Dump of assembler code from 0x7fffd2b8dc60 to 0x7fffd2b8dc8d:
   0x00007fffd2b8dc60 <..@1408.branch_instr+0>: movd   (%rdx),%mm0
   0x00007fffd2b8dc63 <ff_emu_edge_vfix3_mmx.top_loop+0>:       movd   %mm0,%eax
   0x00007fffd2b8dc66 <ff_emu_edge_vfix3_mmx.top_loop+3>:       mov    %ax,(%rdi)
   0x00007fffd2b8dc69 <ff_emu_edge_vfix3_mmx.top_loop+6>:       shr    $0x10,%eax
   0x00007fffd2b8dc6c <ff_emu_edge_vfix3_mmx.top_loop+9>:       mov    %al,0x2(%rdi)
   0x00007fffd2b8dc6f <ff_emu_edge_vfix3_mmx.top_loop+12>:      add    %rsi,%rdi
   0x00007fffd2b8dc72 <ff_emu_edge_vfix3_mmx.top_loop+15>:      dec    %r8
   0x00007fffd2b8dc75 <ff_emu_edge_vfix3_mmx.top_loop+18>:      jne    0x7fffd2b8dc63 <ff_emu_edge_vfix3_mmx.top_loop>
=> 0x00007fffd2b8dc77 <ff_emu_edge_vfix3_mmx.body_loop+0>:      mov    (%rdx),%eax
   0x00007fffd2b8dc79 <ff_emu_edge_vfix3_mmx.body_loop+2>:      mov    %ax,(%rdi)
   0x00007fffd2b8dc7c <ff_emu_edge_vfix3_mmx.body_loop+5>:      shr    $0x10,%eax
   0x00007fffd2b8dc7f <ff_emu_edge_vfix3_mmx.body_loop+8>:      mov    %al,0x2(%rdi)
   0x00007fffd2b8dc82 <ff_emu_edge_vfix3_mmx.body_loop+11>:     add    %rsi,%rdi
   0x00007fffd2b8dc85 <ff_emu_edge_vfix3_mmx.body_loop+14>:     add    %rcx,%rdx
   0x00007fffd2b8dc88 <ff_emu_edge_vfix3_mmx.body_loop+17>:     dec    %r9
   0x00007fffd2b8dc8b <ff_emu_edge_vfix3_mmx.body_loop+20>:     jne    0x7fffd2b8dc77 <ff_emu_edge_vfix3_mmx.body_loop>
End of assembler dump.
(gdb) info register
rax            0x7f7e   32638
rbx            0x0      0
rcx            0x130    304
rdx            0x7fffb76f2ffd   140736270905341
rsi            0x130    304
rdi            0x7fffc40725c0   140736482190784
rbp            0x9      0x9
rsp            0x7fffd08c2758   0x7fffd08c2758
r8             0x0      0
r9             0x1      1
r10            0x1      1
r11            0x7fffb76f27ad   140736270903213
r12            0x9      9
r13            0x3      3
r14            0x130    304
r15            0x7fffc4071d70   140736482188656
rip            0x7fffd2b8dc77   0x7fffd2b8dc77 <ff_emu_edge_vfix3_mmx.body_loop>
eflags         0x10202  [ IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

comment:4 Changed 5 years ago by michael

some valgrind --track-origins=yes output might (or might not) be interresting

comment:5 Changed 5 years ago by michael

also is this mmx/sse specific or it happens also without asm?

comment:6 Changed 5 years ago by cehoyos

$ valgrind --track-origins=yes vlc out.avi --noaudio

==4398== Conditional jump or move depends on uninitialised value(s)
==4398==    at 0x23CE8309: mpeg4_decode_mb (mpeg4videodec.c:125)
==4398==    by 0x23AA0D53: decode_slice (h263dec.c:235)
==4398==    by 0x23AA1DA2: ff_h263_decode_frame (h263dec.c:701)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==  Uninitialised value was created by a heap allocation
==4398==    at 0x4C290FE: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x4C291A7: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x24971269: av_malloc (in /usr/lib64/libavutil.so.52.48.101)
==4398==    by 0x23E10F7B: av_fast_malloc (utils.c:146)
==4398==    by 0x23AA22DD: ff_h263_decode_frame (h263dec.c:758)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==
[0x1e88d9c8] main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 694 ms)
[0x1e88d9c8] main input error: ES_OUT_RESET_PCR called
[0x6818848] main vout display error: Failed to resize display
[mpeg4 @ 0x6657da0] warning: first frame is no keyframe
==4398== Conditional jump or move depends on uninitialised value(s)
==4398==    at 0x23CE8C02: mpeg4_decode_mb (mpeg4videodec.c:140)
==4398==    by 0x23AA0D53: decode_slice (h263dec.c:235)
==4398==    by 0x23AA1DA2: ff_h263_decode_frame (h263dec.c:701)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==  Uninitialised value was created by a heap allocation
==4398==    at 0x4C290FE: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x4C291A7: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x24971269: av_malloc (in /usr/lib64/libavutil.so.52.48.101)
==4398==    by 0x23E10F7B: av_fast_malloc (utils.c:146)
==4398==    by 0x23AA22DD: ff_h263_decode_frame (h263dec.c:758)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==
==4398== Use of uninitialised value of size 8
==4398==    at 0x23C61B29: ff_h263_decode_motion (get_bits.h:558)
==4398==    by 0x23CE9A26: mpeg4_decode_mb (mpeg4videodec.c:1401)
==4398==    by 0x23AA0D53: decode_slice (h263dec.c:235)
==4398==    by 0x23AA1DA2: ff_h263_decode_frame (h263dec.c:701)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==  Uninitialised value was created by a heap allocation
==4398==    at 0x4C290FE: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x4C291A7: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x24971269: av_malloc (in /usr/lib64/libavutil.so.52.48.101)
==4398==    by 0x23E10F7B: av_fast_malloc (utils.c:146)
==4398==    by 0x23AA22DD: ff_h263_decode_frame (h263dec.c:758)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==
==4398== Use of uninitialised value of size 8
==4398==    at 0x23C61B29: ff_h263_decode_motion (get_bits.h:558)
==4398==    by 0x23CE9A3F: mpeg4_decode_mb (mpeg4videodec.c:1402)
==4398==    by 0x23AA0D53: decode_slice (h263dec.c:235)
==4398==    by 0x23AA1DA2: ff_h263_decode_frame (h263dec.c:701)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==  Uninitialised value was created by a heap allocation
==4398==    at 0x4C290FE: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x4C291A7: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4398==    by 0x24971269: av_malloc (in /usr/lib64/libavutil.so.52.48.101)
==4398==    by 0x23E10F7B: av_fast_malloc (utils.c:146)
==4398==    by 0x23AA22DD: ff_h263_decode_frame (h263dec.c:758)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==
==4398== Invalid read of size 4
==4398==    at 0x23F5FC8B: ??? (videodsp.asm:333)
==4398==    by 0x23F6066B: emulated_edge_mc_sse2 (videodsp_init.c:175)
==4398==    by 0x23D27662: mpeg_motion (mpegvideo_motion.c:333)
==4398==    by 0x23D28146: ff_MPV_motion (mpegvideo_motion.c:958)
==4398==    by 0x23D100F2: ff_MPV_decode_mb (mpegvideo.c:2796)
==4398==    by 0x23AA1036: decode_slice (h263dec.c:243)
==4398==    by 0x23AA1DA2: ff_h263_decode_frame (h263dec.c:701)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==  Address 0x29e1fffd is not stack'd, malloc'd or (recently) free'd
==4398==
==4398==
==4398== Process terminating with default action of signal 11 (SIGSEGV)
==4398==  Access not within mapped region at address 0x29E20000
==4398==    at 0x23F5FC8B: ??? (videodsp.asm:333)
==4398==    by 0x23F6066B: emulated_edge_mc_sse2 (videodsp_init.c:175)
==4398==    by 0x23D27662: mpeg_motion (mpegvideo_motion.c:333)
==4398==    by 0x23D28146: ff_MPV_motion (mpegvideo_motion.c:958)
==4398==    by 0x23D100F2: ff_MPV_decode_mb (mpegvideo.c:2796)
==4398==    by 0x23AA1036: decode_slice (h263dec.c:243)
==4398==    by 0x23AA1DA2: ff_h263_decode_frame (h263dec.c:701)
==4398==    by 0x23E14021: avcodec_decode_video2 (utils.c:2062)
==4398==    by 0x2368C218: ??? (in /usr/lib64/vlc/plugins/codec/libavcodec_plugin.so)
==4398==    by 0x584C61F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x584DA3F: ??? (in /usr/lib64/libvlccore.so.7.0.0)
==4398==    by 0x5054E0D: start_thread (in /lib64/libpthread-2.15.so)
==4398==  If you believe this happened as a result of a stack
==4398==  overflow in your program's main thread (unlikely but
==4398==  possible), you can try to increase the size of the
==4398==  main thread stack using the --main-stacksize= flag.
==4398==  The main thread stack size used in this run was 8388608.

comment:7 Changed 5 years ago by michael

  • Priority changed from normal to important

crashes are important

comment:8 Changed 5 years ago by reimar

The code is completely different now, and the uninitialized data should have been fixed in 21b25537fb8f77b098575e90d8b24556451badf3.
However this uninitialized memory should not have allowed a crash to happen, so the deeper issue might still be hiding.
Is this still possible to reproduce somehow?

comment:9 Changed 5 years ago by cehoyos

Does not crash anymore on OpenSuse? with vlc 2.1.2 and libavcodec 2.1.3
Does it still crash on Arch Linux?

comment:10 Changed 5 years ago by cehoyos

  • Summary changed from Segfault with sample. to Crash in libavcodec with vlc

comment:11 Changed 5 years ago by SevereOverfl0w

Still seeing this segfault on Arch Linux with VLC 2.1.4 revision 2.1.4-0-g2a072be

comment:12 Changed 5 years ago by cehoyos

Please install debug packages and provide a current backtrace.

comment:13 Changed 5 years ago by michael

Cannot reproduce any anomaly under valgrind or crash with vlc HEAD + ffmpeg HEAD

comment:14 Changed 5 years ago by michael

Cannot reproduce any anomaly under valgrind or crash without valgrind with vlc 2.1.4-0-g2a072be and ffmpeg HEAD nor ffmpeg 2.1.3
That is on linux x86-64

comment:15 Changed 5 years ago by michael

  • Reproduced by developer set
  • Resolution set to fixed
  • Status changed from new to closed

Succeeded reproducing some of the anomalies in valgrind with ffmpeg release/1.2 and vlc 2.1.4
my earlier attempt failed to reproduce them as videolans build system linked to a different libavcodec.so than what i had tried to specify.
Fixed in release/1.2 and release/2.1 (will fix other affected ones as well), will be in the next release from these branches
Dont hesitate to test though, there is a small chance that a problem remains, as i could not reproduce all the anomalies posted in this thread. FFmpeg head should not have been affected

comment:16 follow-up: Changed 5 years ago by er.anshul.maheshwari@gmail.com

michael,
It would be good if you elaborate "my earlier attempt failed to reproduce them as videolans build system linked to a different libavcodec.so than what i had tried to specify.
are there some more option given at the time of build of vlc.

I have pending ticket #2716, since I was unable to reproduce crash using vlc.

comment:17 in reply to: ↑ 16 Changed 5 years ago by michael

Replying to er.anshul.maheshwari@…:

michael,
It would be good if you elaborate "my earlier attempt failed to reproduce them as videolans build system linked to a different libavcodec.so than what i had tried to specify.
are there some more option given at the time of build of vlc.

I used AVCODEC_CFLAGS to force videolan to link to my libavcodec.so which worked with the first vlc version i tested but failed with a older one, which seems to need AVCODEC_LIBS to be set

Note: See TracTickets for help on using tickets.