memory corruption/core dump using alpha overlay in current git ffmpeg

[MarkZV]

Although the same command was working with an earlier ffmpeg (864fdfa0627e21ee0b69e957c3413114185623a7), after updating ffmpeg to the latest git head (1fabd950355849fe8df77226e5f048cd6bdcfb6a) memory corruption and a core dump are encountered using some combinations of video filters.

This is on Mac OS X 10.6.8:

$ ffmpeg -i lik.mp4 -r 15 -loop 1 -i lik.jpeg -filter_complex '[1] format=rgba,fade=out:15:15:alpha=1 [C]; [0] setsar=1,yadif,mp=eq2=1.1 [P]; [P][C] overlay [V]' -map '[V]' -y out.mp4
ffmpeg version 1.1.git-1fabd95 Copyright (c) 2000-2013 the FFmpeg developers
  built on Apr  6 2013 18:53:57 with gcc 4.2.1 (GCC) (Apple Inc. build 5666) (dot 3)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libass --enable-libvpx --enable-libspeex --enable-libfreetype --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/gcc-4.2 --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-nonfree --enable-libfdk-aac --enable-libfaac
  libavutil      52. 25.100 / 52. 25.100
  libavcodec     55.  2.100 / 55.  2.100
  libavformat    55.  1.100 / 55.  1.100
  libavdevice    55.  0.100 / 55.  0.100
  libavfilter     3. 49.100 /  3. 49.100
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'lik.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.1.100
  Duration: 00:00:06.07, start: 0.000000, bitrate: 59 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 180x180 [SAR 1:1 DAR 1:1], 57 kb/s, 15 fps, 15 tbr, 50k tbn, 30 tbc
    Metadata:
      handler_name    : VideoHandler
[image2 @ 0x102847600] max_analyze_duration 5000000 reached at 5000000 microseconds
Input #1, image2, from 'lik.jpeg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: mjpeg, yuvj420p, 180x180 [SAR 1:1 DAR 1:1], 25 fps, 25 tbr, 25 tbn, 25 tbc
[Parsed_mp_4 @ 0x102023c60] 'eq2' is a wrapped MPlayer filter (libmpcodecs). This filter may be removed
once it has been ported to a native libavfilter.
[libx264 @ 0x1028bbc00] using SAR=1/1
[libx264 @ 0x1028bbc00] using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
[libx264 @ 0x1028bbc00] profile High, level 1.1
[libx264 @ 0x1028bbc00] 264 - core 129 - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=15 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.1.100
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 180x180 [SAR 1:1 DAR 1:1], q=-1--1, 50k tbn, 15 tbc
Stream mapping:
  Stream #0:0 (h264) -> setsar
  Stream #1:0 (mjpeg) -> format
  overlay -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
ffmpeg(43912,0x7fff705a3cc0) malloc: *** error for object 0x105810e08: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Abort trap (core dumped)
$ 

[Carl Eugen Hoyos]

Regression since b0012de.

[Cigaes]

I believe the bug is in vf_mp: the filter_frame wraps the refcounted data planes from the incoming frame into a mp_image_t, then ff_vf_next_put_image takes the data planes from the mp_image_t and wraps them into a new (refcounted) frame.

With eq2, the planes 1 and 2 are passed unchanged, that means the data planes will end up wrapped into two distinct refcounted buffers, which is not good.

Note that commit b0012de only changes the order various parts are called: things working before that would only be a fragile coincidence.

[takis]

I could reproduce the crash with the given sample, commandline, and GIT revision (1fabd950355849fe8df77226e5f048cd6bdcfb6a). But it's working for me with current GIT head (8aea2f05dc56f7e7d60767dd27ba8e846a05e8ae).

[Carl Eugen Hoyos]

Fixed by Michael in 9b672d4.