VAAPI mpeg2 decoding crashes

[Timo R.]

Starting somewhere between ffmpeg-1.1.4 and ffmpeg-1.2 something happend with the vaapi code in ffmpeg that's causing crashes when decoding mpeg2.

I'm not 100% sure if only mpeg2 is affected, and i'm also not sure what might be causing this. It Looks like vaapi is not properly initialized, as the pic param buffer is missing.

A possible backtrace looks like this: ​http://bpaste.net/show/rFq5TRFGCp4sh1eYvjnS/
This one is from a vlc git master build. The same happens in XBMC. Both using latest master ffmpeg/libavcodec.

[Carl Eugen Hoyos]

Does only mpeg2 crash or also h264 (and vc1)?
How do you know 1.2 is affected if you tested git master?

Please post the backtrace on this tracker, do not use external resources and please find the version introducing the bug using git bisect (I will help you with git bisect if necessary).

[Timo R.]

VLC backtrace

[Timo R.]

I first noticed vaapi was crashing with ffmpeg-1.2. After applying b066d90211 to it, it stopped crashing all the time when using vaapi, it only crashed when trying to decode mpeg2.
I then tested if git master still has this issue and that was the case.

I'll try to bisect it, but it might take a while as there are a lot of commits involed and i'll have to apply b066d90211 to avoid that crash. And each step usualy also needs a rebuild of VLC.

[Timo R.]

Ok, i'm done with bisecting and ended up with the following:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
2c541554076cc8a72e7145d4da30389ca763f32f
b7fe35c9e50e1701274364adf7280bf4a02b092b
We cannot bisect more!

2c541554076c was the last commit to test, and i couldn't compile it for some reason, as configure suddenly started not knowing most of the options i usualy pass to it(Using gentoo git ebuild for easy building+testing). After I finaly managed to configure it, it failed to compile with

make: *** No rule to make target `tools/ffescape'.  Stop.

I then manualy tested the b7fe35c9e50e17 commit, which configures and builds without problems, but also has the crash on mpeg2 decode.
On each step, i manualy applyed the fix from b066d90211 if neccessary.

H264 decodes fine, mpeg2 always crashes. Didn't test any other codecs. Program used to test libavcodec was a vlc git build.

[Carl Eugen Hoyos]

Regression since b7fe35c

[Michael Niedermayer]

Is the backtrace with b066d90211 applied ?

[Timo R.]

b066d90211 was applied

[Carl Eugen Hoyos]

Please test if this problem was fixed in d61681f.

[Timo R.]

No, still exactly the same crash.

#0  0x00007ffff742fba5 in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff743101b in __GI_abort () at abort.c:91
#2  0x00007ffff7428c4e in __assert_fail_base (fmt=<optimized out>, assertion=0x7fffc5c2f8c9 "obj_buffer", file=0x7fffc5c2f86e "i965_drv_video.c", line=<optimized out>, 
    function=<optimized out>) at assert.c:94
#3  0x00007ffff7428cf2 in __GI___assert_fail (assertion=0x7fffc5c2f8c9 "obj_buffer", file=0x7fffc5c2f86e "i965_drv_video.c", line=1780, 
    function=0x7fffc5c2f5b0 <__PRETTY_FUNCTION__.11269> "i965_decoder_render_picture") at assert.c:103
#4  0x00007fffc5bfb9e8 in i965_decoder_render_picture (num_buffers=6, buffers=<optimized out>, context=<optimized out>, ctx=<optimized out>) at i965_drv_video.c:1780
#5  i965_RenderPicture (ctx=<optimized out>, context=<optimized out>, buffers=<optimized out>, num_buffers=6) at i965_drv_video.c:2040
#6  0x00007fffc98c00df in ff_vaapi_render_picture (vactx=0x7fffc0001e70, surface=<optimized out>)
    at /var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999/libavcodec/vaapi.c:70
#7  0x00007fffc98c0418 in ff_vaapi_mpeg_end_frame (avctx=0x7fffcc00c1e0) at /var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999/libavcodec/vaapi.c:207
#8  0x00007fffc97d64ed in slice_end (pict=0x7fffcc00baa0, avctx=0x7fffcc00c1e0) at /var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999/libavcodec/mpeg12dec.c:1874
#9  decode_chunks (avctx=<optimized out>, picture=0x7fffcc00baa0, got_output=0x7fffe170bd2c, buf=0x7fffcc053100 "", buf_size=59957)
    at /var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999/libavcodec/mpeg12dec.c:2146
#10 0x00007fffc97d790a in mpeg_decode_frame (avctx=0x7fffcc00c1e0, data=0x7fffcc00baa0, got_output=0x7fffe170bd2c, avpkt=<optimized out>)
    at /var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999/libavcodec/mpeg12dec.c:2458
#11 0x00007fffc98b9290 in avcodec_decode_video2 (avctx=0x7fffcc00c1e0, picture=0x7fffcc00baa0, got_picture_ptr=0x7fffe170bd2c, avpkt=0x7fffe170bcc0)
    at /var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999/libavcodec/utils.c:1950
#12 0x00007fffca317cc0 in DecodeVideo (p_dec=0x7fffcc03a178, pp_block=<optimized out>) at avcodec/video.c:582
#13 0x00007ffff7166f98 in DecoderDecodeVideo (p_dec=0x7fffcc03a178, p_block=0x7fffcc053080) at input/decoder.c:1483
#14 0x00007ffff7167d36 in DecoderProcessVideo (b_flush=false, p_block=0x7fffcc053080, p_dec=0x7fffcc03a178) at input/decoder.c:1845
#15 DecoderProcess (p_dec=0x7fffcc03a178, p_block=<optimized out>) at input/decoder.c:2018
#16 0x00007ffff7167ea3 in DecoderThread (p_data=0x7fffcc03a178) at input/decoder.c:939
#17 0x00007ffff79acec6 in start_thread (arg=0x7fffe170c700) at pthread_create.c:305
#18 0x00007ffff74e27fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

[Carl Eugen Hoyos]

Could you test if the crash is still reproducible with 200ef1e ?

[Timo R.]

Unfortunatley not, still exactly the same assertion crash.
i965_decoder_render_picture: Assertion `obj_buffer' failed.

[Timo R.]

This fixes it for me:
​https://lists.libav.org/pipermail/libav-devel/2013-May/046901.html

[Carl Eugen Hoyos]

Thank you for testing!

[Carl Eugen Hoyos]

Does elupus' patch also fix the crash?
​http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/164345

[Timo R.]

elupus just reverted his that patch in the ffmpeg version shipped with xbmc, in favor of the one i linked.

[Carl Eugen Hoyos]

Should be finally fixed in master and 1.2 after I created an enormous mess when committing.
Thank you for the constant testing!