Assertion when playing h264 with vlc using vdpau

[Timo R.]

When playing any h264 video in vlc(tested with version 2.1.4) using vdpau hwaccel, it aborts with the following assertion:

Assertion desc->flags & (1 << 4) failed at libavcodec/utils.c:722

Bisecting the issue ended up on the following two commits:

5c1d7246cd65dc4db1b6dc36e29ce39fc1068f3f
f2f99f07a5f1f74a91503cc6c3730c06498cee43

Playing h264 without vdpau acceleration works fine in vlc.
So does playing it with ffplay.
Also using the ffmpeg cli util with hwaccel vdpau works fine.

[Carl Eugen Hoyos]

Which FFmpeg version did you test / is this reproducible with current FFmpeg git head?

[Timo R.]

Yes, it still happens on current git master, which is where i first noticed it.
I then went backwards until i found a commit that still worked, and then bisected between those, with the result i already posted.

[Michael Niedermayer]

Can someone try if patch below fixes it:

diff --git a/libavcodec/pthread_frame.c b/libavcodec/pthread_frame.c
index 9e11038..2a67f4d 100644
--- a/libavcodec/pthread_frame.c
+++ b/libavcodec/pthread_frame.c
@@ -376,7 +376,7 @@ FF_ENABLE_DEPRECATION_WARNINGS
                 p->result = ff_get_buffer(p->avctx, p->requested_frame, p->requested_flags);
                 break;
             case STATE_GET_FORMAT:
-                p->result_format = p->avctx->get_format(p->avctx, p->available_formats);
+                p->result_format = ff_get_format(p->avctx, p->available_formats);
                 break;
             default:
                 call_done = 0;

[Timo R.]

Yes, with this patch applied to the current ffmpeg master VLC 2.1.4 can play video with VDPAU acceleration again.

[Michael Niedermayer]

Patch applied

Fixed in e36916a63f6fd0644461ecbd6b08eeb8dfab60ab

thanks for testing