H.264 Decoding Fails on Raw Annex B Streams from VideoToolbox

[David Goodman]

Summary of the bug:
ffmpeg can successfully decode my stream of H264 Annex B NALs without a container (generated with VideoToolbox on an iOS device) using the x264 decoder, or even "-hwaccel cuda" on my Nvidia GPU, but fails when using d3d11va.

The bug seems to be in the d3d11va_validate_output() function in dxva2.c. I haven't been able to trace it down completely and create a proper fix yet, but I was able get decoding to work for my own use cases with a hacky workaround here: ​https://github.com/LIV/FFmpeg/commit/35e8d409085d225d7cbd4553b59bd40a8a6b9716. I'm forcing ffmpeg to use DXVA2_ModeH264_E regardless of the input stream, so this obviously isn't a proper solution.

How to reproduce:

% ffmpeg -hwaccel d3d11va -i h264_annexb_1440x1280.h264 output.mp4
'''ffmpeg version:'''
4.2.2 and 20200628-4cfcfb3
'''built on:'''
Downloaded from ffmpeg.zeranoe.com/builds, and was able to reproduce with self-built 4.2.2 on Windows 10 1909.

Note that the white bottom half of my sample video is normal (it's an alpha channel).

[David Goodman]

Output log generated with -report

[David Goodman]

I'm having trouble with the VideoLAN File Uploader, so here's a Google Drive link for a sample clip that makes causes the issue -- hope this is acceptable: ​https://drive.google.com/file/d/1iUWPFUE7ODLucp0dNIJGZKWTiwTcCZja/view?usp=sharing

[Balling]

Unfortunately, we do not have h265_annexbtomp4 or even h264_annexbtomp4 but it is there ​https://github.com/ireader/media-server/commit/76e62c806aaa5e1f97c626a52df0b03c2b2b47b1 (and other commits). After that we can tell whether this file is corrupted or not. Just saying.

[Elon Musk]

File is invalid.

[Balling]

Replying to richardpl:

File is invalid.

But we are not able to restore from Annex B back, are not we??

P.S. Looks like we can after all, sorry ;)