H.264 Decoding Fails on Raw Annex B Streams from VideoToolbox
|Reported by:||David Goodman||Owned by:|
|Blocking:||Reproduced by developer:||no|
|Analyzed by developer:||no|
Summary of the bug:
ffmpeg can successfully decode my stream of H264 Annex B NALs without a container (generated with VideoToolbox on an iOS device) using the x264 decoder, or even "-hwaccel cuda" on my Nvidia GPU, but fails when using d3d11va.
The bug seems to be in the d3d11va_validate_output() function in dxva2.c. I haven't been able to trace it down completely and create a proper fix yet, but I was able get decoding to work for my own use cases with a hacky workaround here: https://github.com/LIV/FFmpeg/commit/35e8d409085d225d7cbd4553b59bd40a8a6b9716. I'm forcing ffmpeg to use DXVA2_ModeH264_E regardless of the input stream, so this obviously isn't a proper solution.
How to reproduce:
% ffmpeg -hwaccel d3d11va -i h264_annexb_1440x1280.h264 output.mp4 '''ffmpeg version:''' 4.2.2 and 20200628-4cfcfb3 '''built on:''' Downloaded from ffmpeg.zeranoe.com/builds, and was able to reproduce with self-built 4.2.2 on Windows 10 1909.
Note that the white bottom half of my sample video is normal (it's an alpha channel).