Hardware accelerated decoding fails on M1 macs for certain videos encoded with h264

[mikffmpeg]

Summary of the bug:
Using hardware acceleration (i.e. videotoolbox) to decode the frames of some video streams encoded with h264 fails on recent M1 mac computers.

How to reproduce:
Ensure you are using a Mac with an Apple M1 chip.
Download the attached sample file and use ffmpeg to target it with

% ffmpeg -hwaccel videotoolbox -i m1_not_decodable_with_acceleration.ts -an output.ts

The terminal will then repeatedly print

[h264 @ 0x7f85b5f32ec0] hardware accelerator failed to decode picture
Error while decoding stream #0:0: Unknown error occurred

Other details:
This bug does not occur with all videos encoded with h264.

Although the issue can be reproduced with the most recent ffmpeg binary (version N-106494-g1291568c98-tessus), when looking at the source code of FFmpeg 3.3, it seems that the failure is caused by a call to ​VTDecompressionSessionDecodeFrame. When decoding works, ​videotoolbox_decoder_callback is called with a non-null image_buffer. When decoding fails, image_buffer is null and status is kVTVideoDecoderBadDataErr (-12909). The result is that the VTContext's frame field stays null, which then triggers a null pointer check down the line. In either case, VTDecompressionSessionDecodeFrame returns successfully.

I should also state I couldn’t spot any difference in the arguments passed to videotoolbox when I used lldb on Intel and on the M1, even though on Intel FFmpeg decodes all frames without issues. I also didn’t see any differences in the code path leading up to the decoding call.

[mikffmpeg]

sample video that fails when decoded

[mikffmpeg]

result of ffmpeg -v 9 -loglevel 99 -report -i

[mikffmpeg]

result of ffmpeg -hwaccel videotoolbox -v 9 -loglevel 99 -report -i m1_not_decodable_with_acceleration.ts -an output2.ts

[Alex Agranovsky]

Same problem also described in #9016.

[Alex Agranovsky]

This seems to happen, when the bitstream provided for decoding doesn't contain SPS/PPS.
For example, using the provided m1_not_decodable_with_acceleration.ts​:

ffmpeg -i m1_not_decodable_with_acceleration.ts -vcodec copy 
m1_not_decodable_with_acceleration.h264 
ffmpeg -hwaccel videotoolbox -i m1_not_decodable_with_acceleration.h264 -v debug /tmp/img%05d.jpg

will still fail, but

ffmpeg -i m1_not_decodable_with_acceleration.ts -vcodec copy 
m1_not_decodable_with_acceleration.mkv 
ffmpeg -hwaccel videotoolbox -i m1_not_decodable_with_acceleration.mkv -v debug /tmp/img%05d.jpg

works. The difference seems to be what is fed into the decoder:

Broken with .h264 (and .ts):

SPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
67 4d 00 2a 9d a8 1e 00 89 f9 70 06 e0 20 20 28
00 00 00 08 00 00 01 94 20
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< SPS-init - end
PPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
68 ee 3c
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PPS-init - end
decoding - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
00 02 04 b7 65 b8 00 00 0e ab 00 cf f7 49 24 77

Working with .mkv:

SPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
67 4d 00 2a 9d a8 1e 00 89 f9 70 06 e0 20 20 28
00 00 00 08 00 00 01 94 20
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< SPS-init - end
PPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
68 ee 3c
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PPS-init - end
decoding - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
00 00 00 02 09 f0 00 00 00 1b 67 4d 00 2a 9d a8
1e 00 89 f9 70 06 e0 20 20 28 00 00 03 00 08 00
00 03 01 94 20 00 00 00 04 68 ee 3c 80 00 00 00
05 06 e5 01 41 80 00 02 04 b7 65 b8 00 00 0e ab
00 cf f7 49 24 77 ... matches after this

As you see, there are SPS/PPS NALUs present in the bitstream provided when reading from mkv -- and that seems to make Apple gods happy.

On Intel Mac, of course things work in all the above scenarios.

[Alex Agranovsky]

So, I've found something interesting:
PPS NAL unit, as it appears in the bitstream, is 4 bytes long. PPS when it being used by videotoolbox.c, is 3 bytes long.

I've tested a theory, and indeed, if I add the missing last byte, 0x80, to PPS being used for videotoolbox initialization, previously failing streams work with videotoolbox on M1.

However now I'm finding myself in H264 parser code, which I'm not overfly familiar with, and seeing some definite weirdness there: ff_h264_decode_picture_parameter_set is being called twice, and gb parameter first indicates the length of 5, and length of 3 in the second call.

What is the expectation? Is the parsed PPS buffer supposed to preserve the size of PPS NALU? I'm attaching the raw error.h264 I'm working with ... these are the initial NAL units in that file:

00 00 00 02             9 - Access unit delimiter
09 f0           
00 00 00 1b             7 - SPS
67 4d 00 2a 9d a8 1e 00 89 f9 70 06 e0 20 20 28 
00 00 03 00 08 00 00 03 01 94 20 
00 00 00 04             8 - PPS
68 ee 3c 80             
00 00 00 05             6 - SEI
06 e5 01 41 80          
00 02 04 b7             5 - IDR
65 b8 00 00 0e ab 00 cf f7 49 24 77 ... 

[Balling]

I've tested a theory, and indeed, if I add the missing last byte

You mean you saw it here: ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/bf15fa33c739b0a4b4d32bedeec6f326e3b175e0.1652120184.git.ffmpegagent@gmail.com/

[Balling]

Should be fixed in c534d9f72a89542ed639071b1ae15893aadf1f18. Please test!

Hope it is not going to break any other code.

[mikffmpeg]

Replying to Balling:

Should be fixed in c534d9f72a89542ed639071b1ae15893aadf1f18. Please test!

Hope it is not going to break any other code.

It looks like this commit does indeed fix the issue... great work!