Opened 3 years ago
Closed 2 years ago
#9713 closed defect (fixed)
Hardware accelerated decoding fails on M1 macs for certain videos encoded with h264
Reported by: | mikffmpeg | Owned by: | |
---|---|---|---|
Priority: | normal | Component: | avcodec |
Version: | git-master | Keywords: | h264 |
Cc: | Blocked By: | ||
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description (last modified by )
Summary of the bug:
Using hardware acceleration (i.e. videotoolbox) to decode the frames of some video streams encoded with h264 fails on recent M1 mac computers.
How to reproduce:
Ensure you are using a Mac with an Apple M1 chip.
Download the attached sample file and use ffmpeg to target it with
% ffmpeg -hwaccel videotoolbox -i m1_not_decodable_with_acceleration.ts -an output.ts
The terminal will then repeatedly print
[h264 @ 0x7f85b5f32ec0] hardware accelerator failed to decode picture Error while decoding stream #0:0: Unknown error occurred
Other details:
This bug does not occur with all videos encoded with h264.
Although the issue can be reproduced with the most recent ffmpeg binary (version N-106494-g1291568c98-tessus), when looking at the source code of FFmpeg 3.3, it seems that the failure is caused by a call to VTDecompressionSessionDecodeFrame. When decoding works, videotoolbox_decoder_callback is called with a non-null image_buffer. When decoding fails, image_buffer is null and status is kVTVideoDecoderBadDataErr (-12909). The result is that the VTContext's frame field stays null, which then triggers a null pointer check down the line. In either case, VTDecompressionSessionDecodeFrame returns successfully.
I should also state I couldn’t spot any difference in the arguments passed to videotoolbox when I used lldb on Intel and on the M1, even though on Intel FFmpeg decodes all frames without issues. I also didn’t see any differences in the code path leading up to the decoding call.
Attachments (4)
Change History (14)
by , 3 years ago
Attachment: | m1_not_decodable_with_acceleration.ts added |
---|
by , 3 years ago
Attachment: | ffmpeg-20220405-121427.log added |
---|
result of ffmpeg -v 9 -loglevel 99 -report -i
comment:1 by , 3 years ago
Description: | modified (diff) |
---|---|
Keywords: | avcodec m1 mac h264 added |
comment:2 by , 3 years ago
Description: | modified (diff) |
---|
by , 3 years ago
Attachment: | ffmpeg-20220405-170825.log added |
---|
result of ffmpeg -hwaccel videotoolbox -v 9 -loglevel 99 -report -i m1_not_decodable_with_acceleration.ts -an output2.ts
comment:3 by , 3 years ago
Description: | modified (diff) |
---|
comment:5 by , 3 years ago
This seems to happen, when the bitstream provided for decoding doesn't contain SPS/PPS.
For example, using the provided m1_not_decodable_with_acceleration.ts
:
ffmpeg -i m1_not_decodable_with_acceleration.ts -vcodec copy m1_not_decodable_with_acceleration.h264 ffmpeg -hwaccel videotoolbox -i m1_not_decodable_with_acceleration.h264 -v debug /tmp/img%05d.jpg
will still fail, but
ffmpeg -i m1_not_decodable_with_acceleration.ts -vcodec copy m1_not_decodable_with_acceleration.mkv ffmpeg -hwaccel videotoolbox -i m1_not_decodable_with_acceleration.mkv -v debug /tmp/img%05d.jpg
works. The difference seems to be what is fed into the decoder:
Broken with .h264 (and .ts): SPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 67 4d 00 2a 9d a8 1e 00 89 f9 70 06 e0 20 20 28 00 00 00 08 00 00 01 94 20 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< SPS-init - end PPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 68 ee 3c <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PPS-init - end decoding - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 00 02 04 b7 65 b8 00 00 0e ab 00 cf f7 49 24 77 Working with .mkv: SPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 67 4d 00 2a 9d a8 1e 00 89 f9 70 06 e0 20 20 28 00 00 00 08 00 00 01 94 20 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< SPS-init - end PPS-init - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 68 ee 3c <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PPS-init - end decoding - begin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 00 00 00 02 09 f0 00 00 00 1b 67 4d 00 2a 9d a8 1e 00 89 f9 70 06 e0 20 20 28 00 00 03 00 08 00 00 03 01 94 20 00 00 00 04 68 ee 3c 80 00 00 00 05 06 e5 01 41 80 00 02 04 b7 65 b8 00 00 0e ab 00 cf f7 49 24 77 ... matches after this
As you see, there are SPS/PPS NALUs present in the bitstream provided when reading from mkv -- and that seems to make Apple gods happy.
On Intel Mac, of course things work in all the above scenarios.
by , 3 years ago
Attachment: | error.h264 added |
---|
comment:6 by , 3 years ago
So, I've found something interesting:
PPS NAL unit, as it appears in the bitstream, is 4 bytes long. PPS when it being used by videotoolbox.c, is 3 bytes long.
I've tested a theory, and indeed, if I add the missing last byte, 0x80, to PPS being used for videotoolbox initialization, previously failing streams work with videotoolbox on M1.
However now I'm finding myself in H264 parser code, which I'm not overfly familiar with, and seeing some definite weirdness there: ff_h264_decode_picture_parameter_set
is being called twice, and gb
parameter first indicates the length of 5, and length of 3 in the second call.
What is the expectation? Is the parsed PPS buffer supposed to preserve the size of PPS NALU? I'm attaching the raw error.h264 I'm working with ... these are the initial NAL units in that file:
00 00 00 02 9 - Access unit delimiter 09 f0 00 00 00 1b 7 - SPS 67 4d 00 2a 9d a8 1e 00 89 f9 70 06 e0 20 20 28 00 00 03 00 08 00 00 03 01 94 20 00 00 00 04 8 - PPS 68 ee 3c 80 00 00 00 05 6 - SEI 06 e5 01 41 80 00 02 04 b7 5 - IDR 65 b8 00 00 0e ab 00 cf f7 49 24 77 ...
comment:7 by , 3 years ago
I've tested a theory, and indeed, if I add the missing last byte
You mean you saw it here: https://patchwork.ffmpeg.org/project/ffmpeg/patch/bf15fa33c739b0a4b4d32bedeec6f326e3b175e0.1652120184.git.ffmpegagent@gmail.com/
follow-up: 9 comment:8 by , 3 years ago
Should be fixed in c534d9f72a89542ed639071b1ae15893aadf1f18. Please test!
Hope it is not going to break any other code.
comment:9 by , 2 years ago
Replying to Balling:
Should be fixed in c534d9f72a89542ed639071b1ae15893aadf1f18. Please test!
Hope it is not going to break any other code.
It looks like this commit does indeed fix the issue... great work!
comment:10 by , 2 years ago
Keywords: | avcodec m1 mac removed |
---|---|
Resolution: | → fixed |
Status: | new → closed |
sample video that fails when decoded