8-bit pixel format is used when decoding 10-bit video using VideoToolbox

[fumoboy007]

Summary of the bug:
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange is always used regardless of the bit depth of the video.

[fumoboy007]

Here is an unfinished patch. It is unfinished because we should probably check at runtime for macOS 10.13+ before using the 10-bit pixel format since that is when it was introduced. I don’t have time in the immediate future to figure out how to check the macOS version at runtime.

[Aman]

Could you please submit git-format-patch version of the patch including Author info so we can commit it? There are some patterns used to define missing symbols on older versions of macOS which I can add to make this patch complete.

[fumoboy007]

Thanks for picking this up! I’ve attached the patch. Let me know if there is anything else you need.

[Aman]

Thanks. I forgot to specify that ffmpeg-devel is the preferred place for patches, so they can be more widely reviewed.

As far as macOS compatibility, you can refer to how I added compatibility checks for kCMVideoCodecType_HEVC here:

​https://github.com/FFmpeg/FFmpeg/commit/3d4f8b9184a4693c577e0b73496e6cc1989c6bbf#diff-c9d8dd2e290f045e209331df387c0bbaR43
​https://github.com/FFmpeg/FFmpeg/commit/3d4f8b9184a4693c577e0b73496e6cc1989c6bbf#diff-e2d5a00791bce9a01f99bc6fd613a39dR2084
​https://github.com/FFmpeg/FFmpeg/commit/3d4f8b9184a4693c577e0b73496e6cc1989c6bbf#diff-e2d5a00791bce9a01f99bc6fd613a39dR5822

If you feel so inclined, take a stab at adding compatibility using the approach above and send the new patch to the ffmpeg-devel list. If not, I will plan to pick this up in a few weeks and do the same. Thanks for your contribution!

[Carl Eugen Hoyos]

Patch applied as 036b4b0f85933f49a7094b5b568a93f68c9cd544