MediaCodec Encoder: Pixel ColorSpace of many Adreno GPUs is not supported

[Ben T]

MediaCodec Encoder: Pixel ColorSpace of many Adreno GPUs is not supported (<= Adreno 650 in our testing)

What you were trying to accomplish?

I am trying to transcode an MP4 file using the MediaCodec encoder support that was recently added by Zhao Zhili <​zhilizhao@tencent.com> here (​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0ff18a7d6d496f89d9e007ddd1cad7116baf5c7c)
The problem you encountered:

On multiple phones containing an Adreno GPU, the encoding process immediately fails because the color space expected by the GPU is unsupported.

The exact error output:

“does not support color format 19”

Phone information:

Model Name: Samsung Galaxy S10

Model Number: SM-G793U1

GPU Information: Adreno 640

Other phones that were tested and not working with this same error:

Brand Model name Model number OS GPU

Huawei Honor 10 COL-L29 Android 8.1 Mali-G72
Xiaomi Redmi Note 9S Redmi Note 9S Android 11 Adreno (TM) 618
Samsung Galaxy Z Flip SM-F700U1 Android 11 Adreno (TM) 640
Razer Razer phone Phone Android 7.1.1 Adreno (TM) 540
Essential PH-1 PH-1 Android 8.1 Adreno (TM) 541
Huawei Nexus 6P Nexus 6P Android 8.1 Adreno (TM) 430
Samsung Galaxy S6 SM-G920A Android 7 Mali-T760
Samsung Galaxy Tab S4 (Tablet) SM-T830 Android 10 Adreno (TM) 540
Samsung Galaxy S20 FE SM-G780G/DS Android 12 Adreno (TM) 650
Samsung Galaxy S10 SM-G973U1 Android 11 Adreno (TM) 640
Samsung Galaxy Note10 SM-N970U1 Android 12 Adreno (TM) 640

The following phones did not exhibit this behavior and were able to encode without issues:

Brand Model name Model number OS GPU

Samsung Galaxy S21 Ultra 5G SM-G998U1 Android 12 Adreno (TM) 660
Huawei P10 plus VKY-L29 Android 8.1 Mali-G71
Huawei P40 Lite JNY-LX1 Android 10 Mali-G52
Samsung Galaxy S8 SM-G950F Android 9 Mali-G71
Samsung Galaxy S10+ Sm-G975F Android 12 Mali-G76
Huawei P30 ELE-L29 Android 10 Mali-G76
Google Pixel 4XL Pixel 4XL Android 13 Adreno (TM) 640
Samsung Galaxy S7 SM-G930F Android 8 Mali-T880
Samsung Galaxy S10 SM-G973F Android 11 Mali-G76
Samsung Galaxy S20 FE SM-G780F Android 11 Mali G77
Google Pixel 6 Pixel 6 Android 13 Mali-G78

The exact command line you were using:

-hwaccel mediacodec -hwaccel_output_format mediacodec -i "/data/user/0/com.test.ffmpeg_kit_android_demo/cache/test_input.mp4" -c:v h264_mediacodec -ndk_codec 1 -y "/data/user/0/com.test.ffmpeg_kit_android_demo/cache/test_output.mp4" -loglevel trace
Console output:

See attached console_output.log
Input Files:

See attached test_input.mp4

Note: This is built and run using FFMPEG-Kit

[Ben T]

Console Log

[Ben T]

Input File

[Ben T]

Copy-Pasting response from Zhao Zhili in the mailing list where I originally incorrectly submitted this ticket:

Thank you for your valuable test!

There is an unexpected behavior in the test. The test was conducted with ffmpeg-kit, not ffmpeg cmdline.
With FFmpeg cmdline and -hwcontext mediacodec, the encoder should have "-pix_fmt mediacodec" by
default, and have an AVMediaCodecDeviceContext (with Android 8.0 or higher). The encoder doesn't
setup properly with ffmpeg-kit.

pix_fmt mediacodec should be supported by most of Android devices. It do have its limitation, e.g., it needs
a Surface object or require Android 8.0, and it doesn't work with avfilter.

The above issue can be fixed on the ffmpeg-kit's side. Now comes the dark corner of Mediacodec. It should
work without pix_fmt mediacodec. Can you try specify the encoder pix_fmt as nv12, that's

-hwaccel mediacodec -hwaccel_output_format mediacodec -i test_input.mp4 -c:v h264_mediacodec -ndk_codec 1 -pix_fmt nv12 -y test_output.mp4

I guess these devices work with nv12.

There is an old story "MediaCodec and device compatibility"

​https://groups.google.com/g/android-platform/c/p_MoSk0JPNM

Ten years on, the situation doesn't improved much:

1. There is an ugly API to get color format supported by a codec.
2. It's only available in Java.
3. Different codec component have different color format.

Yes, we can use a bunch of JNI to get the supported color format. It must be done before create
AVCodecContext. But different mediacodec component have different color format, user can select
mediacodec by name (see the following patch), we have only one AVCodec.pix_fmts.

avcodec/mediacodecenc: add option to select codec by name
​https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_729F52ABE34E4770972DB89AF840549A8208@qq.com/

We can add query the supported color format and add some hints when mediacodec configure failed.
Or we can add a helper function for user to query the color format. It will be a bunch of JNI code, limited
by the ugly MediaCodecInfo.CodecCapabilities API. It's easy to be done with Java. TBH, it's much easier with
trial and error, try nv12 then yuv420.

We can do something with FFCodec.init_static_data, but we need a strategy first. I don't want to go with a long
blacklist/whitelist.

We only need one pixel format which supported by all devices, COLOR_FormatSurface doesn't count, and
COLOR_FormatYUV420Flexible doesn't count.

Maybe Google can fix these issues with another 10 year.

[Ben T]

Using -px_fmt nv12 does resolve the issue on a quick test that I ran on a phone that was failing. I'll verify that it resolves the issue on the remainder of the phones.

What you mentioned about using some sort of method to guess the most likely correct pixel format sounds reasonable.

Maybe Google can fix these issues with another 10 year.

Always how it is ;)

[Balling]

ColorSpace is not the same as color format. Color format is just pixel format.

[Balling]

Android loves to do NV12 everywhere (that is Semi Planar), that is just how it always has been. That is because NV12 is just using 12 bits per pixel, where Y is 8 bit, while 4 other bits are used for Cb + Cr.

Just by googling "does not support color format 19" ​https://itecnote.com/tecnote/android-illegal-exception-on-mediacodec-configure/ that talks about it and anyway

​https://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities#COLOR_FormatYUV420SemiPlanar

is deprecated since android 6.0.

Maybe Google can fix these issues with another 10 year.

You mean Qualcomm codec? Cause it works in emulator ​https://stackoverflow.com/questions/17339576/error-omx-ti-ducati1-video-mpeg4e-does-not-support-color-format-android

[quinkblack]

Replying to Balling:

Android loves to do NV12 everywhere (that is Semi Planar), that is just how it always has been. That is because NV12 is just using 12 bits per pixel, where Y is 8 bit, while 4 other bits are used for Cb + Cr.

Just by googling "does not support color format 19" ​https://itecnote.com/tecnote/android-illegal-exception-on-mediacodec-configure/ that talks about it and anyway

​https://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities#COLOR_FormatYUV420SemiPlanar

is deprecated since android 6.0.

Maybe Google can fix these issues with another 10 year.

You mean Qualcomm codec? Cause it works in emulator ​https://stackoverflow.com/questions/17339576/error-omx-ti-ducati1-video-mpeg4e-does-not-support-color-format-android

Yes, I known NV12 works for most of devices, and YUV420p works for a lot of devices and mediacodec implementaions, including Google's software decoder/encoder.

Neither NV12 nor YUV420p is mandatory by Google. The mandatory one is COLOR_FormatYUV420Flexible, which can be "COLOR_FormatYUV411Planar, COLOR_FormatYUV411PackedPlanar, COLOR_FormatYUV420Planar, COLOR_FormatYUV420PackedPlanar, COLOR_FormatYUV420SemiPlanar and COLOR_FormatYUV420PackedSemiPlanar".

[Balling]

mandatory one is COLOR_FormatYUV420Flexible

Cause that is what deprecation was in favour of.