Support for HEVC with Alpha

[leemartin]

At WWDC this year, Apple introduced support for HEVC with Alpha to both their app and browser platforms. You can find their talk on the subject here:

​https://developer.apple.com/videos/play/wwdc2019/506

I'm an amateur at FFmpeg but I went down a rabbit hole today, trying to create a video in this new format using the latest version of FFmpeg. I took a rather simple approach of using ffprobe on one of their example video files (found in the resources on the link above) to get a profile and attempting to tune an ffmpeg encoding command to fit the same profile.

While the following command does recreate the profile, it does not retain the transparency of the source prores .mov file.

% ffmpeg -i puppets_with_alpha_prores.mov -c:v libx265 -tag:v hvc1 -pix_fmt yuv420p -colorspace bt709 -color_primaries bt709 -color_trc bt709 puppets_with_alpha_hevc.mov

Apple is pretty clear about how they are achieving this encoding in their talk and examples. Naturally, I didn't follow anything they said and simply tried to hack a solution.

So I'm wondering if this is something FFmpeg can do or would be able to support in the future?

Thanks for the help!

[Carl Eugen Hoyos]

A small sample file will be useful.

[leemartin]

Replying to cehoyos:

A small sample file will be useful.

Attached Apple's sample file. You can also see it in action on iOS 13 Safari with the following code block

<video src="puppets_with_alpha_hevc.mov" autoplay muted playsinline loop></video>

As seen in this tweet:

​https://twitter.com/leemartin/status/114133658079747686

[Carl Eugen Hoyos]

Replying to leemartin:

Replying to cehoyos:

A small sample file will be useful.

As seen in this tweet:

​https://twitter.com/leemartin/status/114133658079747686

404

[leemartin]

Sorry about that. Here's the right link:

​https://twitter.com/leemartin/status/1141336580797476865

Replying to cehoyos:

Replying to leemartin:

Replying to cehoyos:

A small sample file will be useful.

As seen in this tweet:

​https://twitter.com/leemartin/status/114133658079747686

404

[shaocaholica]

You would need to use a pix_fmt that has an alpha channel as well. Any of the yuv(a) formats thats also supported by h.265.

[leemartin]

Using my command above, I try swapping the pix_fmt for 'yuva420p' I receive the warning:

Incompatible pixel format 'yuva420p' for codec 'libx265', auto-selecting format 'yuv420p'

Is there an alpha supported pixel format you can suggest?

[shaocaholica]

I just mean if the feature is added you would need an alpha channel pix_fmt. The one you end up using will depend on whats supported as far as color sub sampling and bit depth.

[Carl Eugen Hoyos]

Note that this ticket - as shown by the sample attached - is about supporting decoding of transparency-containing (Apple) files and providing the transparency.
Encoding requests cannot be made here as FFmpeg does not contain a native hevc encoder.

[pecus]

Apple interoperability profile proposal

[pecus]

In case it might help, Apple had released a technical document that I did not see referenced here, so I uploaded it.

I really hope ffmpeg will provide support for decoding and encoding of transparency/alpha supporting video (I know that encoding depends on libx265 providing it)

[Balling]

Support in hevc_videotoolbox ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210223050227.65698-1-bouno@rouge.plala.or.jp/

Also nvidia started to add this thingy. See: #9088

As I understand x265 does not support alpha ​https://bitbucket.org/multicoreware/x265_git/issues/577/support-for-alpha-transparency-per-apple

[Alexander]

It would be great to see HEVC alpha transparency supported in FFMpeg. It might look like a minor issue but it’s really not. In WebGL toolkits like Verge3D, transparent textures are used to create various cool effects. In Chrome/Firefox/Edge we have WebM, but for Safari (for both desktop and mobile), HEVC is the only option. Right now we ​recommend our users to use FFMpeg with VideoToolbox encoder, but it is limited to Macs only. Thank you for consideration!

[Balling]

Some support ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_5F1F4DA9CFDF5521A752B8C9161A4D23A007@qq.com/

And ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_E0CDAECFC0550E2778B890163D4D39201F07@qq.com/

There is also support added in one of Nvidia's projects. ​https://github.com/nvpro-samples/vk_video_samples/issues/14

[logiclrd]

As I understand x265 does not support alpha

It does now. As of version 4.0, which dropped a month or two back, there is a --alpha command-line option. With this option, the input is assumed to be a yuva420p raw stream. As of writing this, to get this latest version you must build x265 from source. At least on my OS, the central package manager doesn't yet have version 4.0.

I have found the following sequence which seems to produce an MP4 file with HEVC with alpha content, but I am not on a Mac and have found no way to test it because the decoding support described by this ticket is not yet implemented. :-)

x265 --input-res 606x692 --fps 30000/1001 --alpha --crf 20 Zap.yuva420p Zap.hevc
mkvmerge Zap.hevc -o Zap.mkv
ffmpeg -itsscale 0.3336666666 -i Zap.mkv -c copy -vtag hvc1 Zap.mp4 -y

The -itsscale option is to fix an issue with the framerate. I couldn't find a way to get mkvmerge to produce the right output (there may be a way). By default, it seems to emit HEVC frames timestamped to 10fps. This scale factor converts from 10fps to 30000/1001 fps.

Hopefully this information is helpful to someone.

There is a tool called Zond 265 which does some fascinating inspection of internal H.265 encoding details. It does not support alpha channels yet, and I was told that the reason is that FFMPEG doesn't decode HEVC with Alpha yet.

I also found this related issue in Trac: #10901

(Does Trac have a mechanism for relationships between tickets?? A quick web search suggests that the answer is Yes, but only if you use an unmaintained plugin called TracTicketReferencePlugin...)

[Igor]

I noticed a recent commit that adds (some) support for packaging HEVC with alpha layer.
​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/17e4746687abc53f0a042620a7af6dd6cea44b80

However, I still cannot get a video that works. I am trying the following

ffmpeg -vcodec libvpx-vp9 -i input.webm -c:v libx265 -x265-params "alpha=1" output.mp4

Output

Input #0, matroska,webm, from 'input.webm':
  Metadata:
    encoder         : libwebm-0.3.0.0
    creation_time   : 2023-11-29T12:59:06.000000Z
  Duration: 00:00:03.00, start: 0.000000, bitrate: 4790 kb/s
  Stream #0:0(eng): Video: vp9 (libvpx-vp9) (Profile 0), yuva420p(tv), 1200x1638, SAR 1:1 DAR 200:273, 30 fps, 30 tbr, 1k tbn (default)
    Metadata:
      alpha_mode      : 1
[libvpx-vp9 @ 0x5591173d7000] v1.9.0

Stream mapping:
  Stream #0:0 -> #0:0 (vp9 (libvpx-vp9) -> hevc (libx265))
Press [q] to stop, [?] for help
[libvpx-vp9 @ 0x5591173d7000] v1.9.0
x265 [info]: HEVC encoder version 4.1+54-fa2770934
x265 [info]: build info [Linux][GCC 10.2.1][64 bit] 8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 12 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 3 / wpp(26 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias  : 25 / 250 / 40 / 5.00 
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-28.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing lslices=8 deblock sao alpha
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf61.9.101
  Stream #0:0(eng): Video: hevc (hev1 / 0x31766568), yuva420p(tv, progressive), 1200x1638 [SAR 1:1 DAR 200:273], q=2-31, 30 fps, 15360 tbn (default)
    Metadata:
      encoder         : Lavc61.28.100 libx265
      alpha_mode      : 1
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A

Funny that ffmpeg understands that the output should be in yuva420p but the resulting file isn't recognized by Safari or QuickTime on Mac, but plays on Windows without transparency (though like the input file).

ffprobe does not recognize alpha layer (I guess when it says Scalability type 2 not supported)

[hevc @ 0x56075b908c40] Scalability type 2 not supported
[hevc @ 0x56075b908c40] Ignoring unsupported VPS extension
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.9.101
  Duration: 00:00:03.00, start: 0.000000, bitrate: 854 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, progressive), 1200x1638 [SAR 1:1 DAR 200:273], 842 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc61.28.100 libx265
[hevc @ 0x56075b925b40] Scalability type 2 not supported
[hevc @ 0x56075b925b40] Ignoring unsupported VPS extension

[Igor]

Replying to logiclrd:

I have found the following sequence which seems to produce an MP4 file with HEVC with alpha content, but I am not on a Mac and have found no way to test it because the decoding support described by this ticket is not yet implemented. :-)

x265 --input-res 606x692 --fps 30000/1001 --alpha --crf 20 Zap.yuva420p Zap.hevc
mkvmerge Zap.hevc -o Zap.mkv
ffmpeg -itsscale 0.3336666666 -i Zap.mkv -c copy -vtag hvc1 Zap.mp4 -y

I also tried using x265 cli to encode separately into *.hevc and then package into mp4, but the result is equal to using ffmpeg directly.

I see you use mkvmerge, why this interim step is needed?

[James]

Replying to Igor:

I noticed a recent commit that adds (some) support for packaging HEVC with alpha layer.
​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/17e4746687abc53f0a042620a7af6dd6cea44b80

However, I still cannot get a video that works. I am trying the following

ffmpeg -vcodec libvpx-vp9 -i input.webm -c:v libx265 -x265-params "alpha=1" output.mp4

Can you upload the output? I don't have a new enough libx265 to generate one.

[Igor]

Replying to James:

Replying to Igor:

I noticed a recent commit that adds (some) support for packaging HEVC with alpha layer.
​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/17e4746687abc53f0a042620a7af6dd6cea44b80

However, I still cannot get a video that works. I am trying the following

ffmpeg -vcodec libvpx-vp9 -i input.webm -c:v libx265 -x265-params "alpha=1" output.mp4

Can you upload the output? I don't have a new enough libx265 to generate one.

Here you go ​https://drive.google.com/file/d/1mYBiPWaHyWjlMPz4r09EweIKEuWJrtjF/view?usp=sharing

[James]

The file seems to have all the required parameter sets and alpha layer slices, so the only thing it's missing is the hvc1 tag. Use -tag:v hvc1 for the output.

You can also remove the custom alpha=1 x265-param as the library will detect the pixel format you're passing to it has an alpha plane.

[Igor]

Replying to James:

The file seems to have all the required parameter sets and alpha layer slices, so the only thing it's missing is the hvc1 tag. Use -tag:v hvc1 for the output.

You can also remove the custom alpha=1 x265-param as the library will detect the pixel format you're passing to it has an alpha plane.

Appreciate your effort to look into the file, adding a tag indeed did the trick!

So, for the folks, that are looking for a solution, here what I used:

ffmpeg -vcodec libvpx-vp9 -i input.webm -c:v libx265 -tag:v hvc1 output.mp4

Please mind that I compiled both ffmpeg and x265 from the upstream.