HEVC bitstream with alpha layer to mp4

[John]

Summary

I used NVIDIA's Video Codec SDK to create an HEVC bitstream containing an alpha layer. I tried to mux this bitstream into an MP4 container with no success. I was hoping this would work because:

1. It was a copy, not a re-encode (no need to involve libx265)
2. This change gave me hope: ​https://github.com/FFmpeg/FFmpeg/commit/ad326379c6634505c9ebc46964057441761008bc

No cigar. FFmpeg did create an MP4 file but the MP4 will not play in QuickTime and does not seem to retain the alpha layer. Whenever a nuh_layer_id > 0 is encountered FFmpeg outputs:
missing picture in access unit

I've attached the ffmpeg -v trace ... output to this request as well as the HEVC bitstream containing the alpha layer.

How to reproduce

% ffmpeg -i bsWithAlpha.265 -c:v copy -tag:v hvc1 outputWithAlpha.mp4

[John]

HEVC bitstream with alpha layer

[Carl Eugen Hoyos]

Please test current FFmpeg git head, the only version supported on this bug tracker.

[Balling]

It does not even ffplay correctly. What do you want from it? I am also not sure we support annex b to mp4 properly and fully, etc, etc. I mean there is a bug that blocks Mastering display luminance : min: 0.0000 cd/m2, max: 0 cd/m2 in -bsf:v hevc_metadata! It is a joke, okay. Sigh. ​https://github.com/FFmpeg/FFmpeg/blob/741565a1e69c45ce4848f01e45be5e66a68efa2f/libavcodec/cbs_h264_syntax_template.c#L816

P.S. Sorry ;) That was fixed by ​https://github.com/FFmpeg/FFmpeg/commit/4bf338480035238b3ff595ebd761688fc0db9e11#diff-fce57dfc99a19823f0294919d3086ae05d3bb6d170b613df3946753147b70d59L816

Yeah!

[John]

Replying to cehoyos:

Please test current FFmpeg git head, the only version supported on this bug tracker.

I tested again this morning with the same results

% date  
Tue Feb  2 08:24:29 PST 2021
% git pull
Already up to date.
% git remote -v
origin	https://git.ffmpeg.org/ffmpeg.git (fetch)
origin	https://git.ffmpeg.org/ffmpeg.git (push)

[Balling]

Related: ​https://forums.developer.nvidia.com/t/hevc-with-alpha-layer/167353
​https://github.com/johnhe4/nvenc_h265_transparency

Please, next time provide all the links.

[Carl Eugen Hoyos]

Replying to Balling:

Related: ​https://forums.developer.nvidia.com/t/hevc-with-alpha-layer/167353
​https://github.com/johnhe4/nvenc_h265_transparency

Both look unrelated to this ticket.

Should not this be closed as a duplicate of #7965?

This ticket is about an issue with libavformat, the other ticket is libavcodec-only.

[Balling]

Both look unrelated to this ticket.

Both (the code and nvidia comment) were done by the same user that reported this issue...

[John]

Yes, those links are related but I don't think they affect the scope. Sorry for not posting.

No, this ticket is not a duplicate of #7965. Perhaps the best way to explain why is to detail my target use case.

I am looking to segment out video using a Linux instance in the cloud. This segmentation process will result in two layers: a foreground layer and a background layer. I want to stream the foreground layer over the Internet to an Apple device that will render this foreground layer over whatever background the app chooses.

I know of two traditional methods to accomplish this that make inefficient use of bandwidth and/or push complexity to the end device. Traditional method 1 is to send both layers as separate streams, or if possible, as separate tracks in a single stream. This requires the device to spin up two decoders, perform post and or pre-decode synchronization, and possibly deal with multiple-multiple bitrates. Traditional method 2 is to have the Linux server overwrite the background with a predefined chroma key. This requires a carefully chosen chroma key that will not unexpectedly occur in the foreground, and requires custom shaders on the end device.

Alpha support in HEVC seems (to me) a much better solution than the traditional methods, if it is supported.

This ticket is different than #7965 because it specifically asks for muxing support only. I would like FFmpeg to mux an existing HEVC bitstream which includes alpha (encoded with whatever, in this case NVidia but can be anything) into an MP4 (or TS file) that Apple can decode. From my limited understanding of FFmpeg, any changes relating to this ticket wouldn't need to touch the encoder or decoder but would instead touch muxers and parsers.

Here is a link from Apple that includes more information for HEVC with alpha, including the profile constraints on Apple devices: ​https://developer.apple.com/videos/play/wwdc2019/506/

I noticed some activity on #7965, that is great to see. I imagine this ask will grow as more non-Apple encoders begin to support HEVC with alpha.

[quinkblack]

Remux works fine with samples generated from Apple's videotoolbox.

The reason to current issue is that nvenc alpha layer encoding is broken.
​http://ffmpeg.org/pipermail/ffmpeg-devel/2022-December/305204.html

Nothing we can do until Nvidia fixed their SDK.