Opened 3 years ago

Closed 15 months ago

#9088 closed defect (worksforme)

HEVC bitstream with alpha layer to mp4

Reported by: John Owned by:
Priority: normal Component: avformat
Version: git-master Keywords: hevc mov alpha
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary

I used NVIDIA's Video Codec SDK to create an HEVC bitstream containing an alpha layer. I tried to mux this bitstream into an MP4 container with no success. I was hoping this would work because:

  1. It was a copy, not a re-encode (no need to involve libx265)
  2. This change gave me hope: https://github.com/FFmpeg/FFmpeg/commit/ad326379c6634505c9ebc46964057441761008bc

No cigar. FFmpeg did create an MP4 file but the MP4 will not play in QuickTime and does not seem to retain the alpha layer. Whenever a nuh_layer_id > 0 is encountered FFmpeg outputs:
missing picture in access unit

I've attached the ffmpeg -v trace ... output to this request as well as the HEVC bitstream containing the alpha layer.

How to reproduce

% ffmpeg -i bsWithAlpha.265 -c:v copy -tag:v hvc1 outputWithAlpha.mp4

Attachments (2)

trace.txt (27.2 KB ) - added by John 3 years ago.
bsWithAlpha.265 (1.8 MB ) - added by John 3 years ago.
HEVC bitstream with alpha layer

Download all attachments as: .zip

Change History (11)

by John, 3 years ago

Attachment: trace.txt added

by John, 3 years ago

Attachment: bsWithAlpha.265 added

HEVC bitstream with alpha layer

comment:1 by John, 3 years ago

Version: unspecifiedgit-master

comment:2 by Carl Eugen Hoyos, 3 years ago

Component: undeterminedavformat
Keywords: mov added; h.265 transparency missing access removed
Type: enhancementdefect

Please test current FFmpeg git head, the only version supported on this bug tracker.

in reply to:  2 comment:3 by Balling, 3 years ago

It does not even ffplay correctly. What do you want from it? I am also not sure we support annex b to mp4 properly and fully, etc, etc. I mean there is a bug that blocks Mastering display luminance : min: 0.0000 cd/m2, max: 0 cd/m2 in -bsf:v hevc_metadata! It is a joke, okay. Sigh. https://github.com/FFmpeg/FFmpeg/blob/741565a1e69c45ce4848f01e45be5e66a68efa2f/libavcodec/cbs_h264_syntax_template.c#L816

P.S. Sorry ;) That was fixed by https://github.com/FFmpeg/FFmpeg/commit/4bf338480035238b3ff595ebd761688fc0db9e11#diff-fce57dfc99a19823f0294919d3086ae05d3bb6d170b613df3946753147b70d59L816

Yeah!

Last edited 3 years ago by Balling (previous) (diff)

in reply to:  2 comment:4 by John, 3 years ago

Replying to cehoyos:

Please test current FFmpeg git head, the only version supported on this bug tracker.

I tested again this morning with the same results

% date  
Tue Feb  2 08:24:29 PST 2021
% git pull
Already up to date.
% git remote -v
origin	https://git.ffmpeg.org/ffmpeg.git (fetch)
origin	https://git.ffmpeg.org/ffmpeg.git (push)

comment:5 by Balling, 3 years ago

Status: newopen

Related: https://forums.developer.nvidia.com/t/hevc-with-alpha-layer/167353
https://github.com/johnhe4/nvenc_h265_transparency

Please, next time provide all the links.

Should not this be closed as a duplicate of #7965?

Last edited 3 years ago by Balling (previous) (diff)

in reply to:  5 comment:6 by Carl Eugen Hoyos, 3 years ago

Replying to Balling:

Related: https://forums.developer.nvidia.com/t/hevc-with-alpha-layer/167353
https://github.com/johnhe4/nvenc_h265_transparency

Both look unrelated to this ticket.

Should not this be closed as a duplicate of #7965?

This ticket is about an issue with libavformat, the other ticket is libavcodec-only.

comment:7 by Balling, 3 years ago

Both look unrelated to this ticket.

Both (the code and nvidia comment) were done by the same user that reported this issue...

comment:8 by John, 3 years ago

Yes, those links are related but I don't think they affect the scope. Sorry for not posting.

No, this ticket is not a duplicate of #7965. Perhaps the best way to explain why is to detail my target use case.

I am looking to segment out video using a Linux instance in the cloud. This segmentation process will result in two layers: a foreground layer and a background layer. I want to stream the foreground layer over the Internet to an Apple device that will render this foreground layer over whatever background the app chooses.

I know of two traditional methods to accomplish this that make inefficient use of bandwidth and/or push complexity to the end device. Traditional method 1 is to send both layers as separate streams, or if possible, as separate tracks in a single stream. This requires the device to spin up two decoders, perform post and or pre-decode synchronization, and possibly deal with multiple-multiple bitrates. Traditional method 2 is to have the Linux server overwrite the background with a predefined chroma key. This requires a carefully chosen chroma key that will not unexpectedly occur in the foreground, and requires custom shaders on the end device.

Alpha support in HEVC seems (to me) a much better solution than the traditional methods, if it is supported.

This ticket is different than #7965 because it specifically asks for muxing support only. I would like FFmpeg to mux an existing HEVC bitstream which includes alpha (encoded with whatever, in this case NVidia but can be anything) into an MP4 (or TS file) that Apple can decode. From my limited understanding of FFmpeg, any changes relating to this ticket wouldn't need to touch the encoder or decoder but would instead touch muxers and parsers.

Here is a link from Apple that includes more information for HEVC with alpha, including the profile constraints on Apple devices: https://developer.apple.com/videos/play/wwdc2019/506/

I noticed some activity on #7965, that is great to see. I imagine this ask will grow as more non-Apple encoders begin to support HEVC with alpha.

comment:9 by quinkblack, 15 months ago

Resolution: worksforme
Status: openclosed

Remux works fine with samples generated from Apple's videotoolbox.

The reason to current issue is that nvenc alpha layer encoding is broken.
http://ffmpeg.org/pipermail/ffmpeg-devel/2022-December/305204.html

Nothing we can do until Nvidia fixed their SDK.

Note: See TracTickets for help on using tickets.