Support hevc NAL units 62 and 63

[Carl Eugen Hoyos]

​http://thread.gmane.org/gmane.comp.video.ffmpeg.user/63074

$ ffmpeg -i LG_2_DEMO_4K_O_H_1_Dolby\ Comparison_OLED.mkv
ffmpeg version N-80901-gfebc862 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 47.100 /  6. 47.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[hevc @ 0x2e00e00] Skipping NAL unit 63
    Last message repeated 8 times
[hevc @ 0x2e00e00] Skipping NAL unit 62
Input #0, mpegts, from 'LG_2_DEMO_4K_O_H_1_Dolby Comparison_OLED.mkv':
  Duration: 00:01:15.46, start: 3.944678, bitrate: 32672 kb/s
  Program 2
    Stream #0:0[0x21]: Video: hevc (Main 10) ([36][0][0][0] / 0x0024), yuv420p10le(tv), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 23.98 tbc
    Stream #0:1[0x23]: Audio: eac3 (EAC3 / 0x33434145), 48000 Hz, 5.1(side), fltp, 768 kb/s
At least one output file must be specified

[Carl Eugen Hoyos]

​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket5688/

[Hendrik]

The dolby vision enhancement layer doesn't actually use NAL units 63 and 62, it uses a special syntax that uses 0x7E01 and 0x7C01 as separator that makes the Dolby Extension Layer (EL) appear as unspecified/unused NAL units (62 and 63), but the actual NAL information starts after that extra header.

A more fitting description would be support for the Dolby Vision Enhancement Layer.

[Balling]

Replying to heleppkes:

The dolby vision enhancement layer doesn't actually use NAL units 63 and 62,

But have you read ​https://www.reddit.com/r/ffmpeg/comments/a32yv4/in_the_future_would_it_be_possible_to_remux_a/
​https://www.dolby.com/us/en/technologies/dolby-vision/dolby-vision-profiles-levels_v1.3.2.pdf

It is even more difficult, no?

Besides that can you comment on ​https://bitbucket.org/multicoreware/x265/commits/all?search=dolby+vision
They added support for dolby vision and hdr10+. What part of it is implemented and is "Dolby Vision Enhancement Layer" implemented?
Are you going to support it? And in NVENC support also added: ​https://github.com/rigaya/NVEnc/issues/119

Also I want to ask about this patch for ffmpeg ​https://patchwork.ffmpeg.org/patch/10708/
It has new MKTAG('v','d','e','p'), so I suppose it is important. And this one?​https://patchwork.ffmpeg.org/patch/11828/

"It's still writing to an AVBufferRef in set_side_data() that may or may
not be writable." And so what? Solve it and pull already.

[Balling]

Replying to cehoyos:

​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket5688/

In reddit discussion in previous comment there was an interesting fact, I quote: "The model used on Blu-ray goes a bit further. Instead of just metadata there’s actually a second 10 bit HEVC stream. This provides even higher quality." This IS actually true! Here is March 2018 Dolby Vision + Dolby Atmos Blu-ray disk, which has .m2ts files with two HEVC video streams inside!
magnet:?xt=urn:btih:1F89889E0642497F74B0C54C98E079E4A8303C65&dn=DOLBY_VISION_CES_March_2018

Video #1
ID                          : 4113 (0x1011)
Menu ID                     : 1 (0x1)
Format                      : HEVC
Format/Info                 : High Efficiency Video Coding
Format profile              : Main 10@L5.1@High
HDR format                  : SMPTE ST 2086, HDR10 compatible
Codec ID                    : 36
Duration                    : 3 min 7 s
Width                       : 3 840 pixels
Height                      : 2 160 pixels
Display aspect ratio        : 16:9
Frame rate                  : 23.976 (24000/1001) FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0 (Type 2)
Bit depth                   : 10 bits
Color range                 : Limited
Color primaries             : BT.2020
Transfer characteristics    : PQ
Matrix coefficients         : BT.2020 non-constant
Mastering display color pri : Display P3
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2
Maximum Content Light Level : 1000 cd/m2
Maximum Frame-Average Light : 471 cd/m2

Video #2
ID                          : 4117 (0x1015)
Menu ID                     : 1 (0x1)
Format                      : HEVC
Format/Info                 : High Efficiency Video Coding
Format profile              : Main 10@L5.1@High
HDR format                  : SMPTE ST 2086, HDR10 compatible
Codec ID                    : 36
Duration                    : 3 min 7 s
Width                       : 1 920 pixels
Height                      : 1 080 pixels
Display aspect ratio        : 16:9
Frame rate                  : 23.976 (24000/1001) FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0 (Type 2)
Bit depth                   : 10 bits
Color range                 : Limited
Color primaries             : BT.2020
Transfer characteristics    : PQ
Matrix coefficients         : BT.2020 non-constant
Mastering display color pri : Display P3
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2

Audio #1
ID                          : 4352 (0x1100)
Menu ID                     : 1 (0x1)
Format                      : AC-3 MLP FBA 16-ch
Format/Info                 : Audio Coding 3 + Meridian Lossless Packing FBA with 16-channel presentation
Commercial name             : Dolby TrueHD with Dolby Atmos
Muxing mode                 : Stream extension
Codec ID                    : 131
Duration                    : 3 min 7 s
Bit rate mode               : Variable
Bit rate                    : 640 kb/s
Maximum bit rate            : 9 621 kb/s
Channel(s)                  : 8 channels
Channel layout              : L R C LFE Ls Rs Lb Rb
Sampling rate               : 48.0 kHz
Frame rate                  : 31.250 FPS (1536 SPF)
Bit depth                   : 16 bits
Compression mode            : Lossy
Stream size                 : 14.3 MiB (1%)
Service kind                : Complete Main
Number of dynamic objects   : 15
Bed channel count           : 1 channel
Bed channel configuration   : LFE

This is very interesting. Can you or anybody else comment on how it is theoretically possible to decode it? I will try to upload one or two .m2ts files, so you can check it yourself. The second stream is really interesting, it is very gray, like a negative, but sometimes there are colors... Rather crasy one))

[Balling]

NAL 62 is the RPU, and NAL 63 is the EL... Yeah, very simple. (Profile 5 does not have EL, so no support for NAL 63, only NAL 62.)

[Balling]

RPU 62 is done here: bc68fd1050bd82e59d8ce7da909a0bcaf2b61197

Parsing can be extracted from patents and dovi_tool. I will also add a nice test pattern with the use of same hevc stream but differemt meta. ​https://drive.google.com/file/d/1-8C0ZYnIhEqOVJQ532hIJobP5XR6A7gj/view?usp=drivesdk

[Balling]

v4 of the patch that finally adds parsing for NLQ (that is still RPU in NAL 62) ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220806140235.144-2-tcChlisop0@gmail.com/

For NAL 63: ​https://github.com/quietvoid/vs-nlq (this can be used to merge EL and luma reshaped ( and in some cases chroma and/or MMR reshaped BL together)

[microchip]

When is this finally going to be fixed? You all sleeping or what?