Support hevc NAL units 62 and 63

[Carl Eugen Hoyos]

​http://thread.gmane.org/gmane.comp.video.ffmpeg.user/63074

$ ffmpeg -i LG_2_DEMO_4K_O_H_1_Dolby\ Comparison_OLED.mkv
ffmpeg version N-80901-gfebc862 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 47.100 /  6. 47.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[hevc @ 0x2e00e00] Skipping NAL unit 63
    Last message repeated 8 times
[hevc @ 0x2e00e00] Skipping NAL unit 62
Input #0, mpegts, from 'LG_2_DEMO_4K_O_H_1_Dolby Comparison_OLED.mkv':
  Duration: 00:01:15.46, start: 3.944678, bitrate: 32672 kb/s
  Program 2
    Stream #0:0[0x21]: Video: hevc (Main 10) ([36][0][0][0] / 0x0024), yuv420p10le(tv), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 23.98 tbc
    Stream #0:1[0x23]: Audio: eac3 (EAC3 / 0x33434145), 48000 Hz, 5.1(side), fltp, 768 kb/s
At least one output file must be specified

[Carl Eugen Hoyos]

​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket5688/

[Hendrik]

The dolby vision enhancement layer doesn't actually use NAL units 63 and 62, it uses a special syntax that uses 0x7E01 and 0x7C01 as separator that makes the Dolby Extension Layer (EL) appear as unspecified/unused NAL units (62 and 63), but the actual NAL information starts after that extra header.

A more fitting description would be support for the Dolby Vision Enhancement Layer.

[Balling]

Replying to heleppkes:

The dolby vision enhancement layer doesn't actually use NAL units 63 and 62,

But have you read ​https://www.reddit.com/r/ffmpeg/comments/a32yv4/in_the_future_would_it_be_possible_to_remux_a/
​https://www.dolby.com/us/en/technologies/dolby-vision/dolby-vision-profiles-levels_v1.3.2.pdf

It is even more difficult, no?

Besides that can you comment on ​https://bitbucket.org/multicoreware/x265/commits/all?search=dolby+vision
They added support for dolby vision and hdr10+. What part of it is implemented and is "Dolby Vision Enhancement Layer" implemented?
Are you going to support it? And in NVENC support also added: ​https://github.com/rigaya/NVEnc/issues/119

Also I want to ask about this patch for ffmpeg ​https://patchwork.ffmpeg.org/patch/10708/
It has new MKTAG('v','d','e','p'), so I suppose it is important. And this one?​https://patchwork.ffmpeg.org/patch/11828/

"It's still writing to an AVBufferRef in set_side_data() that may or may
not be writable." And so what? Solve it and pull already.

[Balling]

Replying to cehoyos:

​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket5688/

In reddit discussion in previous comment there was an interesting fact, I quote: "The model used on Blu-ray goes a bit further. Instead of just metadata there’s actually a second 10 bit HEVC stream. This provides even higher quality." This IS actually true! Here is March 2018 Dolby Vision + Dolby Atmos Blu-ray disk, which has .m2ts files with two HEVC video streams inside!
magnet:?xt=urn:btih:1F89889E0642497F74B0C54C98E079E4A8303C65&dn=DOLBY_VISION_CES_March_2018

Video #1
ID                          : 4113 (0x1011)
Menu ID                     : 1 (0x1)
Format                      : HEVC
Format/Info                 : High Efficiency Video Coding
Format profile              : Main 10@L5.1@High
HDR format                  : SMPTE ST 2086, HDR10 compatible
Codec ID                    : 36
Duration                    : 3 min 7 s
Width                       : 3 840 pixels
Height                      : 2 160 pixels
Display aspect ratio        : 16:9
Frame rate                  : 23.976 (24000/1001) FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0 (Type 2)
Bit depth                   : 10 bits
Color range                 : Limited
Color primaries             : BT.2020
Transfer characteristics    : PQ
Matrix coefficients         : BT.2020 non-constant
Mastering display color pri : Display P3
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2
Maximum Content Light Level : 1000 cd/m2
Maximum Frame-Average Light : 471 cd/m2

Video #2
ID                          : 4117 (0x1015)
Menu ID                     : 1 (0x1)
Format                      : HEVC
Format/Info                 : High Efficiency Video Coding
Format profile              : Main 10@L5.1@High
HDR format                  : SMPTE ST 2086, HDR10 compatible
Codec ID                    : 36
Duration                    : 3 min 7 s
Width                       : 1 920 pixels
Height                      : 1 080 pixels
Display aspect ratio        : 16:9
Frame rate                  : 23.976 (24000/1001) FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0 (Type 2)
Bit depth                   : 10 bits
Color range                 : Limited
Color primaries             : BT.2020
Transfer characteristics    : PQ
Matrix coefficients         : BT.2020 non-constant
Mastering display color pri : Display P3
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2

Audio #1
ID                          : 4352 (0x1100)
Menu ID                     : 1 (0x1)
Format                      : AC-3 MLP FBA 16-ch
Format/Info                 : Audio Coding 3 + Meridian Lossless Packing FBA with 16-channel presentation
Commercial name             : Dolby TrueHD with Dolby Atmos
Muxing mode                 : Stream extension
Codec ID                    : 131
Duration                    : 3 min 7 s
Bit rate mode               : Variable
Bit rate                    : 640 kb/s
Maximum bit rate            : 9 621 kb/s
Channel(s)                  : 8 channels
Channel layout              : L R C LFE Ls Rs Lb Rb
Sampling rate               : 48.0 kHz
Frame rate                  : 31.250 FPS (1536 SPF)
Bit depth                   : 16 bits
Compression mode            : Lossy
Stream size                 : 14.3 MiB (1%)
Service kind                : Complete Main
Number of dynamic objects   : 15
Bed channel count           : 1 channel
Bed channel configuration   : LFE

This is very interesting. Can you or anybody else comment on how it is theoretically possible to decode it? I will try to upload one or two .m2ts files, so you can check it yourself. The second stream is really interesting, it is very gray, like a negative, but sometimes there are colors... Rather crasy one))

[Balling]

NAL 62 is the RPU, and NAL 63 is the EL... Yeah, very simple. (Profile 5 does not have EL, so no support for NAL 63, only NAL 62.)

[Balling]

RPU 62 is done here: bc68fd1050bd82e59d8ce7da909a0bcaf2b61197

Parsing can be extracted from patents and dovi_tool. I will also add a nice test pattern with the use of same hevc stream but different meta. ​https://drive.google.com/file/d/1-8C0ZYnIhEqOVJQ532hIJobP5XR6A7gj/view?usp=drivesdk

[Balling]

v4 of the patch that finally adds parsing for NLQ (that is still RPU in NAL 62) ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220806140235.144-2-tcChlisop0@gmail.com/

For NAL 63: ​https://github.com/quietvoid/vs-nlq (this can be used to merge EL and luma reshaped (and in some cases chroma and/or MMR reshaped BL together))

[microchip]

When is this finally going to be fixed? You all sleeping or what?

[Balling]

This is now fully supported with mpv support for EL. The merging on Dual layer support is here: ​https://github.com/mpv-player/mpv/pull/17932

Dynamic metadata is supported with --target-colorspace-hint-mode=source-dynamic, only artistic metadata is not yet supported in mpv. Dolby Vision 422p12 passthrough is also kinda implemented at least on some players it triggers Windows 11 OS Dolby Vision Extension overlay notification.

Also it is important to note that NAL62 can be stored wrongly because of some ancient bug in dovi_tool. This is now also fixed in 89bdd9e1a50735be7478ab8c07303f3d2fe59d85

[kasper93]

Replying to Balling:

Dynamic metadata is supported with --target-colorspace-hint-mode=source-dynamic

For future readers: dynamic metadata is supported in any target mode. Please read the documentation at ​https://mpv.io/manual/master/#options-target-colorspace-hint-mode to understand what this option does and avoid getting the wrong impression from the post above.

[Balling]

Artistic metadata is still not supported, see this sample in DV8 ​https://drive.google.com/file/d/1-8C0ZYnIhEqOVJQ532hIJobP5XR6A7gj/view

Should I open a new ticket for that? Quitvoid is aware of this though.

[kasper93]

Replying to Balling:

Artistic metadata is still not supported, see this sample in DV8 ​https://drive.google.com/file/d/1-8C0ZYnIhEqOVJQ532hIJobP5XR6A7gj/view

Should I open a new ticket for that? Quitvoid is aware of this though.

If you can provide relevant information. I'm not going to guess how all those trims should plug into tonemapper.

[Balling]

Yes, this was discussed here ​https://code.videolan.org/videolan/libplacebo/-/merge_requests/207#note_294152

Since then new RGB CMY trims were intrudced. I am kinda scared to imagine what was added in Dolvy Vision v2. LOL.