Bitstream filter hevc_metadata fails with "bit_equal_to_one out of range" on Linux but works on Windows for hevc_nvenc

[Sonic-Y3k]

Summary of the bug:

When setting the hevc_metadata bitstream filter in combination with hevc_nvenc the following error message appears:

[AVBSFContext @ 0x558a46855680] bit_equal_to_one out of range: 0, but must be in [1,1].
[AVBSFContext @ 0x558a46855680] Failed to read unit 0 (type 39).
[AVBSFContext @ 0x558a46855680] Failed to read packet.
Error applying bitstream filters to an output packet for stream #0:0.

Strangely enough, if the same ffmpeg version is cross compiled for Windows it just works without any issues.

How to reproduce:

% ffmpeg -i bbb_sunflower_2160p_30fps_normal.mp4 -c:v hevc_nvenc -bsf:v "hevc_metadata=colour_primaries=9" -f hevc -y -t 00:00:10 /dev/null

ffmpeg N-97380-g55d830f69a
built on 17.04.2020

linux nvidia driver: 440.82
windows nvidia driver: 445.75

[Sonic-Y3k]

Linux Log-File

[Sonic-Y3k]

Windows Log-File

[Carl Eugen Hoyos]

You do realise that compressing the logs is a certain way to make understanding a ticket as difficult as possible?

Is encoding necessary to reproduce or can you provide an input file that leads to the same behaviour?

[Sonic-Y3k]

I think that encoding is necessary to reproduce this behavior. As an input file I used the big buck bunny example (​link). But I really experienced this behavior with every single file I tried.

If you want, I can upload a dd-ed sample to the ftp.

[Carl Eugen Hoyos]

So if you first encode the raw hevc file and then stream-copy it into another hevc file using the bitstream filter, the issue is not reproducible?

[Sonic-Y3k]

Example Input

[Sonic-Y3k]

Replying to cehoyos:

So if you first encode the raw hevc file and then stream-copy it into another hevc file using the bitstream filter, the issue is not reproducible?

If I first encode the file with

ffmpeg -i bbb_sunflower_2160p_30fps_normal.mp4 -c:v hevc_nvenc -f hevc -t 00:00:10 test_hevc.hevc

and afterwards execute the following:

ffmpeg -i test_hevc.hevc -c:v copy -bsf:v "hevc_metadata=colour_primaries=9" -f hevc test_out_hevc.hevc

the error occurs again:

[hevc_metadata @ 0x55620167e880] bit_equal_to_one out of range: 0, but must be in [1,1].
[hevc_metadata @ 0x55620167e880] Failed to read unit 3 (type 39).
[hevc_metadata @ 0x55620167e880] Failed to read packet.
Error applying bitstream filters to an output packet for stream #0:0.
[hevc_metadata @ 0x55620167e880] bit_equal_to_one out of range: 0, but must be in [1,1].
[hevc_metadata @ 0x55620167e880] Failed to read unit 3 (type 39).
[hevc_metadata @ 0x55620167e880] Failed to read packet.
Error applying bitstream filters to an output packet for stream #0:0.

So it is reproducible.

I attached the raw hevc stream to this ticket as "nvenc_encoded_hevc_stream.hevc"

Edit: With that encoded file, the error also occurs in windows.
Edit2: The NALs seem to have a different Length aswell:

[Sonic-Y3k]

nvenc_encoded_hevc_stream.hevc

[Sonic-Y3k]

Difference between Nal Units in Windows and Linux

[mkver]

The cbs_h265 code completely ignores the possibility of reserved_payload_extension_data that may be present at the end of every SEI message. The uploaded sample has 8 zero bits of reserved_payload_extension_data present.

(The current version of the spec says that reserved_payload_extension_data shall not be present in bitstreams conforming to this version of the specification, so its presence may very well be a bug somewhere. But even then cbs_h265 should not error out and has to treat this as potential reserved_payload_extension_data; right now this seriously impairs cbs_h265's forward compatibility.)

[Carl Eugen Hoyos]

Additionally, isn't it a fundamental issue that the bitstream filters do not know that containers exist which allow unreliable transmission and damage (like transport streams)?

[mkver]

The cbs-stuff is designed to be picky so that they indeed may drop whole packets when fed with invalid input (which may be a result of errors during transmission). So while it would be easily possible to relax this a bit (e.g. don't fail if an inessential unit like a SEI seems invalid; instead just drop the unit), it has therefore not been added to master.

[James]

The extra byte in the SEI message is most assuredly bogus, so it would probably be worth looking into why nvenc generated it.

Nonetheless, reserved_payload_extension_data parsing has been implemented in 3d51d3b42d0eee10301b06aaa814454c5e38eed6, so the sample should no longer fail with CBS based bitstream filters.