Nvenc b_ref_mode middle does not write max_num_reorder_frames == 2 in SPS

[Balling]

Summary of the bug: see #6810, we can just add vui->bitstreamRestrictionFlag = 1; to ffmpeg code to write bitstream_restriction_flag 1 in SPS and that will allow Nvidia driver to set max_num_reorder_frames to 2, which is needed for -b_ref_mode middle and max_num_reorder_frames to 1 which is needed for b_ref_mode each.

Also see ​https://forums.developer.nvidia.com/t/sei-nal-unit-resend-each-packet-on-h264/145653/3

And people crying about it in ​https://github.com/bluenviron/mediamtx/issues/989#issuecomment-1163015224

How to reproduce:

PS > ffmpeg.exe -lavfi "ddagrab=framerate=240" -c:v h264_nvenc -b_ref_mode middle ddagrab.h264
ffmpeg version N-116525-gde7f999481-20240803 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14.1.0 (crosstool-NG 1.26.0.93_a87bf7f)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --enable-shared --disable-static --disable-w32threads --enable-pthreads --enable-iconv --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-libxml2 --enable-lzma --enable-fontconfig --enable-libharfbuzz --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --enable-libdavs2 --enable-libdvdread --enable-libdvdnav --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libaribcaption --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libvvenc --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-libs=-lgomp --extra-ldflags=-pthread --extra-ldexeflags= --cc=x86_64-w64-mingw32-gcc --cxx=x86_64-w64-mingw32-g++ --ar=x86_64-w64-mingw32-gcc-ar --ranlib=x86_64-w64-mingw32-gcc-ranlib --nm=x86_64-w64-mingw32-gcc-nm --extra-version=20240803
  libavutil      59. 31.100 / 59. 31.100
  libavcodec     61. 11.100 / 61. 11.100
  libavformat    61.  5.101 / 61.  5.101
  libavdevice    61.  2.100 / 61.  2.100
  libavfilter    10.  2.102 / 10.  2.102
  libswscale      8.  2.100 /  8.  2.100
  libswresample   5.  2.100 /  5.  2.100
  libpostproc    58.  2.100 / 58.  2.100
Stream mapping:
  ddagrab:default -> Stream #0:0 (h264_nvenc)
Press [q] to stop, [?] for help
Output #0, h264, to 'ddagrab.h264':
  Metadata:
    encoder         : Lavf61.5.101
  Stream #0:0: Video: h264 (Main), d3d11(pc, gbr/bt709/iec61966-2-1, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 2000 kb/s, 240 fps, 240 tbn
      Metadata:
        encoder         : Lavc61.11.100 h264_nvenc
      Side data:
        cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
[out#0/h264 @ 00000243bca73e80] video:1981KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.000000%
frame=  951 fps=113 q=42.0 Lsize=    1981KiB time=00:00:03.95 bitrate=4107.6kbits/s dup=945 drop=0 speed=0.468x
Exiting normally, received signal 2.

And then using VQAnalyzer you see that Annex B has wrong SPS. As you can see because flag is not set the motion_vectors_over_pic_boundaries_flag and log2_max_mv_length_horizontal and max_num_reorder_frames are not populated.

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Balling]

Another complaint from another person on nvidia server, IRC people can only troll and talk about me...

​https://forums.developer.nvidia.com/t/how-to-specify-encoder-sps-parameters-bitstream-restriction-flag-1/310564

[Balling]

So why does it write max_num_reorder_frames 2 for b_ref_mode each, while it should be max_num_reorder_frames 1?

[Timo R.]

You will probably have to forward that question to nvidia. I see no obvious way to influence that value via the nvenc API.

[Balling]

Maybe each is not actual each? I am not reading through dense H.264 spec to see whether it is correct.

[Timo Rothenpieler <timo@rothenpieler.org>]

In d724584d/ffmpeg:

avcodec/nvenc: set bitstreamRestrictionFlag when neccesary

This mimics the behaviour of libx264 for setting this flag.

Fixes #11131

[Timo Rothenpieler <timo@rothenpieler.org>]

In d724584d/ffmpeg:

avcodec/nvenc: set bitstreamRestrictionFlag when neccesary

This mimics the behaviour of libx264 for setting this flag.

Fixes #11131