NVENC: Warn explicitly when specific bit rate is out of range

[alexpigment]

Summary of the enhancement:

Give an informative error when the user specifies a bit rate that is out of the encoder's range. NVENC appears to cap out at 144Mbps for progressive encoding and 37.5Mbps for interlaced encoding.

Currently, there is a generic "InitializeEncoder failed: invalid param (8)" message that is given. This can cause (and has caused, in my case) the user to waste a considerable amount of time trying to change various parameters in the command line when the actual failure is the result of an invisible bit rate ceiling (which is especially low for interlaced).

How to reproduce:

Progressive FAILURE:

% ffmpeg -f lavfi -i smptebars -c:v h264_nvenc -s 1920x1080 -aspect 16:9 -profile:v high -r 29.97 -t 5 -rc cbr_hq -b:v 144.1M 1080p.mp4

Progressive SUCCESS:

% ffmpeg -f lavfi -i smptebars -c:v h264_nvenc -s 1920x1080 -aspect 16:9 -profile:v high -r 29.97 -t 5 -rc cbr_hq -b:v 144M 1080p.mp4

Interlaced FAILURE:

% ffmpeg -f lavfi -i smptebars -c:v h264_nvenc -s 1920x1080 -aspect 16:9 -profile:v high -r 29.97 -t 5 -rc cbr_hq -flags ildct -b:v 37.6M 1080i.mp4

Interlaced SUCCESS:

% ffmpeg -f lavfi -i smptebars -c:v h264_nvenc -s 1920x1080 -aspect 16:9 -profile:v high -r 29.97 -t 5 -rc cbr_hq -flags ildct -b:v 37.6M 1080i.mp4

ffmpeg version N-86330-gbd1179e
built on 2017-06-01

[alexpigment]

There is a typo in the code under "Interlaced SUCCESS". The bitrate used should be -b:v 37.5M , otherwise it will fail in the same way as the Failure case above it.

[alexpigment]

Well, I assumed the reproduction steps would lead to the error message listed in the description, but I guess that's incorrect. Providing console dump.

C:\>ffmpeg.exe -f lavfi -i smptebars -c:v h264_nvenc -s 1920x1080 -aspect 16:9 -
profile:v high -r 29.97 -t 5 -rc cbr_hq -b:v 144.1M 1080p.mp4
ffmpeg version N-89343-g83ecdc9a92 Copyright (c) 2000-2017 the FFmpeg developers

  built with gcc 7.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --e
nable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libblur
ay --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopu
s --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --ena
ble-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-lib
x264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-z
lib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-c
uvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-l
ibmfx
  libavutil      56.  4.100 / 56.  4.100
  libavcodec     58.  6.102 / 58.  6.102
  libavformat    58.  2.103 / 58.  2.103
  libavdevice    58.  0.100 / 58.  0.100
  libavfilter     7.  5.100 /  7.  5.100
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, lavfi, from 'smptebars':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1
DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
[h264_nvenc @ 000000000061ffe0] InitializeEncoder failed: invalid param (8)
Error initializing output stream 0:0 -- Error while opening encoder for output s
tream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!