NVenc encodes broken? Now treble file size for exactly same commandline

[hydra3333]

Summary of the bug:

Something broke with NVenc h264 encoding, after commit 37140ebd87f549eae86a5b548d717a1e97203dd6.

With exactly the same comandlines, the file sizes are vastly different, as shown below.

Examples of the commandlines and their very different file sizes are shown below.

That is with nvidia driver version 445.87.

A later nvidia driver is available, 446.14, HOWEVER this only makes it worse and the file sizes are even greater.

I will source the filesizes under 446.14 and post them in a comment under this report.

How to reproduce:
I copied the parameters as they were used in the workflow, including the audio processing, for completeness.

"ffmpeg.exe" -loglevel warning -init_hw_device opencl=ocl:0.0 -filter_hw_device ocl -i "D:\TEMP\VRDTVS-SCRATCH\01-test-Channel2-HD.VRDTVS.qsf.mp4" -map_metadata -1 -vsync 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0.5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv420p" -strict experimental -c:v h264_nvenc -pix_fmt nv12 -preset slow -bf 2 -g 50  -rc:v vbr_hq -rc-lookahead:v 32 -cq 22 -qmin 16 -qmax 25 -coder 1 -movflags +faststart+write_colr -profile:v high -level 5.1 -af loudnorm=I=-16:TP=0.0:LRA=11:measured_I=-25.67:measured_LRA=9.00:measured_TP=-5.70:measured_thresh=-36.57:offset=0.07:linear=true:print_format=summary -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -strict experimental -movflags +faststart+write_colr -y "G:\HDTV\0-test-audio-sync-qsf5-vs-qsf6\01-ffmpeg_nvenc_MY_HOME_ENCODE_latest_ffmpeg.mp4" 
"ffmpeg-older-37140ebd87f549eae86a5b548d717a1e97203dd6.exe" -loglevel warning -init_hw_device opencl=ocl:0.0 -filter_hw_device ocl -i "D:\TEMP\VRDTVS-SCRATCH\01-test-Channel2-HD.VRDTVS.qsf.mp4" -map_metadata -1 -vsync 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0.5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv420p" -strict experimental -c:v h264_nvenc -pix_fmt nv12 -preset slow -bf 2 -g 50  -rc:v vbr_hq -rc-lookahead:v 32 -cq 22 -qmin 16 -qmax 25 -coder 1 -movflags +faststart+write_colr -profile:v high -level 5.1 -af loudnorm=I=-16:TP=0.0:LRA=11:measured_I=-25.67:measured_LRA=9.00:measured_TP=-5.70:measured_thresh=-36.57:offset=0.07:linear=true:print_format=summary -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -strict experimental -movflags +faststart+write_colr -y "G:\HDTV\0-test-audio-sync-qsf5-vs-qsf6\01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_37140ebd87f549eae86a5b548d717a1e97203dd6.mp4" 
05/06/2020  09:06 PM     5,072,015,090 01-ffmpeg_nvenc_MY_HOME_ENCODE_latest_ffmpeg.mp4
05/06/2020  09:15 PM     1,469,357,761 01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_37140ebd87f549eae86a5b548d717a1e97203dd6.mp4

"ffmpeg.exe" -loglevel warning -init_hw_device opencl=ocl:0.0 -filter_hw_device ocl -i "D:\TEMP\VRDTVS-SCRATCH\02-test-Channel44-SD.VRDTVS.qsf.mpg" -map_metadata -1 -vsync 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0.5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv420p" -strict experimental -c:v h264_nvenc -pix_fmt nv12 -preset slow -bf 2 -g 50  -rc:v vbr_hq -rc-lookahead:v 32 -cq 22 -qmin 16 -qmax 25 -coder 1 -movflags +faststart+write_colr -profile:v high -level 5.1 -af loudnorm=I=-16:TP=0.0:LRA=11:measured_I=-27.86:measured_LRA=9.60:measured_TP=-8.37:measured_thresh=-38.36:offset=0.09:linear=true:print_format=summary -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000  -y "G:\HDTV\0-test-audio-sync-qsf5-vs-qsf6\02-ffmpeg_MY_HOME_ENCODE_latest_ffmpeg.mp4" 
"ffmpeg-older-37140ebd87f549eae86a5b548d717a1e97203dd6.exe" -loglevel warning -init_hw_device opencl=ocl:0.0 -filter_hw_device ocl -i "D:\TEMP\VRDTVS-SCRATCH\02-test-Channel44-SD.VRDTVS.qsf.mpg" -map_metadata -1 -vsync 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0.5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv420p" -strict experimental -c:v h264_nvenc -pix_fmt nv12 -preset slow -bf 2 -g 50  -rc:v vbr_hq -rc-lookahead:v 32 -cq 22 -qmin 16 -qmax 25 -coder 1 -movflags +faststart+write_colr -profile:v high -level 5.1 -af loudnorm=I=-16:TP=0.0:LRA=11:measured_I=-27.86:measured_LRA=9.60:measured_TP=-8.37:measured_thresh=-38.36:offset=0.09:linear=true:print_format=summary -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000  -y "G:\HDTV\0-test-audio-sync-qsf5-vs-qsf6\02-ffmpeg_MY_HOME_ENCODE_ffmpeg_37140ebd87f549eae86a5b548d717a1e97203dd6.mp4" 
05/06/2020  09:25 PM     5,136,480,910 02-ffmpeg_MY_HOME_ENCODE_latest_ffmpeg.mp4
05/06/2020  09:35 PM     2,199,752,030 02-ffmpeg_MY_HOME_ENCODE_ffmpeg_37140ebd87f549eae86a5b548d717a1e97203dd6.mp4

"ffmpeg.exe" -loglevel warning -init_hw_device opencl=ocl:0.0 -filter_hw_device ocl -i "D:\TEMP\VRDTVS-SCRATCH\03-test-Channel9-HD-Rock.VRDTVS.qsf.mp4" -map_metadata -1 -vsync 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0.5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv420p" -strict experimental -c:v h264_nvenc -pix_fmt nv12 -preset slow -bf 2 -g 50  -rc:v vbr_hq -rc-lookahead:v 32 -cq 22 -qmin 16 -qmax 25 -coder 1 -movflags +faststart+write_colr -profile:v high -level 5.1 -af loudnorm=I=-16:TP=0.0:LRA=11:measured_I=-23.47:measured_LRA=8.70:measured_TP=-3.40:measured_thresh=-34.15:offset=0.52:linear=true:print_format=summary -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -strict experimental -movflags +faststart+write_colr -y "G:\HDTV\0-test-audio-sync-qsf5-vs-qsf6\03-ffmpeg_nvenc_MY_HOME_ENCODE_latest_ffmpeg.mp4" 
"ffmpeg-older-37140ebd87f549eae86a5b548d717a1e97203dd6.exe" -loglevel warning -init_hw_device opencl=ocl:0.0 -filter_hw_device ocl -i "D:\TEMP\VRDTVS-SCRATCH\03-test-Channel9-HD-Rock.VRDTVS.qsf.mp4" -map_metadata -1 -vsync 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0.5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv420p" -strict experimental -c:v h264_nvenc -pix_fmt nv12 -preset slow -bf 2 -g 50  -rc:v vbr_hq -rc-lookahead:v 32 -cq 22 -qmin 16 -qmax 25 -coder 1 -movflags +faststart+write_colr -profile:v high -level 5.1 -af loudnorm=I=-16:TP=0.0:LRA=11:measured_I=-23.47:measured_LRA=8.70:measured_TP=-3.40:measured_thresh=-34.15:offset=0.52:linear=true:print_format=summary -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -strict experimental -movflags +faststart+write_colr -y "G:\HDTV\0-test-audio-sync-qsf5-vs-qsf6\03-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_37140ebd87f549eae86a5b548d717a1e97203dd6.mp4" 
05/06/2020  09:53 PM     6,881,051,313 03-ffmpeg_nvenc_MY_HOME_ENCODE_latest_ffmpeg.mp4
05/06/2020  10:10 PM     2,100,748,069 03-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_37140ebd87f549eae86a5b548d717a1e97203dd6.mp4

"ffmpeg.exe" -version 
ffmpeg version git-2020-06-04-7f81785-Hydra3333/python_cross_compile_script_v100 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 11.0.0 (GCC) 20200517 (experimental)
configuration: --arch=x86_64 --target-os=mingw64 --cross-prefix=x86_64-w64-mingw32- --pkg-config=pkg-config --pkg-config-flags=--static --disable-shared --enable-static --disable-w32threads --enable-pthreads --enable-cross-compile --target-exec=wine --enable-runtime-cpudetect --enable-gpl --enable-version3 --extra-version=Hydra3333/python_cross_compile_script_v100 --enable-pic --enable-bzlib --enable-zlib --enable-lzma --enable-fontconfig --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libbluray --enable-libcdio --enable-avisynth --enable-vapoursynth --enable-librtmp --enable-libcaca --enable-iconv --enable-libxml2 --enable-gmp --enable-gnutls --enable-libzimg --enable-libx264 --enable-libx265 --enable-libvpx --enable-libdav1d --enable-libaom --enable-libxvid --enable-gray --enable-libopus --enable-libmp3lame --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libsoxr --enable-librubberband --enable-libass --enable-libtesseract --enable-libwebp --enable-ffnvcodec --enable-cuvid --enable-cuda-llvm --enable-opengl --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-libmfx --enable-amf --enable-opencl --enable-opengl --enable-vulkan --enable-filter=scale_vulkan --enable-filter=avgblur_vulkan --enable-filter=chromaber_vulkan --enable-filter=overlay_vulkan --enable-libtwolame --enable-libzvbi --enable-libgsm --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libsnappy --enable-frei0r --enable-filter=frei0r --enable-libsrt --enable-libbs2b --enable-libwavpack --enable-libilbc --enable-libgme --enable-libflite --enable-sdl2 --enable-libopenmpt --enable-libmysofa --enable-libvidstab --enable-libmodplug --disable-schannel --extra-libs='-lpsapi -lintl -liconv -lssp' --extra-cflags=-DLIBTWOLAME_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DLIBXML_STATIC --extra-cflags=-DGLIB_STATIC_COMPILATION --enable-nonfree --enable-libfdk-aac --enable-decklink --prefix=/home/u/Desktop/_working/workdir/win64_output/ffmpeg_git.installed --disable-shared --enable-static
libavutil      56. 49.100 / 56. 49.100
libavcodec     58. 90.100 / 58. 90.100
libavformat    58. 44.100 / 58. 44.100
libavdevice    58.  9.103 / 58.  9.103
libavfilter     7. 84.100 /  7. 84.100
libswscale      5.  6.101 /  5.  6.101
libswresample   3.  6.100 /  3.  6.100
libpostproc    55.  6.100 / 55.  6.100

"ffmpeg-older-37140ebd87f549eae86a5b548d717a1e97203dd6.exe" -version 
ffmpeg version N-97933-g37140ebd87-Hydra3333/python_cross_compile_script_v100 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 11.0.0 (GCC) 20200517 (experimental)
configuration: --arch=x86_64 --target-os=mingw64 --cross-prefix=x86_64-w64-mingw32- --pkg-config=pkg-config --pkg-config-flags=--static --disable-shared --enable-static --disable-w32threads --enable-pthreads --enable-cross-compile --target-exec=wine --enable-runtime-cpudetect --enable-gpl --enable-version3 --extra-version=Hydra3333/python_cross_compile_script_v100 --enable-pic --enable-bzlib --enable-zlib --enable-lzma --enable-fontconfig --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libbluray --enable-libcdio --enable-avisynth --enable-vapoursynth --enable-librtmp --enable-libcaca --enable-iconv --enable-libxml2 --enable-gmp --enable-gnutls --enable-libzimg --enable-libx264 --enable-libx265 --enable-libvpx --enable-libdav1d --enable-libaom --enable-libxvid --enable-gray --enable-libopus --enable-libmp3lame --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libsoxr --enable-librubberband --enable-libass --enable-libtesseract --enable-libwebp --enable-ffnvcodec --enable-cuvid --enable-cuda-llvm --enable-opengl --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-libmfx --enable-amf --enable-opencl --enable-opengl --enable-vulkan --enable-filter=scale_vulkan --enable-filter=avgblur_vulkan --enable-filter=chromaber_vulkan --enable-filter=overlay_vulkan --enable-libtwolame --enable-libzvbi --enable-libgsm --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libsnappy --enable-frei0r --enable-filter=frei0r --enable-libsrt --enable-libbs2b --enable-libwavpack --enable-libilbc --enable-libgme --enable-libflite --enable-sdl2 --enable-libopenmpt --enable-libmysofa --enable-libvidstab --enable-libmodplug --disable-schannel --extra-libs='-lpsapi -lintl -liconv -lssp' --extra-cflags=-DLIBTWOLAME_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DLIBXML_STATIC --extra-cflags=-DGLIB_STATIC_COMPILATION --enable-nonfree --enable-libfdk-aac --enable-decklink --prefix=/home/u/Desktop/_working/workdir/win64_output/ffmpeg_git.installed --disable-shared --enable-static
libavutil      56. 48.100 / 56. 48.100
libavcodec     58. 87.101 / 58. 87.101
libavformat    58. 43.100 / 58. 43.100
libavdevice    58.  9.103 / 58.  9.103
libavfilter     7. 83.100 /  7. 83.100
libswscale      5.  6.101 /  5.  6.101
libswresample   3.  6.100 /  3.  6.100
libpostproc    55.  6.100 / 55.  6.100

[hydra3333]

1. 37140ebd87f549eae86a5b548d717a1e97203dd6 11 days ago Andreas Rheinhardt avformat/id3v2: Remove unnecessary indirection

gives lower file sizes

2. c0f01eaf12479b5d0160e1a8d10a73744fac5ab2 8 days ago Andreas Rheinhardt avformat/oggdec: Avoid duplicating buffer when adding...

under nvidia driver 446.14 the result is:
gives lower file sizes

3. 7f76f20fa0a8aad6de50634a5cb60dc53117d6ab 3 days ago Jun Zhao checkasm: sw_rgb: Fix mixed declaration and code

under nvidia driver 446.14 the result is:
gives lower file sizes

4. 0842fd2c25c050e595083ba1e8c5176da3d55c9b 3 days ago Roman Arzumanyan avcodec/nvenc: zero avg and max bitrate in CQ mode

under nvidia driver 446.14 the result is:

5. 470bbf60d4cd34abd2cabea40910af6c0fc2366c 24 hours ago Roman Arzumanyan avcodec/nvenc: honor max bitrate in CQ mode

under nvidia driver 446.14 the result is:
HIGHER file size

6. b6c8444e2387d9a75b24d5ec1d8308fe6e37645b (MASTER) 46 min ago James Almer avutil/buffer: separate public and internal flags insid... master

under nvidia driver 446.14 the result is:
HIGHER file size

So, it broke between
7f76f20fa0a8aad6de50634a5cb60dc53117d6ab 3 days ago Jun Zhao checkasm: sw_rgb: Fix mixed declaration and code
and
470bbf60d4cd34abd2cabea40910af6c0fc2366c 24 hours ago Roman Arzumanyan avcodec/nvenc: honor max bitrate in CQ mode

So ... I suspect 0842fd2c25c050e595083ba1e8c5176da3d55c9b broke it first, as it did this

diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index cbdddd3..a9f9600 100644 (file)
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -947,12 +947,17 @@ static av_cold void nvenc_setup_rate_control(AVCodecContext *avctx)
     if (ctx->zerolatency)
         ctx->encode_config.rcParams.zeroReorderDelay = 1;
 
-    if (ctx->quality)
-    {
+    if (ctx->quality) {
         //convert from float to fixed point 8.8
         int tmp_quality = (int)(ctx->quality * 256.0f);
         ctx->encode_config.rcParams.targetQuality = (uint8_t)(tmp_quality >> 8);
         ctx->encode_config.rcParams.targetQualityLSB = (uint8_t)(tmp_quality & 0xff);
+
+        av_log(avctx, AV_LOG_VERBOSE, "CQ(%d) mode enabled.\n", tmp_quality);
+
+        //CQ mode shall discard max & avg bitrate;
+        avctx->bit_rate    = ctx->encode_config.rcParams.averageBitRate = 0;
+        avctx->rc_max_rate = ctx->encode_config.rcParams.maxBitRate     = 0;
     }
 }

Then 470bbf60d4cd34abd2cabea40910af6c0fc2366c did not fix the break, as it did this

diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 880454c..e269c71 100644 (file)
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -956,9 +956,9 @@ static av_cold void nvenc_setup_rate_control(AVCodecContext *avctx)
 
         av_log(avctx, AV_LOG_VERBOSE, "CQ(%d) mode enabled.\n", tmp_quality);
 
-        //CQ mode shall discard max & avg bitrate;
-        avctx->bit_rate    = ctx->encode_config.rcParams.averageBitRate = 0;
-        avctx->rc_max_rate = ctx->encode_config.rcParams.maxBitRate     = 0;
+        //CQ mode shall discard avg bitrate & honor max bitrate;
+        ctx->encode_config.rcParams.averageBitRate = avctx->bit_rate = 0;
+        ctx->encode_config.rcParams.maxBitRate = avctx->rc_max_rate;
     }
 }

File sizes produced with the various commits under nvidia driver 446.14 (in ascending order of commit) :

05/06/2020  11:08 PM     1,476,054,817 01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_37140ebd87f549eae86a5b548d717a1e97203dd6.mp4
06/06/2020  12:33 AM     1,476,054,817 01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_c0f01eaf12479b5d0160e1a8d10a73744fac5ab2.mp4
06/06/2020  01:13 AM     1,476,054,817 01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_7f76f20fa0a8aad6de50634a5cb60dc53117d6ab.mp4
06/06/2020  01:58 AM     5,109,921,859 01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_0842fd2c25c050e595083ba1e8c5176da3d55c9b.mp4
06/06/2020  01:25 AM     5,109,921,859 01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_470bbf60d4cd34abd2cabea40910af6c0fc2366c.mp4
06/06/2020  01:35 AM     5,109,921,859 01-ffmpeg_nvenc_MY_HOME_ENCODE_ffmpeg_b6c8444e2387d9a75b24d5ec1d8308fe6e37645b.mp4

Where to next ?

Separately to the above, tomorrow I hope to downgrade the nvidia driver and do a couple of commit comparisons.

[Timo R.]

Prior to the commit, the -cq option for nvenc was broken, because it got its max bitrate capped by the internal default bitrate of lavc.
It now ignores the set bitrate, as it should, when in -cq mode.
If you want to intentionally cap its bitrate, use the -maxrate option.

[hydra3333]

OK.

It is a very significant difference in file size, and unexpected for me at least.

I guess I'll have to experiment with -maxrate to find the right balance for my use cases.

By the way, for interest - with nothing else changed, updating to the latest nvidia driver 446.14 and using nvenc encoding adds some mb to the encoded file.

[Timo R.]

Before those fixes, you could literally change the -cq value from 30 to 10, and not get any difference in file size and quality, because the default bitrate capped it down to a quality even worse than 30.

So that behavior clearly was broken and did not work in any useful fashion.
Instead of experimenting with maxrate, you should experiment with the cq value itself, which now correctly works as intended.
Obviously, if you have a strict maxrate requirement, you can set that. But if you want vbr with a target quality in mind, there is no need for a maxrate.

[hydra3333]

OK. Thank you.