av1_nvenc bugs

[Jacky Koning]

Summary of the bug:
According to ffmpeg -h encoder=av1_nvenc -level goes from 0 - 24 where 0 = 2 and 23 = 7.3.
So I tried using 7.3 and the following error was shown
[av1_nvenc @ 000002811fa86240] InitializeEncoder failed: invalid param (8): Unsupported Level.
How to reproduce:
% ffmpeg -f lavfi -i testsrc -c:v av1_nvenc -level 7.3 -pix_fmt yuv420p -f null -

BtbN > FFmpeg-Builds > ffmpeg-master-latest-win64-lgpl.zip > N-109120-g59b16355ec-20221117
build on 17-11-2022

That command actually showed a second bug because ffmpeg -f lavfi -i testsrc outputs yuv444p (or at least the encoder pretends it does, it actually says RGB24) and ffmpeg -h encoder=av1_nvenc says it supports yuv444p and yuv444p16le but it then goes
[av1_nvenc @ 00000176457a0c40] YUV444P not supported

I tried using -pix_fmt yuv444p and yuv444p16le but it also errors. So to be consistent I went over all listed Supported pixel formats and the following don't work.

yuv444p yuv444p16le gbrp gbrp16le (cuda d3d11 these are probably not supposed to be in there)

Then lastly I have a question. Does this support the dual encoders from the 4000 series in any way? I can't seem to find information on it.

[Jacky Koning]

Another thing I just remembered is -qp for h264_nvenc and hevc_nvenc goes from -1 to 51 and for av1_nvnec it goes from -1 to 255. (The -cq is 0 to 51 on all of them)

You could pull these together so it makes sense by deviding the second one by 5.

[James]

Regarding the second part, it's not a bug. The nvenc wrapper supports yuv444p as input but not necessarily for all encoders and not necessarily with your GPU. If you get "YUV444P not supported" it's because your GPU returned an error when queried about NV_ENC_CAPS_SUPPORT_YUV444_ENCODE for the current encoder (av1). It might work with h264 or hevc.

testsrc generates rgb24, but since nvenc doesn't support it, a scaler is autoinserted to convert it to the closest supported format, in this case yuv444p, which is then rejected by your GPU. You're better passing the encoder the format you want and know it supports instead of relying on autoinserted scalers somewhere in the filterchain. For AV1, use cuda (decoders using nvdec/cuda hwaccel output it), d3d11 (same for d3d11va hwaccel decoders), or yuv420p. With your command line example above, "testsrc,format=yuv420p" will do it.

[Balling]

but since nvenc doesn't support it, a scaler is autoinserted to convert it to the closest supported format, in this case yuv444p

That is wrong, gbrp is closest.

[Jacky Koning]

It is not converting the format at all

Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> av1 (av1_nvenc))
Press [q] to stop, [?] for help
[av1_nvenc @ 0000029855600d40] YUV444P not supported
[av1_nvenc @ 0000029855600d40] No capable devices found
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

Look at hevc_nvenc

Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> hevc (hevc_nvenc))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf59.34.101
  Stream #0:0: Video: hevc (Rext), gbrp(pc, gbr/unknown/unknown, progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 2000 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc59.52.102 hevc_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A

In that one it indeed auto converts to gbrp. But I still think 'this' is a bug.
We are clearly able to use yuv444p with hevc_nvenc so why not with av1_nvnec? Like why is it in the list when there is only 1 supported encoder which is included in every 4000 series GPU when it does not support it? In my opinion either it should not be in the list and then be converted to the next best or it should work.

Which could lead to another discussion point that might throw that argument out of the window. h264_nvenc does not support yuv420p10le either and it will also error out.

Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
[h264_nvenc @ 000002aa47630dc0] 10 bit encode not supported
[h264_nvenc @ 000002aa47630dc0] No capable devices found
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

But even h264_nvenc supports yuv444p. yuv444p is usefull for lossless converting to apply several video filters after which you convert it back.