Better warning for silently skipped color primaries meddling?

[fmnijk]

When using any NVENC encoder (av1_nvenc, hevc_nvenc, h264_nvenc) with zscale or scale filter to set color space parameters, the color space gets incorrectly set to bt470bg instead of bt709 when not explicitly specifying a pixel format.

## Steps to reproduce:

1. Use a 16bit EXR file with linear transfer characteristics
2. Use zscale or scale to explicitly set matrix/color space to bt709
3. Encode with any NVENC encoder without specifying pixel format

Note: I have provided an example EXR file rendered with Unreal Engine 5 as an attachment for reference and testing purposes.

## Expected behavior:
Output video should have color_space=bt709 as specified in the filter.

## Actual behavior:
Output video has color_space=bt470bg despite explicitly setting matrix/color space to bt709.

## Problem examples:With zscale:
ffmpeg -i input_%04d.exr -vf "zscale=matrix=bt709:transfer=iec61966-2-1" -c:v av1_nvenc output.mp4

With scale:
ffmpeg -i input_%04d.exr -vf "scale=out_color_matrix=bt709:out_transfer=iec61966-2-1" -c:v av1_nvenc output.mp4

Output properties for both cases:
"color_space": "bt470bg", # Should be bt709
"color_transfer": "iec61966-2-1"

## Working example (without NVENC):
ffmpeg -i input_%04d.exr -vf "zscale=matrix=bt709:transfer=iec61966-2-1" output.mp4

Output properties:
"color_space": "bt709",
"color_transfer": "iec61966-2-1"

## Workarounds:

1. Adding any format filter to the chain fixes the issue:

ffmpeg -i input_%04d.exr -vf "zscale=matrix=bt709:transfer=iec61966-2-1,format=yuv420p" -c:v av1_nvenc output.mp4

2. Explicitly specifying pixel format as a parameter also works:

ffmpeg -i input_%04d.exr -vf "zscale=matrix=bt709:transfer=iec61966-2-1" -pix_fmt yuv420p -c:v av1_nvenc output.mp4

Either way correctly outputs:
"color_space": "bt709",
"color_transfer": "iec61966-2-1"

## System Information:
ffmpeg version N-119165-gf3f1a48a07-20250407
OS: Windows 10 22H2
GPU: RTX 4090
NVIDIA Driver: NVIDIA Studio Driver 572.83

[Balling]

color space gets incorrectly set to bt470bg instead of bt709 when not explicitly specifying a pixel format.

Pixel format has nothing to do with colorspace/primaries/transfer. It is completly unrelated, well, besides xyz12 and rgb stuff, where colorspace is set to 0. E.g. pixel format P010 may be SDR 10 bit.

It is expected that BT.601 is default matrix in most situations. There is no bug here, you have to force bt709 matrix or bt2020-ncl matrix if you need it.

[fmnijk]

If it's not related to pixel format, why does the color space change when I add the pixel format parameter?

To demonstrate:

First command:
ffmpeg -y -i input_%04d.exr -vf "scale=out_color_matrix=bt709" -c:v av1_nvenc output.mp4
Result:
"color_space": "bt470bg", # Should be bt709

Second command:
ffmpeg -y -i input_%04d.exr -vf "scale=out_color_matrix=bt709,format=yuv420p" -c:v av1_nvenc output.mp4
Result:
"color_space": "bt709"

The only difference is adding format=yuv420p to the filter chain, which fixes the issue. This strongly suggests there is a bug in how NVENC encoders handle color space when no format filter is explicitly specified.

[MasterQuestionable]

͏    The input EXR seems to be of certain RGB colorspace ("gbrapf32le"?).
͏    The BT.709 etc. color handling applies only for YUV.

͏    Compare:
͏    ffmpeg -v debug -hide_banner -nostdin -nostats -f lavfi -i "sws_flags=+print_info; testsrc,scale=out_color_matrix=bt709" -vframes 1 -f null -
͏    ffmpeg -v debug -hide_banner -nostdin -nostats -f lavfi -i "sws_flags=+print_info; testsrc,scale=out_color_matrix=bt709,format=yuv420p" -vframes 1 -f null -
͏    ffmpeg -v debug -hide_banner -nostdin -nostats -f lavfi -i "sws_flags=+print_info; testsrc2,scale=out_color_matrix=bt709" -vframes 1 -f null -
͏    .
͏    https://ffmpeg.org/ffmpeg-filters.html#testsrc
͏    "testsrc2" defaults "yuv420p".

[fmnijk]

The EXR file is gbrapf16le.

Why when using libx264, ffmpeg automatically converts the input to YUV (yuv444p10le) and applies the correct color space (bt709), but when using NVENC, ffmpeg doesn't automatically perform this conversion? Is this an intentional difference in behavior between encoders, or should ffmpeg also handle this conversion automatically when using NVENC?

I noticed in the debug logs that with libx264, the scale filter converts from gbrapf16le to yuv444p10le with bt709 color space, while with NVENC without explicit format conversion, the output ends up with bt470bg color space despite specifying bt709.

[MasterQuestionable]

͏    Maybe the encoders after all picked their preferred color primaries, for the implicit conversion?
͏    (debug log should have these details; in particular with "-sws_flags +print_info" alike)

͏    BT.709 is something common (also possibly default).
͏    Try deliberately tagging something extraordinary?
͏    ​https://trac.ffmpeg.org/wiki/colorspace#color_primaries

[fmnijk]

Here's what I found after analyzing the debug logs with both libx264 and av1_nvenc encoders using BT.2020:

1. libx264 Encoder (Works Correctly):
   • The scale filter selects yuv444p10le pixel format: picking yuv444p10le out of 15 ref:gbrapf16le alpha:1
   • Properly applies BT.2020 color space: yuv444p10le(tv, bt2020nc/unknown/linear, progressive)
   • Output maintains correct color space: color_space: "bt2020nc"

2. av1_nvenc Encoder (Problem):
   • The scale filter selects rgba pixel format: picking rgba out of 17 ref:gbrapf16le alpha:1
   • No color space applied: rgba(pc, gbr/unknown/linear, progressive)
   • Final output has incorrect color space: color_space: "bt470bg"

[Timo R.]

The internal rgb->yuv converter of nvenc is incredibly simplistic, and does not support any color space stuff.
If you want proper color space handling for conversion between rgb->yuv, convert via other means.

[MasterQuestionable]

͏    Well... it appears NVENC was miraculously (erroneously? [1]) reckoned capable of RGB handling.
͏    Thus causing a RGB format being selected: which inapplicable for the color handling.
͏    (demonstrated before with "testsrc")

[ [1]
͏    Probably it indeed supports RGB input:
͏    Merely unable to properly handle anyhow...
͏    (botched internal conversion to YUV) ]

͏    ----

​https://github.com/search?type=commits&q=repo:FFmpeg/FFmpeg+nvenc+rgb&s=committer-date&o=asc
͏    @ 2016-09-07T16:44:10Z:
​https://github.com/FFmpeg/FFmpeg/commit/4aeb7a88ec69aff6490e7f94f5987f7525b7eab7
͏    "NVENC still encodes as YUV, but does the conversion internally: which brings some performance gains."
͏    .
͏    @ 2022-08-06T20:23:33Z:
​https://github.com/FFmpeg/FFmpeg/commit/b71069879010426ef8d7dce614086f8fb90eb0f3
͏    “Hardcode color characteristics for internal RGB2YUV conversion:
͏    NVENC does not appear to use these values as input for its built-in RGB to YUV conversion, but instead sets them on the output as-is.
͏    Testing indicates the input is expected to be sRGB, with the output ending up as limited range BT.470.”

[Balling]

Probably it indeed supports RGB input

Since rather recently yes. ​https://github.com/search?q=repo%3AFFmpeg%2FFFmpeg+nvenc+rgb&type=commits

Last commit hints about the issue here...

scale_cuda exists nowadays

[fmnijk]

If I specify color space conversion, should the filter automatically convert to YUV when color handling applies only for YUV, regardless of whether the encoder supports RGB as input?

In cases where color handling applies only for YUV:

1. If the encoder only supports RGB as input: Automatically convert to YUV -> apply color handling -> convert back to RGB -> send to encoder
2. If the encoder only supports YUV as input: Automatically convert to YUV -> apply color handling -> output YUV directly to encoder
3. If the encoder supports both RGB and YUV as input: Automatically convert to YUV -> apply color handling -> output YUV directly to encoder

[MasterQuestionable]

͏    Probably.
͏    But the whole color primaries thing actually shouldn't exist in the first place:
͏    That just pointlessly sophisticated things.

͏    Further reading:
͏    ​https://github.com/libjxl/libjxl/issues/4056#issuecomment-2607405171
͏    (on ICC profiles, but fundamentally no different)

͏    ----

͏    To summarize your logic:
͏    Simply force YUV conversion for any such invoked color meddling.
͏    (auto format conversion would choose that may best contain the input format, meanwhile supported by the codec)