Colorspace support in FFmpeg

What is colorspace? Why should we care?

Colorspace describes how an array of pixel values should be displayed on the screen. It provides information like how pixel values are stored within a file, and what the range and meaning of those values are.

Basically, it tells the format of the array of pixels (e.g. RGB or YUV), and how the values of each color component should be translated, to be properly displayed by the photons of the screen. (i.e. picking a colorspace randomly seems unlikely to be a good choice...)

The differences between RGB and YUV:

• RGB distinguishes color pixel values into 3 components: Red, Green, Blue. (hence the name)
• YUV uses a different representation schema that represents color pixel values in: Luma (Y, or brightness), Chroma (U and V, two color differences). This format exploits the fact that humans are able to better notice details in luminance (approximated by luma) than in chroma. ^{(Note: YUV represents color in 3 components)}

  Note: The term "YUV" is ambiguous and often used wrongly, including the definition of pixel formats in FFmpeg. A more accurate term for how color is stored in digital video would be ​YCbCr. Y'UV on other other hand specifies a colorspace consisting of luma (Y') and chrominance (UV) components. For more info read ​the respective Wikipedia article. In the following, the term YUV is used as in the FFmpeg pixel formats, referring to YCbCr in digital video.

The conversion between YUV pixel buffer representation and its visual representation depends on the type of the YUV represented in the pixel buffers, which are essentially device-dependent. The accuracy of grey vs color separation depends on how well the transformation matrix matches the underlying RGB space.

Examples are:

• ​BT.601 ("Standard-Definition" or SD)
• ​BT.709 ("High-Definition" or HD)
• ​BT.2020 ("Ultra-High-Definition" or UHD)

These standards describe not just things like how to convert a YUV signal to RGB, but also how a RGB signal should be represented in terms of photon emission, in a device-independent way.

How does FFmpeg identify colorspaces?

In FFmpeg, colorspaces are represented in the form of ​pixel format.

In practical terms, the properties you care about are:

1. The signal format in the pixel buffer:
   1. the signal type: RGB, YUV, or other type, and
   2. the signal bit-depth.
2. For YUV signals, the color range: full/pc/jpeg or limited/restricted/tv/mpeg. For 8bpc, full means 0-255 and limited means 16-235.
3. The transformation matrix between YUV and RGB.
4. The linearization function from RGB to a linear RGB signal.
5. The conversion matrix between the linearized RGB and the device-independent XYZ colorspace.

FFmpeg stores these properties in the AVFrame struct:

• The format (type and bit-depth)->format
• The signal range->color_range
• The YUV/RGB transformation matrix->colorspace
• The linearization function (a.k.a. transformation characteristics)->color_trc
• The RGB/XYZ matrix->color_primaries

How to use FFmpeg to convert the colorspace

Pixel format conversion is often performed by libswscale, which you will most likely access through the FFmpeg Scaler or the scale video filter.

Other video filters include:

• colormatrix (native)
• colorspace (native)
• zscale (z.lib library)

Comparing and contrasting colorspace, colormatrix, libswscale, and zscale

• colorspace and colormatrix can only convert YUV to YUV
• libswscale can convert between YUV and RGB
• colormatrix only supports 8bpc (8-bit per component) pixel formats
• colorspace supports 8bpc, 10bpc, and 12bpc
• colormatrix does not apply gamma and primaries correction (i.e. which exact color "red", "green", and "blue" each is)
• colorspace applies gamma and primaries correction by default. (Use fast=1 to disable this for faster conversion or to make output compatible with output produced by colormatrix.)
• colormatrix is C only, whereas colorspace uses x86 SIMD so it's faster
• colormatrix produces horrible quality for 10bpc and 12bpc
• colorspace produces something decent for 10bpc
• For 8bpc, colorspace seems to produce slightly better quality than colormatrix
• zscale produces better results at all bit depths

colorspace and friends

The easiest way to use these filters is to ensure that the input AVFrames have all relevant struct members set to the appropriate values. On ffmpeg command-line this is what options such as colorspace and color_trc do; you can also put them on the output side to modify what space the output is in.

If everything goes well, libswscale figures out how to do the conversion and you can avoid calling the filter manually. However, AVFrames does not cover every option needed, so advanced use will go back to colorspace or zscale.

See also sws_setColorspaceDetails() and ​F.X.'s colorspace answer.

sRGB

There is no colorspace default called sRGB in FFmpeg, because it's identical to BT.709 except for a different transfer function. Just use one of the following:

• sRGB input, fast: -vf "colorspace=all=bt709:fast=1", aka -colorspace bt709 (on the input side!)
• sRGB input, accurate: -vf "colorspace=all=bt709:itrc=srgb:fast=0", aka -colorspace bt709 -color_trc srgb (on the input side!)

See also ​F.X.'s HDR answer and ​Gyan's sRGB output answer. If srgb does not work, try the more academic name iec61966-2-1.

Examples (Made using ​zeranoe 20190416)

The input source (rgb24):

colormatrix (yuv444p10le):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colormatrix_yuv444p10le.avi"
ffmpeg -i "colormatrix_yuv444p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colormatrix_yuv444p10le.png"

Output bit-identical as `colormatrix` (yuv444p).

colorspace (yuv444p10le):

Note: SSIM > 99.98% with The input source (rgb24). Should be 100% if the conversion algorithms are improved, though.

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p10le.avi"
ffmpeg -i "colorspace_yuv444p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p10le.png"

Reference (yuv444p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 5 -color_primaries 5 -color_trc 6 "yuv444p.avi"
ffmpeg -i "yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "yuv444p.png"

colormatrix (yuv444p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colormatrix_yuv444p.avi"
ffmpeg -i "colormatrix_yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colormatrix_yuv444p.png"

colorspace (yuv444p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p.avi"
ffmpeg -i "colorspace_yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p.png"

colorspace (yuv444p10le -> yuv444p):

ffmpeg -i "colorspace_yuv444p10le.avi" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p10le-yuv444p.avi"
ffmpeg -i "colorspace_yuv444p10le-yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p10le-yuv444p.png"

colorspace with fast=0 iall=bt601-6-625:

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_fast=0_iall=bt601-6-625.avi"
ffmpeg -i "colorspace_fast=0_iall=bt601-6-625.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_fast=0_iall=bt601-6-625.png"

colorspace with fast=0 iall=bt601-6-525:

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-525" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_fast=0_iall=bt601-6-525.avi"
ffmpeg -i "colorspace_fast=0_iall=bt601-6-525.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_fast=0_iall=bt601-6-525.png"

colorspace with fast=1 iall=bt601-6-525:

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-525:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_fast=1_iall=bt601-6-525.avi"
ffmpeg -i "colorspace_fast=1_iall=bt601-6-525.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_fast=1_iall=bt601-6-525.png"

Output bit-identical as `colorspace` (yuv444p10le).

Reference (yuv420p10le):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p10le -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 5 -color_primaries 5 -color_trc 6 "yuv420p10le.avi"
ffmpeg -i "yuv420p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "yuv420p10le.png"

colorspace (yuv420p10le):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv420p10le.avi"
ffmpeg -i "colorspace_yuv420p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv420p10le.png"

Reference (yuv420p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 5 -color_primaries 5 -color_trc 6 "yuv420p.avi"
ffmpeg -i "yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "yuv420p.png"

colorspace (yuv420p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv420p.avi"
ffmpeg -i "colorspace_yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv420p.png"

colorspace (yuv420p10le -> yuv420p):

ffmpeg -i "colorspace_yuv420p10le.avi" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv420p10le-yuv420p.avi"
ffmpeg -i "colorspace_yuv420p10le-yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv420p10le-yuv420p.png"

colorspace (yuv444p10le -> yuv420p):

ffmpeg -i "colorspace_yuv444p10le.avi" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p10le-yuv420p.avi"
ffmpeg -i "colorspace_yuv444p10le-yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p10le-yuv420p.png"