Colorspace support in FFmpeg

What is colorspace? Why should we care?

Colorspace describes how an array of pixel values should be displayed on the screen. It provides information like how pixel values are stored within a file, and what the range and meaning of those values are.

Basically, it tells the format of the array of pixels (e.g. RGB or YUV), and how the values of each color component should be translated, to be properly displayed by the photons of the screen. (i.e. picking a colorspace randomly seems unlikely a good choice...)


The difference between RGB and YUV should be rather obvious:

• RGB distinguishes color pixel values into 3 components: Red, Green, Blue. (hence the name)
• YUV uses a different representation schema that represents color pixel values in: Luma (Y, or brightness), Chroma (U and V, two color differences). This format exploits the fact that humans are able to better notice details in luminance (approximated by luma) than in chroma. ^{(Note: YUV represents color in 3 components)}

  Note: The term "YUV" is ambiguous and often used wrongly, including the definition of pixel formats in FFmpeg. A more accurate term for how color is stored in digital video would be ​YCbCr. Y'UV on other other hand specifies a colorspace consisting of luma (Y') and chrominance (UV) components. For more info read ​the respective Wikipedia article. In the following, the term YUV is used as in the FFmpeg pixel formats, referring to YCbCr in digital video.

The conversion between YUV pixel buffer representation and its visual representation depends on the type of the YUV represented in the pixel buffers, which are essentially device-dependent. The accuracy of grey vs color separation depends on how well the transformation matrix matches the underlying RGB space.

Examples are:

• ​BT.601 ("Standard-Definition" or SD)
• ​BT.709 ("High-Definition" or HD)
• ​BT.2020 ("Ultra-High-Definition" or UHD)

These standards describe not just things like how to convert a YUV signal to RGB, but also how a RGB signal should be represented in terms of photon emission, in a device-independent way.

How does FFmpeg identify colorspaces?

In FFmpeg, colorspaces are represented in the form of ​pixel format.

In practical terms, the things you care are:

1. Whether the pixel buffer contains RGB, YUV or some other type of signals, and the bit-depth.
2. Whether the signals are full range or restricted range. (YUV only, unlikely a problem for other type of signals...) ^{(Note: "Full range" refers to the YUV component values being in the range 0–255, whereas "restricted range" has values between 16–235)}
3. The transformation matrix between YUV and RGB.
4. The linearization function from RGB to a linear RGB signal.
5. The conversion matrix between the linearized RGB and the device-independent XYZ colorspace.

FFmpeg stores all these properties in the AVFrame struct:

• The format (type and bit-depth), in AVFrame->format
• The signal range, in AVFrame->color_range
• The YUV/RGB transformation matrix, in AVFrame->colorspace
• The linearization function (a.k.a. transformation characteristics), in AVFrame->color_trc
• The RGB/XYZ matrix, in AVFrame->color_primaries

How to convert between colorspaces using FFmpeg?

Conversion between RGB/YUV is typically done using swscale. Conversion between different color properties (bit-depth, range, matrix, transfer characteristics, primaries) can be done using the colorspace or colormatrix video filter. There's also a filter using the external library zscale. (for both aforementioned purposes) ^{(...and seems to be a more reliable choice for all these swscale hazards)}


Video filter colorspace, colormatrix have the following relationship:

• They both do only YUV to YUV colorspace conversion; YUV to RGB, and scaling requires swscale.
• colormatrix supports only 8bpc (8-bit per component) pixel formats, whereas colorspace supports 10bpc, 12bpc also.
• colormatrix does not apply gamma and primaries (i.e. which exact color "red", "green", and "blue" each is) correction, whereas colorspace does (it has an option fast=1 to disable this if you want faster conversion, or compatible output with that produced by colormatrix).
  Note: the examples are significantly worse with fast=0 than with fast=1. This is because screenshot sRGB is incorrectly specified as Bt. 601, while it should have been ispace=bt709:itrc=srgb. Don't repeat this mistake!
• colormatrix is C only, whereas colorspace uses x86 SIMD (i.e. it's faster).

Anyway the major difference between them is colormatrix produces horrible quality for anything > 8bpc (8-bit per component)... while colorspace produces something decent, at least for 10bpc (for 8bpc they both produce similar bad quality... probably due to improper design in the algorithms). ^{(floor instead of round on color approximation?..)}
Anyway for 8bpc... colorspace still seems to produce slightly better quality than colormatrix (while it's pointless... as doing things in 10bpc first, then 10bpc -> 8bpc seems to be a better approach... if you don't mind dithering). ^{(​dithering is enforced in swscale YUV 10bpc -> 8bpc)}

Read the filters' respective documentation to read up exactly on how to use them.

-colorspace and friends

The easiest way to use these filters is to ensure that the input AVFrames have all relevant struct members set to the appropriate values. On ffmpeg command-line this is what options such as -colorspace and color_trc do; you can also put them on the output side to modify what space the output is in.

If everything goes well, swscale figures out how to do the conversion and you can avoid calling the filter manually. However, AVFrames does not cover every option needed, so advanced use will go back to -vf colorspace or -vf zscale.

See also sws_setColorspaceDetails() and ​F.X.'s colorspace answer.

sRGB

There is no colorspace default called srgb in ffmpeg, because it's identical to Bt. 709 except for a different transfer function. Just use one of the following:

• sRGB input, fast: -vf "colorspace=all=bt709:fast=1", aka -colorspace bt709 (on the input side!)
• sRGB input, accurate: -vf "colorspace=all=bt709:itrc=srgb:fast=0", aka -colorspace bt709 -color_trc srgb (on the input side!)

See also ​F.X.'s HDR answer and ​Gyan's sRGB output answer. If srgb does not work, try the more academic name iec61966-2-1.

Examples ^{(Build from: ​https://zeranoe.com/builds/win64/static/ffmpeg-20190416-e2f766e-win64-static.zip)}

Caveat: For all these images to be properly displayed... some decent browser shall be a requisite.

The input source (rgb24):

colormatrix (yuv444p10le):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colormatrix_yuv444p10le.avi"
ffmpeg -i "colormatrix_yuv444p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colormatrix_yuv444p10le.png"

Output bit-identical as `colormatrix` (yuv444p).

colorspace (yuv444p10le): ^{(Note: SSIM > 99.98% with The input source (rgb24). Should be 100% if the conversion algorithms are improved though...)}

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p10le.avi"
ffmpeg -i "colorspace_yuv444p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p10le.png"

Reference (yuv444p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 5 -color_primaries 5 -color_trc 6 "yuv444p.avi"
ffmpeg -i "yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "yuv444p.png"

colormatrix (yuv444p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -vf "colormatrix=bt470bg:bt709" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colormatrix_yuv444p.avi"
ffmpeg -i "colormatrix_yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colormatrix_yuv444p.png"

colorspace (yuv444p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p.avi"
ffmpeg -i "colorspace_yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p.png"

colorspace (yuv444p10le -> yuv444p):

ffmpeg -i "colorspace_yuv444p10le.avi" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p10le-yuv444p.avi"
ffmpeg -i "colorspace_yuv444p10le-yuv444p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p10le-yuv444p.png"

colorspace with fast=0 iall=bt601-6-625:

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_fast=0_iall=bt601-6-625.avi"
ffmpeg -i "colorspace_fast=0_iall=bt601-6-625.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_fast=0_iall=bt601-6-625.png"

colorspace with fast=0 iall=bt601-6-525:

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-525" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_fast=0_iall=bt601-6-525.avi"
ffmpeg -i "colorspace_fast=0_iall=bt601-6-525.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_fast=0_iall=bt601-6-525.png"

colorspace with fast=1 iall=bt601-6-525:

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv444p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-525:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_fast=1_iall=bt601-6-525.avi"
ffmpeg -i "colorspace_fast=1_iall=bt601-6-525.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_fast=1_iall=bt601-6-525.png"

Output bit-identical as `colorspace` (yuv444p10le).

Reference (yuv420p10le):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p10le -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 5 -color_primaries 5 -color_trc 6 "yuv420p10le.avi"
ffmpeg -i "yuv420p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "yuv420p10le.png"

colorspace (yuv420p10le):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p10le -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv420p10le.avi"
ffmpeg -i "colorspace_yuv420p10le.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv420p10le.png"

Reference (yuv420p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 5 -color_primaries 5 -color_trc 6 "yuv420p.avi"
ffmpeg -i "yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "yuv420p.png"

colorspace (yuv420p):

ffmpeg -i "origin(rgb24).png" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -vf "colorspace=bt709:iall=bt601-6-625:fast=1" -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv420p.avi"
ffmpeg -i "colorspace_yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv420p.png"

colorspace (yuv420p10le -> yuv420p):

ffmpeg -i "colorspace_yuv420p10le.avi" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv420p10le-yuv420p.avi"
ffmpeg -i "colorspace_yuv420p10le-yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv420p10le-yuv420p.png"

colorspace (yuv444p10le -> yuv420p):

ffmpeg -i "colorspace_yuv444p10le.avi" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -pix_fmt yuv420p -sws_flags spline+accurate_rnd+full_chroma_int -color_range 1 -colorspace 1 -color_primaries 1 -color_trc 1 "colorspace_yuv444p10le-yuv420p.avi"
ffmpeg -i "colorspace_yuv444p10le-yuv420p.avi" -compression_level 10 -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int "colorspace_yuv444p10le-yuv420p.png"