SWSCALE: in_range, in_color_matrix, out_range, out_color_matrix incorrect

[troy_s]

Summary of the bug:
Seems that swscale is still sadly mangling RGB to YCbCr and vice versa transforms. Disabled all ASM speedups to avoid code path issues, as were previously hard coded to 601 coefficents etc.

How to reproduce:
Encode a test chart and check values. Test chart available from ​http://www.belle-nuit.com/test-chart.

Expected versus Results:
BT.709 and sRGB share identical luminance coefficients. An sRGB image, encoded at full range using BT.709 primaries should decode to a perfect 1:1 regarding RGB values, compression notwithstanding.

Sampling the upper row of colors using the provided test chart, we can see some issues with the encoding / decoding step, when using FFPLAY with the corresponding scale commands.

For example, an encode using

./ffmpeg -loop 1 -i ~/Downloads/testchart.tif -t 3 -c:v h264 -vf "scale=in_range=full:in_color_matrix=bt709:out_range=full:out_color_matrix=bt709" out709-full.mp4

Should produce a perfect 1:1 when played back with

./ffplay -vf "scale=in_range=full:in_color_matrix=bt709:out_range=full:out_color_matrix=bt709" out709-full.mp4

When we sample the sRGB / 709 values using an image editor, the top row should result in RGB values as follows:

Name      R   G   B
Yellow:   180 180 16
Cyan:     16  180 180
Magenta:  180 16  180
Green:    16  180 16
Red:      180 16  16
Blue:     16  16  180

Given that 709 and sRGB share identical primaries, and broadcast scaling is set to full range, there should be a perfect 1:1 result with the values. However, what we find with playback via the ffmpeg command set with proper values is:

Name      R   G   B
Yellow:   179 190 22
Cyan:     31  197 180
Magenta:  167 0   176
Green:    30  209 21
Red:      166 0   19
Blue:     17  5   173

There appears to be breakage with regard to the application of the coefficients, as well as possibly misinterpretation of the full range settings.

The are some transforms that yield vaguely close values. However, these suffer from quantization issues where no such should occur. It would seem that all transformations are likely being performed at the relative bit depth for performance. Sadly, this corrupts the values quite significantly.

./ffmpeg -loop 1 -i ~/Downloads/testchart.tif -t 3 -c:v h264 -vf "scale=in_range=full:in_color_matrix=bt601:out_range=full:out_color_matrix=bt601" out601-full.mp4

The following, despite being 601 coefficients, should yield perfect 1:1 with RGB values (RGB being a relative color space) but quantization issues again result.

./ffplay -vf "scale=in_range=full:in_color_matrix=bt601:out_range=full:out_color_matrix=bt601" out601-full.mp4

Further testing of the in_range and out_range specifics will be conducted when the basic color coefficients issue is corrected.

Conclusion:
Some mangling is still present in the color handling in FFMPEG. Quantization issues can likely be expected until a float (likely non-realtime) option is introduced for higher quality frame by frame encoding / decoding.

However, even with quantization issues, the encoding / decoding coefficients are not responding / respected in correct fashion for 709 to 709 (1:1 with sRGB content).

[troy_s]

Tested yuv444p10be and yuv444p16be and the quantization issues still arise. Could be attributed to a rounding error due to bitshifting?

[troy_s]

On advice of michaelni, using the scale option to set range and color coefficients is not updating the flags in the stream:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from './out709-full.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.48.101
  Duration: 00:00:03.00, start: 0.000000, bitrate: 168 kb/s
    Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 1920x1080 [SAR 1:1 DAR 16:9], 163 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler

Setting the flags manually via -colorspace bt709 and -color_range jpeg results in the correct flags being set in the stream:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from './out709-full.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.48.101
  Duration: 00:00:03.00, start: 0.000000, bitrate: 168 kb/s
    Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuvj444p(pc, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 163 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler

[Carl Eugen Hoyos]

Aren't you testing the h264 encoder in your command lines?
Why don't you output to raw yuv files and inspect the values?
Please provide a command line that allows to reproduce the issue together with the complete, uncut console output to make this a valid ticket.

[troy_s]

I will likely need a little help to assert that the output I'll be getting is precisely what we are discussing. I have used the following ffmpeg command to dump the raw YCbCr to a planar file:

./ffmpeg -i ~/Downloads/testchart.tif -pix_fmt yuv444p -vf scale="in_range=full:in_color_matrix=bt709:out_range=full:out_color_matrix=bt709" out.yuv

I am hoping that, given 444 encoding:

1. Three planes of Y, Cb, and Cr respectively at the test chart's 1920x1080
   1. Y at 1920x1080 with full JFIF / JPEG range.
   2. Cb / Cr (again with yuv444p) at 1920x1080 with full JFIF / JPEG range.
2. Encoded using ITU-REC-709 luminance coefficients.

Can we assert that the given line will result in the above?

ffmpeg version N-64740-g6be71e9 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 18 2014 15:56:09 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
  configuration: --enable-gpl --enable-libx264 --disable-asm
  libavutil      52. 92.101 / 52. 92.101
  libavcodec     55. 69.100 / 55. 69.100
  libavformat    55. 48.101 / 55. 48.101
  libavdevice    55. 13.102 / 55. 13.102
  libavfilter     4. 11.102 /  4. 11.102
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from '/home/aphorism/Downloads/testchart.tif':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: tiff, rgb24, 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
File 'out.yuv' already exists. Overwrite ? [y/N] y
Output #0, rawvideo, to 'out.yuv':
  Metadata:
    encoder         : Lavf55.48.101
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc55.69.100 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (tiff (native) -> rawvideo (native))
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=    6075kB time=00:00:00.04 bitrate=1244160.0kbits/s    
video:6075kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

If we can agree on the encoding command to generate the correct YCbCr output, I can proceed from here.

Thanks for all of your help in all of this. Hopefully we can dig through some of the color issues in FFMPEG and generate test units.

[troy_s]

Analysing the resultant output image as a raw RGB image for statistical purposes, based on a simple renaming of the planar file to avoid YUV conversion in ImageMagick:

aphorism@u64x6:~/Develop/ffmpeg$ identify -verbose  -size 1920x1080 -depth 8 -interlace plane rgb:./out.raw
Image: ./out.raw
  Base filename: out.raw
  Format: RGB (Raw red, green, and blue samples)
  Class: DirectClass
  Geometry: 1920x1080+0+0
  Resolution: 72x72
  Print size: 26.6667x15
  Units: Undefined
  Type: TrueColor
  Endianess: LSB
  Colorspace: sRGB
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
  Channel statistics:
    Red:
      min: 16 (0.0627451)
      max: 235 (0.921569)
      mean: 104.958 (0.411601)
      standard deviation: 72.5761 (0.284612)
      kurtosis: -1.32201
      skewness: 0.447203
    Green:
      min: 32 (0.12549)
      max: 240 (0.941176)
      mean: 130.152 (0.510402)
      standard deviation: 37.936 (0.148768)
      kurtosis: 0.516604
      skewness: 0.17742
    Blue:
      min: 0 (0)
      max: 255 (1)
      mean: 1.00469 (0.00393995)
      standard deviation: 12.6817 (0.0497322)
      kurtosis: 251.198
      skewness: 15.0671
  Image statistics:
    Overall:
      min: 0 (0)
      max: 255 (1)
      mean: 78.7052 (0.308648)
      standard deviation: 47.8444 (0.187625)
      kurtosis: 7.50915
      skewness: 1.50966

Assuming the above command generates raw YCbCr in agreement with our estimations, a few things pop out:

1. The luma (Y' == R channel here) is still in broadcast range.
2. The Cb is odd. Given that we have a blue gradient going from 0-255, as well as pure white and pure black swatches, we should expect a full range of Cb along the yellow / blue axis. Instead, we are seeing something that looks very close to broadcast range (16-240 8 bit) with a doubled up offset to 32.
3. The Cr is delivering correct range levels at a cursory glance of the distribution numbers.

[troy_s]

Attached a (hopefully) accurate EXR with the YCbCr values based off of 709 coefficients.

Attached an ODS of the matrix formula used to convert the sRGB (shared primaries with 709) test chart. Notes on values obtained via the formulas revealed by selecting a cell:

• The matrix scales the luma via the coefficients for sRGB / 709 primaries as per XYZ positions. The luminance (Y) values for sRGB / 709 primaries for red, green, and blue are 0.2126, 0.7152, and 0.0722 respectively[1]
• The matrix scales for broadcast range of luma (Y') and Cb / Cr based on 219/255 for luma, and 224/255 respectively.
• The offsets are give to correctly position the values for Y, Cb, and Cr. These offsets are 16, 128, and 128 respectively.

[1] We should revisit the need for unique coefficients to cover variable color spaces, including the need for uniques for input color spaces alone. This extends into the forthcoming ITU-BT-2020 needs that have XYZ absolute space models.

[Michael Niedermayer]

Fixed the full range issue in b53bdae11f1eceea1a2e25a98aee81e1d1954e14
is there a problem remaining ? (its not easy to tell from the floats and tables, we need 8bit per sample reference output because thats what comes out)

[Michael Niedermayer]

a raw image that can be compared with tests/tiny_psnr would be quite usefull for example

[troy_s]

Just performed a git pull and tested the raw YCbCr output again. Looks better, but still not quite there given that there are channels that use the whole range of values.

Working on a better test case, but until then, here's the result which again shows error in the Cb plane from what I can see:

Channel statistics:
    Red:
      min: 0 (0)
      max: 255 (1)
      mean: 103.415 (0.40555)
      standard deviation: 84.4958 (0.331356)
      kurtosis: -1.32109
      skewness: 0.447833
    Green:
      min: 18 (0.0705882)
      max: 255 (1)
      mean: 130.343 (0.51115)
      standard deviation: 43.1051 (0.169039)
      kurtosis: 0.529155
      skewness: 0.18529
    Blue:
      min: 0 (0)
      max: 255 (1)
      mean: 0.940625 (0.00368873)
      standard deviation: 13.2052 (0.051785)
      kurtosis: 269.06
      skewness: 15.9677

Cb showing 18 is close to broadcast range, but still off. No idea what might be causing it, but it suggests more than pure broadcast scaling.

[troy_s]

Ridiculously stupid question likely, but how does one use tiny_psnr? It doesn't appear built by default, nor do I have any clue on how to use it.

[Carl Eugen Hoyos]

$ make tests/tiny_psnr

[troy_s]

ODS spreadsheet of the matrix values used to obtain the EXR result off of the sRGB testchart.

[troy_s]

32 bit float EXR of broadcast scaled 709 values using the matrix and offsets included in adjacent attachment.

[troy_s]

Correction on my part, due to the test pattern selected, the ranges are not going to span the entire gamut. A better test chart is being researched for this purpose.

[troy_s]

8 bit conversion of the 32 bit EXR in PNG format.

[troy_s]

A LibreOffice Calc spreadsheet to calculate YCbCr from RGB values for REC.709, REC.601, and SMPTE240M coerfficients.

[troy_s]

Added a YCbCr calculator for testing raw values. Coefficients and formulas referenced from Poynton and company.

[troy_s]

Can someone instruct me as to how tiny_psnr is intended to be used with a few sample test cases? Thank you.

[Andy Furniss]

Well I fell at the first hurdle trying to follow this - that being that the testchart.tif are just strange.

They have bars labelled 100% but their RGB values as shown by display (image magick) are eg, 16, 16, 235 - I can't see that makes any sense in an RGB tiff.

Other things - your ffplay command converts to 420 then it (maybe sdl) will stretch the yuv to full range rgb on display.

You can't expect to convert rgb to yuv and back without loss - I think doing this will be the best way to test Issues that may exist as you should still be able to see levels Poynton -

​http://www.poynton.com/PDFs/Merging_RGB_and_422.pdf

[troy_s]

In the interest of clarity, I am going to close this ticket and break down issues on a particular case by case basis.

The case by cases thus far registered are:

SWSCALE: Incorrect Values in Full Range Conversion RGB to YCbCr
​https://trac.ffmpeg.org/ticket/3801

SWSCALE: Quantization Errors in Y Studio Range
​https://trac.ffmpeg.org/ticket/3794

The spreadsheets here are still useful for anyone looking to test theoretical values.

[Balling]

BT.709 and sRGB share identical luminance coefficients

sRGB does not have luminance coefficients at all, since it is RGB. sYCC does. And no, sYCC uses BT.601 matrix. Same about Adobe RGB (opRGB) and Adobe YCC (opYCC) (BT.601 matrix too) and CMYK vs. YCCK (again, BT.601). It is bad, since matrix is derived from primaries, but all JPEGs are like that. What can I say.

An sRGB image, encoded at full range using BT.709 primaries should decode to a perfect 1:1 regarding RGB values

It cannot be encoded not with BT.709 primaries, sRGB uses only BT.709 primaries. And no. Reference EOTF for SDR video is different from sRGB, so they will not really decode 1:1.

editor, the top row should result in RGB values as follows:

That is limited RGB. All color pickers use full RGB. See ​https://en.wikipedia.org/wiki/SMPTE_color_bars

There appears to be breakage with regard to the application of the coefficients, as well as possibly misinterpretation of the full range settings.

No there is not. It is just you that did not tag the output files and it decodes using BT.601. See ​https://res18h39.netlify.app/color

Use yellow limited range: 180 180 16, after encoding using BT.709 matrix and rounding is 168, 44, 136 which if you are decoding back with BT.601 will be 179, 191, 22, and here it is! FFmpeg thus is perfect and case closed.

[troy_s]

God your stupidity is painful.

You realize what luminance coefficients are and that they apply to every single RGB encoding colourspace?

I give up. Clown fucking car.

[Balling]

Replying to troy_s:

You realize what luminance coefficients are and that they apply to every single RGB encoding colourspace?

I give up. Clown fucking car.

No, they only apply to YCbCr encoding. And it is luma (not luminance) coefficients, since it is after nonlinear transfer function. We are not encoding for russian SECAM IV that encoded on linear RGB, are we? LOL.

Again, your commands are wrong. You should have added

-color_primaries 1 -color_trc 1 -colorspace 1

to mark it as bt.709/bt.709/bt.709 primaries/transfer/matrix. If you did not you 100% do not know what you are doing. Since this is 101 of encoding videos.

Your issue with full range may be a real one though. It is just hard to test.

[troy_s]

You realize that I actually chatted with M on some of this and he actually fixed some of it?

Climb the fuck down off your motherfucking high idiot tree.

No, they only apply to YCbCr encoding.

No, asshat.

This is embarrassing for the FFMPEG project.

[Balling]

Replying to troy_s:

You realize that I actually chatted with M on some of this and he actually fixed some of it?

Climb the fuck down off your motherfucking high idiot tree.

No, they only apply to YCbCr encoding.

No, asshat.

This is embarrassing for the FFMPEG project.

Yes, it is. I do not work for/on ffmpeg though. I work on Chromium/Chrome. And I never chatted with M or whoever, since I was not on a project 7 years ago. Also, matrix coefficents is more common name for that.

[Balling]

Again, yuv files do not have any metadata and thus ffmpeg defaults to BT.601. The values you reported are correct for BT.601 decoding. If you wanted BT.709 yuv decoding you need to force it by very simple

-vf scale=in_color_matrix=bt709

[troy_s]

Great. So FFMPEG now has someone checking bugs who works on one of the most fucked up browsers on the planet with respect to pixel management.

It all makes sense now.

So here’s a tip; every single RGB colourspace has a set of luminance coefficients given by the middle row of the respective normalized primaries matrix. Guess what that row corresponds to? That’s right, the Y position relative to CIE XYZ. Know what the Y stands for? That’s right... that’s luminance.

So guess where those luminance coefficients are derived from in all YCbCr encodings? That’s right... the same CIE XYZ model. So why are they in every single YCbCr encoding? Because the Y weights correspond to the RGB encoding to optimize the nonlinearly encoded luma plane.

Unfuck yourself before speaking down to folks who actually know their shit.

I’m 100% open to being wrong and making mistakes, but Jesus fuck man, stand the fuck down. There are real issues still present in FFMPEG’s colourimetric handling, and making sure it is handling things properly is critical.

[Balling]

Replying to troy_s:

Great. So FFMPEG now has someone checking bugs who works on one of the most fucked up browsers on the planet with respect to pixel management.

That is just because we removed support for HW acceleration of color management in Windows... In MacOS it still works though. Anyway, we use ffmpeg for a lot of stuff on Android, that is why I am checking for bugs here.

Unfuck yourself before speaking down to folks who actually know their shit.

But I know that. It is called NPM -- normalised primaries matrix. If you want others to understand you use normal terminology as in SMPTE standards, in this case SMPTE RP 177. We are soon adding Chromaticity-derived <constant, non-constant> luminance stuff to Chrome, so I know what I am talking about.

[Balling]

BT.709 and sRGB share identical luminance coefficients

But then again, for sRGB BT.601 matrix is used (that is all YCbCr JPEG BTW), see the part of article I wrote: ​https://en.wikipedia.org/wiki/SRGB#sYCC_extended-gamut_transformation It is suboptimal of course, but... All BT.601 content is wrong anyway! BT.601 was derived from System M primaries with Ill. C! It is different from SMPTE C that uses D65.

[Balling]

The following, despite being 601 coefficients, should yield perfect 1:1 with RGB values (RGB being a relative color space) but quantization issues again result.

No, it should not. FFplay does not really support some BT.601 stuff, because it uses SDL library for decoding (very dumb, I know). If you want correct output use mpv. See #8862

[pdr0]

Replying to Balling:

The following, despite being 601 coefficients, should yield perfect 1:1 with RGB values (RGB being a relative color space) but quantization issues again result.

No, it should not. FFplay does not really support some BT.601 stuff, because it uses SDL library for decoding (very dumb, I know). If you want correct output use mpv. See #8862

Yes , but because of incomplete syntax. FFplay supports "BT.601 stuff" with vf scale as he was using. But for ffplay, when using -vf scale arguments - you just add the format=gbrp to the end after -vf scale arguments (rgb24 works as well) to get the expected "almost" correct +/-3 colors.

-vf scale=in_range=full:in_color_matrix=bt601,format=gbrp

"almost" correct because that is an 8bit pipeline. You'd get psnr inf when using 10bit444 for the RGB => 10bit YCbCr => RGB round trip

Yes, there are other issues with ffplay reading flags automatically and adjusting. But it's the same workaround for #8862 until a proper "auto" fix is implemented

ffplay -i vokoscreenNG.x264_q0_high.mkv -vf scale=in_color_matrix=bt601,format=gbrp

(But on the other hand, sometimes files are flagged improperly. You can make a case for not autoadjusting the rgb conversion)

[Balling]

Replying to pdr0:

Right! Added format=gbrp and the color is now 238, 77, 44 as -vf scale=in_range=full:in_color_matrix=bt601 for that test sample 2.mp4!

You can make a case for not autoadjusting the rgb conversion

But not with defaulting ST240M matrix to BT.601 (P.S. I fixed that part at least in 3249b963421cf5ff0e6adf2e38c86194fdf50929.)

​https://github.com/FFmpeg/FFmpeg/blob/573f05a7533cd9aed3ed895b4fa4ad8fcba4e56a/fftools/ffplay.c#L968