SWSCALE: Quantization Errors in Y Studio Range

[troy_s]

Summary of the bug:

It appears FFMPEG has some +/- single unit quantization errors in the YCbCr transform.

How to reproduce:

Generate a raw YCbCr image and diff against theoretically correct values.

./ffmpeg -i ~/Documents/FFMPEG\ Color\ Research/ARIB_STD-B28.png -pix_fmt yuv444p -vcodec rawvideo -f rawvideo -vf scale="in_range=full:in_color_matrix=bt709:out_range=full:out_color_matrix=bt709" test.yuv
ffmpeg version N-64815-gb53bdae Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 19 2014 16:33:49 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
  configuration: --enable-gpl --enable-libx264 --disable-asm
  libavutil      52. 92.101 / 52. 92.101
  libavcodec     55. 69.100 / 55. 69.100
  libavformat    55. 48.101 / 55. 48.101
  libavdevice    55. 13.102 / 55. 13.102
  libavfilter     4. 11.102 /  4. 11.102
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from '/home/aphorism/Documents/FFMPEG Color Research/ARIB_STD-B28.png':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: png, rgb24, 1920x1080 [SAR 2835:2835 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Output #0, rawvideo, to 'test.yuv':
  Metadata:
    encoder         : Lavf55.48.101
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc55.69.100 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> rawvideo (native))
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=    6075kB time=00:00:00.04 bitrate=1244160.0kbits/s    
video:6075kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

Diff'd versions of the theoretically ideal values versus FFMPEG's output. Lowest red stripe is due to theoretical version preserving the -2% patch, and not relevant.

[troy_s]

out_range incorrect in the sample above, and should be out_range=mpeg

[troy_s]

Difference file. Red areas are discrepencies. Ignore -2% patch, and only note Y gradient.

[troy_s]

Theoretically ideal YCbCr.

[troy_s]

FFMPEG output with studio range.

[troy_s]

SMPTE test pattern, 8 bit.

[Carl Eugen Hoyos]

Replying to troy_s:

out_range incorrect in the sample above, and should be out_range=mpeg

Sorry if I misunderstand but the idea is of course that you provide the actual command line that allows to reproduce the issue you see.
(It is unclear to me if this applies but if it does, please fix it.)

[Carl Eugen Hoyos]

There is something that I don't understand about this ticket (which in reality doesn't matter since I won't be fixing the issue):
Your command line outputs a raw yuv file. If there is something wrong with this file, I would expect you to write "the third pixel from the left has input value x:y:z, FFmpeg outputs a:b:c instead of the expected y:u:v" or "all pixels are 2% too bright" or "the dynamics are all off as in...". Instead you attached rgb24 images: How can they be used to check yuv values?

This is apart from the fact that I wonder if pal8 as an input format (instead of rgb24 and/or bgr24) may have its own subtle issues.
(This reminds me that there is ticket #979 about different output values depending on the input being rgb24 or bgr24.)

[troy_s]

The “diff” is an image based diff which shows precisely what pixels are off from their theoretical.

The RGB format is a relative tri-color space, and given correct handling, is nothing more than a wrapper to make the data easier to compare. Here the Y, Cb, and Cr planes were extracted from the raw YCbCr planar file and recombined into an RGB wrapper.

The red channel corresponds to Y, with green and blue Cb and Cr respectively.

These Y quantization errors seem minor, but the full range output exhibits many more errors that I will report tomorrow. I posted this because the issues may be related.

[Andy Furniss]

rgb to yuv conversions don't honor *_range=full that only works with yuv -> yuv.

The output of your command (I just tested) produced video range - of course by default players will stretch this back to full RGB (and may also sub-sample as well).

If you want full range rgb -> yuv use -pix_fmt yuvj444p

[Andy Furniss]

Oops - ignore the above, my ffmpeg was slightly too old - I've seen the other bug now and see the range not working with rgb to yuv is now fixed.

[troy_s]

Potentially related ticket:

SWSCALE: Incorrect Values in Full Range Conversion RGB to YCbCr
​https://trac.ffmpeg.org/ticket/3801

[troy_s]

Sorry if I misunderstand but the idea is of course that you provide the actual command line that allows to reproduce the issue you see.

Apologies. The following is the fully corrected command to generate the raw YCbCr file output to test-studio.yuv using REC.709 coefficients:

./ffmpeg -i ~/Documents/FFMPEG\ Color\ Research/ARIB_STD-B28.png -pix_fmt yuv444p -vcodec rawvideo -f rawvideo -vf scale="in_range=full:in_color_matrix=bt709:out_range=mpeg:out_color_matrix=bt709" test-studio.yuv
ffmpeg version N-64958-gce8e27e Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 24 2014 10:54:07 with gcc 4.7 (Ubuntu/Linaro 4.7.3-12ubuntu1)
  configuration: --enable-gpl --enable-libx264 --enable-shared --disable-static --enable-pic
  WARNING: library configuration mismatch
  avutil      configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avcodec     configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avformat    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avdevice    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avfilter    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  swscale     configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  swresample  configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  postproc    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  libavutil      52. 92.101 / 52. 92.101
  libavcodec     55. 69.100 / 55. 69.100
  libavformat    55. 49.100 / 55. 49.100
  libavdevice    55. 13.102 / 55. 13.102
  libavfilter     4. 11.102 /  4. 11.102
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from '/home/aphorism/Documents/FFMPEG Color Research/ARIB_STD-B28.png':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: png, rgb24, 1920x1080 [SAR 2835:2835 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Output #0, rawvideo, to 'test-studio.yuv':
  Metadata:
    encoder         : Lavf55.49.100
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc55.69.100 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> rawvideo (native))
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=    6075kB time=00:00:00.04 bitrate=1244160.0kbits/s    
video:6075kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
aphorism@u64x6:~/Develop/ffmpeg$ convert -depth 8 -size 1920x1080 -interlace plane Gray:./test-studio.yuv -set colorspace RGB test-studio.png
aphorism@u64x6:~/Develop/ffmpeg$ convert test-studio-* -set colorspace RGB -combine test-studio-444-out.png
aphorism@u64x6:~/Develop/ffmpeg$ compare ./test-full-444-out.png ~/Documents/FFMPEG\ Color\ Research/ test-studio-diff.png

[Carl Eugen Hoyos]

Replying to troy_s:

  WARNING: library configuration mismatch

This makes a bug report normally invalid. Even if there is nothing suspicious in the following output, it is still recommended that you test and report issues with ./configure && make if this allows to reproduce them.

[troy_s]

Apologies. Bumped to trunk without aligning the system values.

./ffmpeg -i ~/Documents/FFMPEG\ Color\ Research/ARIB_STD-B28.png -pix_fmt yuv444p -vcodec rawvideo -f rawvideo -vf scale="in_range=full:in_color_matrix=bt709:out_range=mpeg:out_color_matrix=bt709" test-studio.yuv
ffmpeg version N-64958-gce8e27e Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 24 2014 10:54:07 with gcc 4.7 (Ubuntu/Linaro 4.7.3-12ubuntu1)
  configuration: --enable-gpl --enable-libx264 --enable-shared --disable-static --enable-pic
  libavutil      52. 92.101 / 52. 92.101
  libavcodec     55. 69.100 / 55. 69.100
  libavformat    55. 49.100 / 55. 49.100
  libavdevice    55. 13.102 / 55. 13.102
  libavfilter     4. 11.102 /  4. 11.102
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from '/home/aphorism/Documents/FFMPEG Color Research/ARIB_STD-B28.png':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: png, rgb24, 1920x1080 [SAR 2835:2835 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Output #0, rawvideo, to 'test-studio.yuv':
  Metadata:
    encoder         : Lavf55.49.100
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc55.69.100 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> rawvideo (native))
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=    6075kB time=00:00:00.04 bitrate=1244160.0kbits/s    
video:6075kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

Yields the same difference map and file results.

[Carl Eugen Hoyos]

I know this isn't related to the problem you reported but is there a reason why you are using --enable-pic?
Maybe a bug in our configure script that we are not aware of?

[troy_s]

Attaching the matrix used for conversion.

0.1825858824	0.6142305882	0.0620070588	+0.062745098
-0.1006437324	-0.3385719539	0.4392156863	+0.5019607843
0.4392156863	-0.3989421626	-0.0402735237	+0.5019607843

The above is the float version of:

0.2126*(219/255)              0.7152*(219/255)              0.0722*(219/255)           +(16/255)
-(0.2126/1.8556)*(224/255)    -(0.7152/1.8556)*(224/255)    0.5*(224/255)              +(128/255)
0.5*(224/255)                 -(0.7152/1.5748)*(224/255)    -(0.0722/1.5748)*(224/255) +(128/255)

(For the curious, the 1.8556 and 1.5748 is nothing more than (Coeff.Red + Coeff.Green) * 2 and (Coeff.Blue + Coeff.Green) * 2 to normalize the values around the origin.)

[Balling]

Did you see this: ​https://trac.ffmpeg.org/ticket/1582#comment:14

Also there is a funny bug in bmp vs. png: ​https://video.stackexchange.com/questions/19944/ffmpeg-bmp-to-yuv-x264-color-shift (that was totally surprising for me, bug #8056, duplicate of #979). This also needs to be retested after 98038e2ee0c40530c42db70ed3d2f5e0c17b175e. This was also totally surprising for me that you used BT.709-1 (first version, OMG) matrix coeff. (​https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.709-1-199311-S!!MSW-E.doc) while those were updated in BT.709-2. It may have influenced this stuff. BTW, it is very NOT funny. And I was thinking why there are so many out-of-gamut stuff with mpv's --gamut-warning in some test patterns (P.S. has nothing to do with swscale, but is was because of x264).

[Balling]

You can't be serious right now...

The gradient in RGB has 256 gradations, while gradient in YCbCr limited only can have 235-16+1. Seriously??? I mean, this is just wow. Of course you will lose some grays in --> YCbCr (here dither plays a role to restore it partially)!

That is why SMPTE RP 219 defines everything for Y! And does not have Full range at all!

Also if you looked accurately into the SMPTE RP 219 standard, you would find out that part of your ramp is 100% white (235, 128, 128) there and part is black or +Q. For Arib they just define the form of ramp (no 100 black or white region), the SAME way SMPTE 219 does. It is 1440 pixels long, and 205 is black (or +Q) and 205 are 100% white. See how it should have been done (not saying it is 100% perfect too, black and white parts of ramp are too big for SMPTE 219). ​https://www.youtube.com/watch?v=PqvEy5JLll4

[Balling]

So, for example, if you will have 188, 188, 188 gray, it will be converted to 177, 128, 128 using BT.709 matrix, but if then you will convert back (after rounding of YCbCr) you will get 187.47, 187.47, 187.47 which is not 188. 188 gray is still there because of dithering though (because of dither all 256 values are preserved). That 188, 188, 188 is one of bad colors in your diff. See ​https://res18h39.netlify.app/color