SWSCALE: Incorrect Values (+-1) in Full Range Conversion RGB to YCbCr

[troy_s]

Summary of the bug:

FFMPEG is potentially delivering values that deviate from theoretical results in RGB to YCbCr conversions. This case applies to RGB to full range YCbCr conversion, using REC.709 coefficients on the input.

./ffmpeg -i ~/Documents/FFMPEG\ Color\ Research/ARIB_STD-B28.png -pix_fmt yuv444p -vcodec rawvideo -f rawvideo -vf scale="in_range=full:in_color_matrix=bt709:out_range=full:out_color_matrix=bt709" test-full.yuv
ffmpeg version N-64958-gce8e27e Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 24 2014 10:54:07 with gcc 4.7 (Ubuntu/Linaro 4.7.3-12ubuntu1)
  configuration: --enable-gpl --enable-libx264 --enable-shared --disable-static --enable-pic
  WARNING: library configuration mismatch
  avutil      configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avcodec     configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avformat    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avdevice    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  avfilter    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  swscale     configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  swresample  configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  postproc    configuration: --enable-gpl --enable-libx264 --disable-asm --enable-shared --disable-static --enable-pic
  libavutil      52. 92.101 / 52. 92.101
  libavcodec     55. 69.100 / 55. 69.100
  libavformat    55. 49.100 / 55. 49.100
  libavdevice    55. 13.102 / 55. 13.102
  libavfilter     4. 11.102 /  4. 11.102
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from '/home/aphorism/Documents/FFMPEG Color Research/ARIB_STD-B28.png':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: png, rgb24, 1920x1080 [SAR 2835:2835 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Output #0, rawvideo, to 'test-full.yuv':
  Metadata:
    encoder         : Lavf55.49.100
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc55.69.100 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> rawvideo (native))
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=    6075kB time=00:00:00.04 bitrate=1244160.0kbits/s    
video:6075kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

The above command delivers the attachment test-full-444-out.png, with Y, Cb, and Cr a pure dump of the results into the R, G, and B channels of the image. No color transformations have been applied.

The file ARIB_STD-B28.png is an 8 bit representation of a SMPTE test pattern, based off of a 32 bit float version.

The file ARIB_STD-B28_YCbCr_full.png is a theoretically correct transformation of the ARIB_STD-B28.png test pattern into YCbCr using REC.709 coefficients. It is based off of a 32 bit EXR input.

The file test-full-diff.png is a heat-map difference, with red areas indicating deviations between the theoretical results and the actual.

The SMPTE test pattern black set test stripes at the very bottom contain one stripe that sources at -2%, and as such, will always be incorrect unless float values are provided for input. As such, it is the single bar that can be ignored.

Potentially Related Issues:
​https://trac.ffmpeg.org/ticket/3794

[troy_s]

A full range theoretical transformed version of the SMPTE test chart.

[troy_s]

An 8 bit input SMPTE test chart.

[troy_s]

Output of FFMPEG, with the raw YCbCr values packed into the RGB channels of a PNG with no color transformations applied.

[troy_s]

A visual difference between the 8 bit SMPTE input versus theoretical and FFMPEG YCbCr output.

[michael]

The 2 png images are nearly identical, i see some +-1 differences from different rounding, which is clearly not what was meant in this bug, though if our rounding is suboptimal then such suboptmal rounding of course would be a bug that iam interrested in.

The big vissible color difference is a fault in how the file was created, one of the pngs has a gamma of 1.0 the other does not. firefox displays them quite differently, other software shows them nearly identical, ignoring gamma i assume

xview, displays this difference:
ARIB_STD-B28_YCbCr_full.png is 1920x1080 PNG image, color type RGB, 8 bit
test-full-444-out.png is 1920x1080 PNG image, color type RGB, 8 bit, file gamma 1.0000

Also the difference png shows massive differences in areas that are bit exactly identical

[troy_s]

The rounding is the issue, and I can assure you that all of the values per channel were encoded 1:1.

The reason a gamma of 1.0 was assigned is likely due to that being required to prevent the PNG color mangling.

I could provide raw files, but that is vastly less easy to diagnose.

[michael]

rawvideo YCbCr reference based on png

[michael]

ffmpeg output vs. the reference:
stddev: 0.39 PSNR: 56.28 MAXDIFF: 1 bytes: 6220800/ 6220800