libswscale does incorrect range conversion for RGB->YUV and RGB->RGB

[gjdfgh]

libswscale seems to do completely confused range conversions for RGB->YUV. RGB->RGB randomly ignores the source range (at least in the unscaled case), and always ignores the destination range. YUV->RGB ignores destination range as well. The only cases that appear to be working as expected are conversions with YUV->YUV.

Here's the output of a test program, that scales a solid image filled with gray=200, using various colorspace and range parameters. "200" is the input value, and the number after that the retrieved luma value or that of a RGB component. "no-scale" means it has been scaled with same source and destination width/height. The test program is attached as swscale.c to this ticket.

no-scale rgb-lim rgb-lim  200 -> 200
no-scale rgb-lim rgb-ful  200 -> 200
no-scale rgb-lim yuv-lim  200 -> 188
no-scale rgb-lim yuv-ful  200 -> 200
no-scale rgb-ful rgb-lim  200 -> 200
no-scale rgb-ful rgb-ful  200 -> 200
no-scale rgb-ful yuv-lim  200 -> 177
no-scale rgb-ful yuv-ful  200 -> 188
no-scale yuv-lim rgb-lim  200 -> 214
no-scale yuv-lim rgb-ful  200 -> 214
no-scale yuv-lim yuv-lim  200 -> 200
no-scale yuv-lim yuv-ful  200 -> 214
no-scale yuv-ful rgb-lim  200 -> 200
no-scale yuv-ful rgb-ful  200 -> 200
no-scale yuv-ful yuv-lim  200 -> 188
no-scale yuv-ful yuv-ful  200 -> 200

scale    rgb-lim rgb-lim  200 -> 200
scale    rgb-lim rgb-ful  200 -> 200
scale    rgb-lim yuv-lim  200 -> 188
scale    rgb-lim yuv-ful  200 -> 200
scale    rgb-ful rgb-lim  200 -> 188
scale    rgb-ful rgb-ful  200 -> 188
scale    rgb-ful yuv-lim  200 -> 177
scale    rgb-ful yuv-ful  200 -> 188
scale    yuv-lim rgb-lim  200 -> 214
scale    yuv-lim rgb-ful  200 -> 214
scale    yuv-lim yuv-lim  200 -> 200
scale    yuv-lim yuv-ful  200 -> 214
scale    yuv-ful rgb-lim  200 -> 200
scale    yuv-ful rgb-ful  200 -> 200
scale    yuv-ful yuv-lim  200 -> 188
scale    yuv-ful yuv-ful  200 -> 200

YUV->RGB ignores destination ranges, and always assumes full destination range.

RGB->YUV looks completely messed up. Maybe someone flipped a bit somewhere?

Scaled RGB->RGB ignores the destination range, and always seems to assume limitted range (so that rgb-lim->* does no conversion). Unscaled RGB->RGB ignores ranges completely, at least when the same pixel format is used.

If there's some logic behind this, I don't get it, and it's not documented.

If libswscale simply doesn't support limitted RGB range (which would lower its usefulness), at least the RGB->RGB case must be fixed.

[gjdfgh]

Test program

[Michael Niedermayer]

There exists just one 8bit RGB range and that is 0..255, the difference in ranges is specific to YUV. I think thats the only source of errors in the table so iam closing this, but if theres another problem that i missed, please reopen.
Also ive changed sws so it ignores the range for RGB instead of doing something quite random if such combination is used

[gjdfgh]

Limited RGB does seem to exist: for RGB output to TVs and projectors.

Having no limited RGB support is probably better to breaking completely unrelated conversions, though.

[Balling]

Can we reopen this for limited RGB support in the future?

[Balling]

Support for limited range rgb is needed. I do not care what you say.

[Balling]

We need to start by either removing the deprecation of yuvj formats or by removing them alltogether (there is patchset from Paul, see ​https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=&submitter=&state=&q=Yuvj&archive=&delegate=). I vote for the first one. It is a joke that even opening JPEG (since jpeg does not support new method with setting range not by pixel_format, also see a funny bug in mpv, ​https://github.com/mpv-player/mpv/issues/9053#issuecomment-1046052804) causes a warning for every tile or whatever! It is nuts! Warnings obviously make it slower out of nothing.

[chris319]

See the following documents:

​​https://tech.ebu.ch/docs/r/r103.pdf

​​https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.709-6-201506-I!!PDF-E.pdf

[Carl Eugen Hoyos]

Please do not reopen ancient tickets.

[Balling]

Please do not reopen ancient tickets.

​https://w3c.github.io/PNG-spec/#cICP-chunk

PNG cICP now requires limited range RGB. HAHAHA, wasted.

[Elon Musk]

It supports both, 0 is limited, 1 is full.

[Balling]

It supports both, 0 is limited, 1 is full.

Indeed. And we do not.

See:

A Matrix Coefficients value othen than 0 signals a transformation between colour difference and RGB representations, which is not used in this International Standard since PNG images are RGB images.

That means limited range is RGB limited range. YCbCr is not supported in PNG even in newest spec.

[Balling]

Yep. ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230131231254.277086-1-leo.izen@gmail.com/