Bias in planar YUV to YUV bit depth conversion

[abc123]

Summary of the bug: Unscaled YUV to YUV conversion (swscale_unscaled.c:planarCopyWrapper) produces output with a bias of -0.375 compared to the ideal real number result. We compare the ffmpeg output to the equation round(x / 4.0) which maps limited-range ITU-R 709 10-bit (64-940 Y, 64-960 UV) to 8-bit (16-235 Y, 16-240 UV).

How to reproduce:

% ffmpeg -pix_fmt yuv420p10le -s 1280x720 -i testGrad_01.yuv -pix_fmt yuv420p -f rawvideo testGrad_02.yuv
ffmpeg version N-59852-g785dc14 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jan 14 2014 22:07:30 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfi
g --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopenco
re-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-li
bvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 62.100 / 52. 62.100
  libavcodec     55. 48.101 / 55. 48.101
  libavformat    55. 23.103 / 55. 23.103
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  1.100 /  4.  1.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
[rawvideo @ 0000000000338080] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from 'testGrad_01.yuv':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 552960 kb/s
    Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le, 1280x720, 552960 kb/s, 25 tbr, 25 tbn, 25 tbc
File 'testGrad_02.yuv' already exists. Overwrite ? [y/N] y
Output #0, rawvideo, to 'testGrad_02.yuv':
  Metadata:
    encoder         : Lavf55.23.103
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1280x720, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=    1350kB time=00:00:00.04 bitrate=276480.0kbits/s
video:1350kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.000000%

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

An example gradient (1280x720, 10-bit, 4:2:0) is attached to this ticket. We can pay a bounty for a fix to this problem.

[Kieran Kunhya]

I think this is expected behaviour because we specifically don't want to round but instead dither.

[abc123]

Dithering is perfectly fine (in fact, expected) but it should not change the average level of the image. If you test the example frame with the command-line I posted, you will see a very obvious change in global color.

[IsaacDian]

Sorry to ask, but why did this issue seem to be ignored? Does it need more samples? FWIW, Libav swscale doesn't have this issue.

[Kieran Kunhya]

Libav swscale doesn't have dither and does a plain shift which is why it doesn't have the issue.

[Elon Musk]

How much is bounty?

[Balling]

but instead dither

Maybe your floyd-steinberg dither is wrong in some place?

I can reproduce it. It looks like you do perfect convertion on the bottom (in checkerborder pattern if you look into the difference that is like 29x2 pixels when it breaks) and then you fail on 59 and 60th, then you get on track again (this time on more little pixels, not 2x2), and again fail, etc. Very interesting.

[Balling]

-filter_complex was using bilinear, even though it ahould be bicubic fixed in 9f14396a5103ec80893db801035ece5d14c0d3c5.

In ffplay you can try to set -sws_dither 0.

[Balling]

Okay. When you play this, one should use

ffplay -pixel_format yuv420p10le -video_size 1280x720 testGrad_01.yuv -vf scale=in_color_matrix=bt709,format=gbrp

Your command line is technically wrong. BT.709 is not the default: it is BT.601. So it does conversion assuming bt601 to bt601, all limited. That surprisingly does not affect the pixels. What does affect the pixels is no dither (recently fixed):

ffmpeg -pix_fmt yuv420p10le -video_size 1280x720 -i testGrad_01.yuv -vf scale=in_range=limited:in_color_matrix=bt709:out_range=limited:out_color_matrix=bt709:sws_dither=0,format=yuv420p -f rawvideo testGrad_03.yuv

That file is different at least.

Now, lets test that 03 file with the use of to PNG conversion.

ffmpeg -pixel_format yuv420p10le -video_size 1280x720 -i testGrad_01.yuv -vf scale=flags=bilinear:in_color_matrix=bt709:sws_dither=none 03testgrad.png

Not a surprise that this produces 16 bit per component png file, yet we know that such 10 bit to 16 bit conversion is broken from #7978.

So you use ffmpeg -pixel_format yuv420p10le -video_size 1280x720 -i testGrad_01.yuv -vf zscale=matrixin=bt709,format=gbrp16le 031zscaletestgrad.png

And same for testGrad_03.yuv.

ffmpeg -pixel_format yuv420p -video_size 1280x720 -i C:\Users\ZAQU\testGrad_03.yuv -vf zscale=matrixin=bt709,format=gbrp16le 031zscaletestgrad.png

So, yeah... That still looks bad.

[pdr0]

@Balling - this is an 8 year old ticket

OP claims that issue isn't present with libav swscale. You get bit identical results with libav. Libav dithers by default too. I tested libav binaries from 2018 and 2014. I couldn't find older binaries. Maybe it didn't dither back then.

avconv -pix_fmt yuv420p10le -s 1280x720 -i testGrad_01.yuv -pix_fmt yuv420p -f rawvideo testGrad_02avconv.yuv

[Balling]

It is not OP but Kieran that claimes this libav is not affected.

There is no libav anymore or at least no real development there. Most of patches from libav are in ffmpeg anyways.

Conversion from 10 bit to 8 is indeed a simple shift. Just as back. If you go from RGB you are supposed to be precise though, there it IS NOT a simple shift.

Oh, and also dither should be turned off by player too then.

[pdr0]

Replying to Balling:

It is not OP but Kieran that claimes this libav is not affected.

Actually it's not Kieran, but Issac Dian who makes that claim ( who I thought was the OP, abc123). It's Kieran's reply that explains why. But swscale still appears to have the issue, and it's separate issue from dither

There is no libav anymore or at least no real development there. Most of patches from libav are in ffmpeg anyways.

True, but that claim about libav was made 8 years ago, when libav development was more active

If you test a ffmpeg binary around that time from the ffmpeg build in the OP (Jan 14 2014) - the results are different than ffmpeg today, either default or dither disabled

The issue is still present with libav 2018, so maybe there was more than one issue. Maybe observations were being confused with dithering, or maybe there were commits that broke and fixed things over time. My guess is problem was always there

The ticket is for YUV to YUV - so ideally that should be examined in YUV. There shouldn't be any talk about RGB conversions or players or PNG's, or you just obfuscate the issue and introduce other variables

The sample provided is 4:2:0, so the U,V planes at 640x360 have to be examined and compared to a "divide by 4" result on each plane.

The Y plane is a vertical grad goes from Y=525 at Ypos 0 , to Y=504 at Ypos 719
U plane goes from U=501 at Ypos 0 , to U=513 at Ypos 359
V planes goes from V=560 at Ypos 0 , to V=508 at Ypos 359

The claim was a bias of -0.375 compared to the "divide by 4" result.

If that is still a bug, let's pick some values that should demonstrate the error .

523/4 = 130.75 => should round up to 131 . -0.375 Bias would mean 130.375 and rounding down to 130

503/4 = 125.75 => should round up to 126 . -0.375 Bias would mean 125.375 and rounding down to 125

559/4 = 139.75 => should round up to 140 . -0.375 Bias would mean 139.375 and rounding down to 139

Y=523 occurs between Ypos 117 to 149 (1280x720)
U=503 occurs between Ypos 74 to 100 (640x360)
V=559 occurs between Ypos 20 to 30 (640x360)

ffmpeg -vf scale, nodither
Y=130
U=125
V=139

ffmpeg zscale, nodither
Y=131
U=126
V=140

I checked a few more coordinates and it looks like issue still present with swscale, but absent with zscale .

[Balling]

Are results worse with dither? I suppose the problem here is that strange double color dither even with dither none, you reproduced that on that AV1 file, rememember? Why I somehow did not.

fluctuates a bit, almost as if output is dithered

​https://trac.ffmpeg.org/ticket/9167#comment:10

[pdr0]

Replying to Balling:

Are results worse with dither? I suppose the problem here is that strange double color dither even with dither none, you reproduced that on that AV1 file, rememember? Why I somehow did not.

fluctuates a bit, almost as if output is dithered

​https://trac.ffmpeg.org/ticket/9167#comment:10

Yes, results are worse with dither. I'll revisit the other ticket later

For this ticket , I attached a 1280x720 yuv444p10le .yuv frame, single color YUV 523,503,559 so it's easier to reproduce the issue and ease of testing

ffmpeg -pix_fmt yuv444p10le -s 1280x720 -i 1280x720_yuv444p10le_523_503_559.yuv -vf zscale=dither=none,format=yuv444p -f rawvideo test02_zscale_nodither.yuv
YUV 130,125,139

ffmpeg -pix_fmt yuv444p10le -s 1280x720 -i 1280x720_yuv444p10le_523_503_559.yuv -vf scale=sws_dither=none,format=yuv444p -f rawvideo test02b_swscale_nodither.yuv
YUV 131,126,140

I tried some sws flags such as accurate_rnd, but no change

[pdr0]

Replying to pdr0:

Replying to Balling:

Are results worse with dither?

Yes, results are worse with dither.

To clarify, I mean "worse" in the sense that it adds another variable. On that solid color, some values increase using swscale to the correct value because of the dithering "noise" added - it's actually "better" in that sense because of 100% wrong, it's some % right. Or if you look at zscale, instead of being 100% right, it adds some % wrong when you add dither. The underlying problem for this ticket should be a math issue somewhere