libaom-av1 damages brightness range by switching yuvj420p to yuv420p

[john123]

Summary of the bug: libaom-av1 reduces brightness range by switching yuvj420p to yuv420p
How to reproduce: encode a yuvj420p input using libaom-av1

The downgrade of quality is listed as a warning as per below, although the implication is not at all clear unless you research the difference between them:

Incompatible pixel format 'yuvj420p' for codec 'libaom-av1', auto-selecting format 'yuv420p'

And indeed the help shows yuvj420p is not supported.

In the head of github source there is a AV_PIX_FMT_YUVJ420P defined, but it's not allowed in av1_pix_fmts etc. Is this intentional? It makes AV1 rather damaging for a yuvj420p input that has a lot of very dark and very bright pixels.

Not sure if I should try to patch in AV_PIX_FMT_YUVJ420P or not... If there's some reason that libaom-av1 is fundamentally incompatible with yuvj420p, it seems at least ffmpeg should recommend some alternative that doesn't result in a degraded output.

[Carl Eugen Hoyos]

Replying to john123:

In the head of github source there is a AV_PIX_FMT_YUVJ420P defined, but it's not allowed in av1_pix_fmts etc. Is this intentional? It makes AV1 rather damaging for a yuvj420p input that has a lot of very dark and very bright pixels.

That makes no sense.

Why don't you provide sample input and command line including the complete, uncut console output to give us a chance to understand your issue?

[john123]

Which part makes no sense? See attached full output, although it doesn't seem very interesting.

Looks like this same bug existed in libx265 up until this recent fix: ​https://github.com/FFmpeg/FFmpeg/commit/80757bed89c361fe2dc56eb61f4cb3b9f9b1aedc

I can't find a trac or anything to go along with it, but the issue it fixed was essentially identical, but with libx265 instead of AV1. See this twitter thread: ​https://twitter.com/daemon404/status/1132605137485291520

[Carl Eugen Hoyos]

Replying to john123:

Which part makes no sense?

You claim that the missing feature you see damages the encoding ...

See attached full output, although it doesn't seem very interesting.

Looks like this same bug existed in libx265 up until this recent fix: 80757bed89c361fe2dc56eb61f4cb3b9f9b1aedc

... but this patch did not fix any damage.

[john123]

Maybe "damage" was misleading somehow, but the encoded version loses significant color quality, which is not what one would expect to happen. The patch sure seems to fix the same unexpected quality loss, just with a different encoder. I'm open to suggestions of an alternative ticket summary.

[Carl Eugen Hoyos]

Replying to john123:

Maybe "damage" was misleading somehow, but the encoded version loses significant color quality

If this is true, the bug is somewhere else (not in the libaom encoder wrapper which sets the colour range).

[pdr0]

Official aomenc does not support full range flag (yet) , so any ffmpeg wrapper based on the library does not either

It's not really a "bug" per se; it's just the default ffmpeg behaviour . It clamps for full range YUV flagged as full range. ie. yuvj420p => yuv420p (Y 0-255 => 16-235 CbCr 0-255 => 16-240 for 8bit) . The clamping is what you are seeing for the changes in levels

Some workarounds to bypass ffmpeg behaviour (clamping full range yuvj420p to yuv420p) would be to either strip the full range flag (using bitstream filters, or 3rd party tools), or -vf scale=in_range=pc:out_range=pc,format=yuv420p or similar using -vf zscale . Basically what you're doing is keeping the data full range but changing the pixel format to yuv420p

But this means output will be full range YUV, but NOT flagged as full range. This can be pose problems, depending on what the target is for playback . Most playback scenarios expect limited range, not full range. Ideally, AV1 should get full range flags, like AVC, HEVC, eventually.

[john123]

Are you saying this is fundamentally different from 80757bed89c361fe2dc56eb61f4cb3b9f9b1aedc ? Regardless of whether it's called a "bug" or a missing enhancement, I'm wondering if I should try to create a patch like 80757bed89c361fe2dc56eb61f4cb3b9f9b1aedc or if that's somehow not applicable to this codec.

[pdr0]

No, it's fundamentally the same. Full range 4:2:0 in ffmpeg parlance is "yuv420j"

You could try making the patch for the pixel format; but the full range flag should also be implemented in the official AOM AV1 library (like it is for official AVC, HEVC specs in the VUI parameters for the bitstream) , otherwise it might break compliance with official streams, official decoders.

[john123]

Was there a trac on 80757bed89c361fe2dc56eb61f4cb3b9f9b1aedc ? That one was a bug, was correctly fixed in ffmpeg, but you're saying this trac is an enhancement and should be fixed somewhere else? If so, where? ​https://aomedia.googlesource.com/aom/ ?

[john123]

The "Incompatible pixel format" warning/error message is there only if you specify "-pix_fmt yuvj420p" on the command line.

There really should also be a warning/error message for the automatic switch from yuvj420p to yuv420p, if it's going to result in a pixel brightness being damaged.

[Balling]

yuvj4xxp or more correctly the "j" part of it that specifies full range is deprecated in favour of setting -vf scale=out_color_matrix=bt709:out_range=pc -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range pc

It was deprecated for 11 years (98aea87b1a3a96b9d82deca09291aaec2f54399e) yet even in 2019 somebody was still adding workarounds (80757bed89c361fe2dc56eb61f4cb3b9f9b1aedc due to ​https://news.ycombinator.com/item?id=20036710). Either undeprecate it or do something that will not print these problematic warnings for new formats like jpeg2000 encoder or av1.

As for a flag in bitstream, it is done: ​https://github.com/FFmpeg/FFmpeg/blame/cb23c1e55359dc1319690517251ee170d0d68bae/libavcodec/libaomenc.c#L1147 also even if it somehow does not work (P.S. I checked it, it does set color_description_present_flag part of AV1 bitstream) you can use -bsf:v av1_metadata=color_range=pc with -c copy to set that tag (that still means you must use -v scale=out_range=pc).

[john123]

Sorry for not following all of that, but did you imply some of those command line arguments would be able to work around the current bug? I tried the 709 blob of arguments and the color_range blob, and neither seemed to do anything. If there is some blob of arguments that would fix it, I would very much appreciate it, since I'm still sitting on MP4 videos that I can't convert to AV1/webm due to this bug.

Is the current bug identical to the John Carmack bug, just with a different codec? I've never understood why one was fixed, but the other has been left broken.

[Balling]

-pix_fmt yuvj420p is deprecated and must not be used. Ever. That is what deprecation usually means.

It works for me.

ffmpeg.exe -i 9.mp4 -vf scale=out_color_matrix=bt709:out_range=pc,format=yuv420p -c:v libaom-av1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range pc out.mp4

The out file is:

Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Scan type                   : Progressive
Bits/(Pixel*Frame)          : 0.000
Stream size                 : 4.24 KiB (79%)
Color range                 : Full
Color primaries             : BT.709
Transfer characteristics    : BT.709
Matrix coefficients         : BT.709
Codec configuration box     : av1C

nclx atom is with exiftool (no range info nclx, it is not for this):

Color Representation            : nclx 1 1 1

Flag in bitstream is set.
ffmpeg out.mp4 -c copy -bsf:v trace_headers -f null -

[trace_headers] 97          color_description_present_flag                              1 = 1
[trace_headers] 98          color_primaries                                      00000001 = 1
[trace_headers] 106         transfer_characteristics                             00000001 = 1
[trace_headers] 114         matrix_coefficients                                  00000001 = 1
[trace_headers] 122         color_range                                                 1 = 1
[trace_headers] 123         chroma_sample_position                                     00 = 0
[trace_headers] 125         separate_uv_delta_q                                         0 = 0
[trace_headers] 126         film_grain_params_present                                   0 = 0
[trace_headers] 127         trailing_one_bit                                            1 = 1

on all frames.

ffplay out.mp4 -vf extractplanes=y shows correct for 709 full range values.

[john123]

same bug, but with 709

[Balling]

"yuv420p(pc, bt709, progressive)"

Looks good.

[john123]

Thanks. So, I tried that and the output corruption is actually worse in terms of black becoming grey. The log does show it did something in that the output was "yuv420p(pc, bt709, progressive)". Visually the output looks about twice as bad with those options as it does with the original options.

[Balling]

"yuvj420p(pc, smpte170m/bt470bg/bt2020-12)"

You cannot be serious right now. bt2020-12 transfer is just bt709. Even EOTF is the same, BT.1886 for these transfers. You do not need it especially for 8 bit file!!! WTF. Why??

Next, smpte170m is BT.601 matrix.

Please provide in.mp4.

Also try this: ffmpeg.exe -i in.mp4 -vf scale=out_color_matrix=smpte170m:out_range=pc,format=yuv420p -c:v libaom-av1 -color_primaries bt470bg -color_trc bt709 -colorspace smpte170m -color_range pc out.mp4

And please. Use mpv player! Other players are so broken...

[john123]

file normal colors, corrupted colors when using ffmpeg av1

[john123]

The new command line blob ffmpeg -i in.mp4 -vf scale=out_color_matrix=bt470bg:out_range=pc,format=yuv420p -c:v libaom-av1 -color_primaries smpte170m -color_trc bt709 -colorspace bt470bg -color_range pc -b:v 0 -cpu-used 8 out-470-709.mp4 yields the original level of brightness range clamping in the original bug report.

Here's an example file that exhibits the usual color corruption during AV1 conversion. It's not identical to the original file, but neither of these files are some obscure thing - they're but just the standard output from video recording on an Android phone.

I used ffmpeg to shrink the file down some in order to be able to attach it here, but the resulting failure when converting to AV1 is identical either way.

[Balling]

Okay, your file is full range it has 255 Y, which is not allowed in limited range (only 1-254 are allowed). That is good!

Next: ffmpeg.exe -i small-33.mp4 -vf scale=out_color_matrix=bt601:out_range=pc,format=yuv420p -c:v libaom-av1 -color_primaries bt470bg -colorspace smpte170m -color_trc smpte170m -color_range pc out.mp4

produces perfect file with 255 same place and NO changes in mpv when you alt-tab on my reference display. I will attach. Even --gamut-warning is almost the same (but then again, mpv is bad with that option in full range). Please use normal player.

[john123]

So I tried comparing browsers and at least one (Android Chrome) renders the ffmpeg av1 output identically to the original color range.

Firefox: color damaged by ffmpeg.
Edge: color damaged by ffmpeg.
Android Chrome: color looks good.

So there is indeed some disagreement between players as to what to do with the ffmpeg output in this case.

Any idea which part of ffmpeg's transformation is problematic? Originally it seemed like the switch from yuvj420p to yuv420p was the main thing, but perhaps it's something else?

[Balling]

Android Chrome does not support full range YCbCr, uses limited instead! At all, in all codecs! Even windows one is very broken now and fixed only in Canary, but even there rises black. I know since I am one of Android Chrome and Chromium color developers. See (yes comment 97, it is a big thread):
​https://bugs.chromium.org/p/chromium/issues/detail?id=1068690#c97

switch from yuvj420p to yuv420p

There is no such thing as yuvj420p with new encoder like AV1 or jpeg2000! It is just not supported. When you used that it selected limited range. Now, modern way is to select yuv420p and then select the range of that yuv420p with -vf scale. Okay?

Where do you open your old AVC file? Looks like you open it somehwere that does not support full range. Oogh.

[Balling]

Again, there is no difference in color at all on two latest attached files here. See attached screenshot of both files open in mpv with name of files, etc. You should cut both of the images and switch fast between them.

[Balling]

There is BTW a cosmetic problem here that prints a warning about deprecated pixel format even with my commands (due to source file being yuvj420p not due to output range). So 51fed95dde7abe838cb55188e28f410fded009c7 was not enough. Oogh.

[john123]

Thanks, so are you saying Android Chrome (which is the only browser I could find that accepts this ffmpeg output) is broken also? Ie, perhaps no browser supports what ffmpeg is doing here - but you're saying ffmpeg is correct and every browser is broken?

More importantly for my immediate case (and for anyone else stumbling on this critical bug), is there some other AV1 format command line options that ffmpeg can output and get correct results across all modern browsers?

[Balling]

"but you're saying ffmpeg is correct and every browser is broken?"

Both Mozilla and Chrome use ffmpeg on all platforms anyway to decode and demux. But Chrome uses HW accelerated GPU shaders to do YCbCr transforms and has specific HW decoder, including newer Directx12on11 and new Mojo Video Decoder, that uses MFF. There is this thing called SurfaceControl API in modern Android to support HDR and stuff. There is a bug somehwere that breaks full range. AFAIK it is in Android itself.

In case of windows you can install latest Chrome Canary and check there or use --disable-gpu Chrome command option to turn off YCbCr shaders. YCbCr shaders in chrome also have a bug that makes them off-by-one (​https://bugs.chromium.org/p/chromium/issues/detail?id=1174638).

[Balling]

It is time to remove yuvj4xxp from ffmpeg so this confusion does not happen anymore and such bugs like these are no longer applicable. The only thing that blocks this is jpeg encoder that now still accepts bt.709 matrix, even though it only supports 601 matrix. Linited range though is being warned upon, as jpeg only supports full range. ​https://github.com/FFmpeg/FFmpeg/commit/059fc2d9da5364627613fb3e6424079e14dbdfd3#diff-c313529a7d19507d84be2cf41c45600bb052693686cef605ee84f0f27459e420R449

Jpeg decoder should migrate too.