Wrong color levels with kmsgrab and vaapi

[kokoko3k]

Hi, i'm trying to capture my screen and encode the result via kmsgrab.
Unfortunately i'm facing overexposed levels when doing so.
It happens under Xorg (didn't tried wayland), with modesetting or intel driver.
How to reproduce:
The following examples produce wrong color levels (bright grey becomes white and dark gray becomes black):

1> timeout 2 ffmpeg -report -v 9 -loglevel 99 -f kmsgrab -i - -vf 'hwmap=derive_device=vaapi,scale_vaapi=format=nv12' -c:v h264_vaapi -f nut - | ffplay -

2> timeout 2 ffmpeg -report -v 9 -loglevel 99 -f kmsgrab -i - -vf 'hwmap=derive_device=vaapi,hwdownload,format=bgr0' -c:v png -f nut - | ffplay -

...this one looks ok instead (needs a linear framebuffer):

3> timeout 2 ffmpeg -report -v 9 -loglevel 99 -f kmsgrab -i - -vf 'hwdownload,format=bgr0' -c:v png -f nut - | ffplay -

The common option between 1,2 and 3 is the use of hwmap=derive_device=vaapi, so i think the problem is there.

Log files are here:
1: ​http://sprunge.us/Zqozr7
2: ​http://sprunge.us/ZAORG7
3: ​http://sprunge.us/3qaCpH

[kokoko3k]

1

[kokoko3k]

2

[kokoko3k]

3

[kokoko3k]

Hi, i tested further, and it seems that the color levels problems appears even with this:

ffmpeg -f x11grab -r 60 -framerate 60 -video_size 3840x1080 -i :0.0+0,0  -b:v 10000k  -maxrate 10000k         -threads 1 -vaapi_device /dev/dri/renderD128 -vf 'hwupload,scale_vaapi=format=nv12' -c:v h264_vaapi -f_strict experimental -syncpoints none -f nut - | ffplay -

...but NOT with this:

ffmpeg -f x11grab -r 60 -framerate 60 -video_size 3840x1080 -i :0.0+0,0  -b:v 10000k  -maxrate 10000k         -threads 1 -vaapi_device /dev/dri/renderD128 -vf 'format=nv12,hwupload' -c:v h264_vaapi -f_strict experimental -syncpoints none -f nut - | ffplay -

The difference is

-vf 'hwupload,scale_vaapi=format=nv12'

vs

-vf 'format=nv12,hwupload'

[Balling]

bright grey becomes white and dark gray becomes black

That is limited vs full problem.

[jkqxz]

kmsgrab has no way to find those properties from the input, so they are all unknown and each component may interpret that in its own way. Use setparams at the start of the filter chain to set the colour properties to the values matching your input to get consistent behaviour.

[kokoko3k]

Thanks, but the problerm is not limited to kmsgrab (see comment 1)
Anyway i tried your suggestion with the following:

ffmpeg -f x11grab -framerate 60 -video_size 1920x1080 -i :0.0+0,0  -b:v 10000k  -maxrate 10000k  -threads 1 -vaapi_device /dev/dri/renderD128 -vf 'setparams=range=limited,hwupload,scale_vaapi=format=nv12' -c:v h264_vaapi -f_strict experimental -syncpoints none -f nut -
ffmpeg -f x11grab -framerate 60 -video_size 1920x1080 -i :0.0+0,0  -b:v 10000k  -maxrate 10000k  -threads 1 -vaapi_device /dev/dri/renderD128 -vf 'setparams=range=limited,hwupload,setparams=range=limited,scale_vaapi=format=nv12,setparams=range=limited' -c:v h264_vaapi -f_strict experimental -syncpoints none -f nut - | mpv --profile=low-latency --untimed -
ffmpeg  -f kmsgrab -i - -vf 'setparams=range=limited,hwmap=derive_device=vaapi,scale_vaapi=format=nv12' -c:v h264_vaapi -f nut - | mpv --profile=low-latency --untimed -
ffmpeg  -f kmsgrab -i - -vf 'setparams=range=limited,hwmap=derive_device=vaapi,setparams=range=limited,scale_vaapi=format=nv12,setparams=range=limited' -c:v h264_vaapi -f nut - | mpv --profile=low-latency --untimed -

And all of them still have the issue.
I'm attaching a picture of it.

[kokoko3k]

screen capture (infinite mirror), look how color changes.

[kokoko3k]

Proper capture (cpu is doing the color conversion)

[Balling]

Do you understand how this works? You encode then show the encoded frame, then encode that, etc. ffplay is failing here, this is a duplicate of #8862. We are not going to fix that, use normal player.

P.S. CPU is also incorrect, first one has color 0x0E1821 and then it is 0x0D1821 (and others!) and then it is worse and worse. You will alway lose on reencoding.

[Balling]

Do you understand how does this work? You encode, then show the encoded frame, then encode that, etc. ffplay is failing here, this is a duplicate of #8862. We are not going to fix that, use normal player (mpv).

P.S. CPU is also incorrect, first one has color 0x0E1821 and then it is 0x0D1821 (and others!) and then it is worse and worse. You will always lose precision on reencoding.

[Balling]

@cehoyos This is interesting, apparently "FFplay depends on an external library that we do not control, I am closing this ticket assuming it cannot be fixed in the FFmpeg codebase" was not totaly correct...

[kokoko3k]

Balling, i apologize if something particulary obvious is beyond me, but mpv has the same problem, if you look closely at the screenshots, you'll notice that i was using mpv.

Other than that, i posted several examples where the problem is not present.
Again, in comment 1, i suspect it is interesting, so i'm going to quote myself (I changed and just tried with mpv):

wrong levels:

ffmpeg -f x11grab -r 60 -framerate 60 -video_size 3840x1080 -i :0.0+0,0  -b:v 10000k  -maxrate 10000k         -threads 1 -vaapi_device /dev/dri/renderD128 -vf 'hwupload,scale_vaapi=format=nv12' -c:v h264_vaapi -f_strict experimental -syncpoints none -f nut - | mpv -

good ones:

ffmpeg -f x11grab -r 60 -framerate 60 -video_size 3840x1080 -i :0.0+0,0  -b:v 10000k  -maxrate 10000k         -threads 1 -vaapi_device /dev/dri/renderD128 -vf 'format=nv12,hwupload' -c:v h264_vaapi -f_strict experimental -syncpoints none -f nut - | mpv -

I understand that encoding means less quality, but the brightness going down is way more evident in the first screenshot and about unnoticeable in the second one.

[Balling]

What are the levels [of file or pipe] in mpv (use "I" to display stats)? You can try to use --video-output-levels=limited for mpv (that is if you are using limited for your display). But better to set metadata in ffmpeg.

[kokoko3k]

Thanks.

• My display has full output level (but it sets the wrong one and i've to use xrandr to manually switch it to full levels every boot; anyway this can't be the source of the issue, since i tried another display which is correctly identified as full range and the 'issue' persists there too).

• "I" in mpv reports "limited" for both the cases in comment 8.

• if i set --video-output-levels=limited in mpv:

In The case when i saw a good picture in comment 8, it becomes whashed out.
In The case when i saw a bad picture in comment 8, it becomes ok. (THANKS!)

I also tried to switch between full and limited with: vf='scale_vaapi=format=nv12:out_range=limited' and with setparams=range=limited but nothing seems to have any effect
I honestly don't know how to set the metadata in ffmpeg, i saw the docs, but it just speaks about the "-metadata" option without goind into what tags one can use to set the levels range (nor i know if i'm looking into the right place).

Do you still think the problem is in the player side or there is something weird going on the ffmpeg one?

[Balling]

it sets the wrong one and i've to use xrandr to manually switch it to full levels every boot;

You can't just use xrandr to switch to full. You need to also set your monitor settings (or is it LG C9 :) ) to high black level / full range. If you will set one to full/high and the other to limited/low... It will F things up. Auto detection is VEEERY broken.

[kokoko3k]

Yeah, i use xrandr plus display controls just to be sure.
Black to white gradients are perfect (no banding, full range).

I'm not sure to close this one; read: my particular issue is solved, but i did not understood if it is just a workaround or not.

[Balling]

Okay, I am closing this. --video-output-levels=limited only affects the display (i.e. output). That means that otherwise your display is set to limited but you somehow misconfigured it to full (as you said yourself). This is also not a duplicate of #8862.

Again, the problem like this cannot be fixed by boot hacks, because all OS depends on this metadata. As for how to set metadata for file... very simple.

-vf scale=out_range=pc -color_range pc, since swscale defaults to limited range bt.601 (zscale default to source range).

but it sets the wrong one

This is called selectable quantisation and if available on YCbCr can cause all kind of problems since nothing supports that.