SDL2 / ffplay cannot do linear transfer/not BT.709 matrix for 4:4:4

[airbete]

Summary of the bug + How to reproduce:

I recorded part of my screen with vokoscreenNG ( x264 / high-4:4:4 / Quality 0 ) where a static png image was displayed.

I then played the resulting mkv file (joined to this ticket) with mpv, vlc and ffplay and took a screenshot in each. Here is what I observed :

Original image == mpv == vlc != ffplay .

This is the screenshot with mpv (identical to vlc and the original):

And this is the screenshot with ffplay:

Clearly ffplay doesn't select the good color space. Also, when I load the clip into kdenlive (which uses ffmpeg/ffplay/ffprobe), I observe the same wrong colors in the clip monitor, project monitor and rendering.

Here are some additional info:

ffplay / ffprobe
Stream #0:0(eng): Video: h264 (High 4:4:4 Predictive), yuv444p(tv, bt470bg/smpte170m/bt709, progressive), 480x300 [SAR 1:1 DAR 8:5], 30 fps, 30 tbr, 1k tbn, 60 tbc (default)
color_range=tv
color_space=bt470bg
color_transfer=bt709
color_primaries=smpte170m

mpv
[vf] [in] 480x300 yuv444p bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [out] 480x300 yuv444p bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264

vlc

The color_range=tv shown by ffprobe looks very strange to me.

Thank you

ps: My version of ffmpeg is ffmpeg-4.2.4-1.fc32.x86_64.rpm

[Carl Eugen Hoyos]

Please test current FFmpeg git head (the only version supported on this bug tracker), please test with ffmpeg instead of ffplay and please provide your ffmpeg command line together with the complete, uncut console output to make this a valid ticket.

[airbete]

Ok, I compiled the git head version and extracted a frame with

> ./ffmpeg -i vokoscreenNG.x264_q0_high.mkv -vframes 1 output.png
ffmpeg version N-98810-ga469d29c08 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 10 (GCC)
  configuration: --enable-sdl
  libavutil      56. 58.100 / 56. 58.100
  libavcodec     58.101.100 / 58.101.100
  libavformat    58. 51.100 / 58. 51.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
Input #0, matroska,webm, from 'vokoscreenNG.x264_q0_high.mkv':
  Metadata:
    encoder         : GStreamer matroskamux version 1.16.2
    creation_time   : 2020-08-22T12:37:13.000000Z
  Duration: 00:00:04.38, start: 0.000000, bitrate: 111 kb/s
    Stream #0:0(eng): Video: h264 (High 4:4:4 Predictive), yuv444p(tv, bt470bg/smpte170m/bt709, progressive), 480x300 [SAR 1:1 DAR 8:5], 30 fps, 30 tbr, 1k tbn, 60 tbc (default)
    Metadata:
      title           : Video
    Stream #0:1(eng): Audio: vorbis, 44100 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'output.png':
  Metadata:
    encoder         : Lavf58.51.100
    Stream #0:0(eng): Video: png, rgb24, 480x300 [SAR 1:1 DAR 8:5], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      title           : Video
      encoder         : Lavc58.101.100 png
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.03 bitrate=N/A speed=1.25x    
video:28kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

The resulting output.png image is, to my surprise, perfect (no color degradation). In fact the same ffmpeg command with version 4.2.4 generates a perfect png too!

However, ffplay (git head version) still produce muddy and degraded colors. There is an unwanted color conversion that takes place somewhere...

[Carl Eugen Hoyos]

FFplay depends on an external library that we do not control, I am closing this ticket assuming it cannot be fixed in the FFmpeg codebase.

[Balling]

The color_range=tv shown by ffprobe looks very strange to me.

Why? It is Color range: Limited indeed. It is correct. All video in the world is limited)) Except Dolby Vision profile 5 if it uses IPTPQc2... You can of course use pixel format of full range.

[Balling]

So, is this SDL library we are talking about? Maybe there is an open bug there? As I understand from Mediainfo output you should dequantize it, then use Matrix coefficients: BT.601 (different from BT.709) and then use BT.709 transfer function (the same as in BT.601) and then as ColourPrimaries are not sRGB but with the same white point D65, you need to colour manage it to display, that last step is broken, let me guess?

[Balling]

As it turns out it does not support sRGB transfer on video too. Sigh. Like at all.

That is -color_primaries bt709 -color_trc iec61966_2_1 -colorspace smpte170m

What it looks like is happening here is that it cannot understand that it is BT.601 matrix!! Why? Well, we can force BT.709 matrix in mpv:
mpv.com gamma=srgb:colormatrix=bt.709:colorlevels=limited 2.mp4

where 2.mp4 is from issue #9167, that has gamma=srgb:colormatrix=bt.601:colorlevels=limited

and ffplay -i 2.mp4 -vf extractplanes=y is 120
ffplay -i 2.mp4 -vf extractplanes=u is 90
ffplay -i 2.mp4 -vf extractplanes=v is 201

and it will looks the same as in ffplay. Just saying.

[Balling]

Okay, so this is just a bug here. ​https://github.com/FFmpeg/FFmpeg/blob/573f05a7533cd9aed3ed895b4fa4ad8fcba4e56a/fftools/ffplay.c#L968

See ​https://github.com/libsdl-org/SDL/issues/2860

So frame->colorspace == AVCOL_SPC_SMPTE170M (SMPTE 170M matrix) should force that mode, but it does not.

Also why AVCOL_SPC_SMPTE240M matrix uses BT.601?? It is a different matrix as derived from SMPTE 170M primaries (yeah, SMPTE 170M matrix is not derived from SMPTE 170M primaries, I know, confusing).
P.S.
Obviously it is not primaries that define the matrix, even though it should be (not the case for JPEG (BT.601 a.k.a. SMPTE 170M matrix is derived from System M primaries ACTUALLY, see SMPTE 177 and ​https://github.com/AOMediaCodec/libavif/pull/430#discussion_r610920396) but BT.709 is the default in JPEG, since that is used in the default sRGB, also not the case for DCI-P3 in BT.2020 matrix as Netflix uses).

I think we should try to fix that. Or at least deprecate ffplaying sYCC files...

Since they migrated to github I will try to open an issue there too.

Workaround:

ffplay -i vokoscreenNG.x264_q0_high.mkv -vf scale=in_color_matrix=bt601,format=gbrp

[Balling]

FFplay also does not support linear transfer. Yes, that is RGB, not R'G'B'. While mpv does. See samples in #6725. ​https://forum.videohelp.com/attachments/43324-1507526652/dpx-sequence.zip

Wow.

[Balling]

What we should do is force -vf scale=in_color_matrix=auto,format=gbrp! It will work even for SMPTE 240M matrix and BT.2020-ncl matrix! Of course for YCbCr pixel formats it will be double converted. Will not work for YCgCo, alas. Why, BTW? Please do not tell me it is fake protection.

[Balling]

So I fixed SMPTE 240M matrix in 3249b963421cf5ff0e6adf2e38c86194fdf50929. Now, the problem as I said is in that code. SDL does not support 444 chroma correctly and that means that in SDL_SetYUVConversionMode(mode) mode will be set to SDL_YUV_CONVERSION_AUTOMATIC, which has different behaviour on linux and windows. On linux the sample here is correctly decoded because the length is correct amount of pixels for BT.601. Now, on windows it does not select BT.601, but BT.709. Who knows why? P.S. I checked linux, it is also broken, BTW, I was wrong. Oogh.

[Balling]

User came from here: ​https://github.com/vkohaupt/vokoscreenNG/issues/97

[Balling]

So how are we going to fix linear transfer? Zscale insertion? I want to add OpenGL on nvidia that supports 10 bit presentation on capable displays like in Photoshop or mpv.

[Balling]

Apparently (sic!) ffplay also cannot do full range BT.601/709/2020 and defaults to limited on 4:4:4 (which was actually in that case Y only, so gray, and for gray it does not matter whether it is 709 or 601 matrix, wow) which was found out here: #9573. Just great. All -vf scale=in_color_matrix=auto,format=gbrp stuff still works.

[Balling]

Actually, I just checked, it does not work on 4:2:0 full range either (only in mkv), but works on 4:4:4 full range. (Dolby Vision sample.) -vf scale=in_color_matrix=auto,format=gbrp works again.

P.S. I am sorry, there is no bug in Dolby Vision, my full range sample as evident from RPU was just not tagged as full range.

[Balling]

Full range is after all affected too on these samples:

​​https://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket524/AVCI50.mov
​​https://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket524/AVCI100.mov

[Balling]

e645a1ddb90a863e129108aad9aa7e2d417f3615 fixed
​​​https://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket524/AVCI50.mov
​​​https://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket524/AVCI100.mov

The main issue still needs

-vf scale=in_color_matrix=auto,format=gbrp

[Balling]

Linux is broken even further. ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220621194617.76561-1-ffmpeg@haasn.xyz/ from here

​https://github.com/mpv-player/mpv/issues/10319

(P.S. Does not fix the issue.)

[Balling]

Apparently if we have 400x576 (576 is a magic number for SD) BT.709 cannot be applied even in 4:2:0 case, see #9936.