ass/subtitle filter incorectly handles colorspaces

[alwinramirus]

Summary of the bug:
I am trying to burn in ass subtitles on to a video. The resulting video differs in color to what media players and subtitle editors display.
The video is in BT.709 limited and the ass subtitles file contains the header line YCbCr Matrix: TV.709.
What I assume happens:
The ass videofilter incorrectly converts the RGB values to BT.601 instead of BT.709 resulting in shifted colors.
How to reproduce:

% ffmpeg -i video.mp4 -vf ass=subtitles.ass out.mp4
ffmpeg version N-104454-gd92fdc7144-20211031 Copyright (c) 2000-2021 the FFmpeg developers

I have attached the files used for testing.

[Balling]

RGB values to BT.601

using BT.601 matrix, not to BT.601. Anyway, I do not get it, there is nothing in ass file??

[alwinramirus]

The ass files just contains a draw command to draw a green rectangle on the left side with the color (0, 245, 0). This is the same color as the video. In every media player this results in just green being displayed. Using ffmpeg (see out.mp4) there are two shades of green. Probably due to using the incorect matrix.

[Balling]

0, 245, 0 is R'G'B' already, not YCbCr, so no YCBCr matrix is needed. I suppose that is c&H00F500. Well, it is not that simple then.

[alwinramirus]

Yes I know that those are RGB values.
This was my guess as to what happens since the resulting rectangle is drawn with the color (0, 206, 0) instead of (0, 245, 0) which is what happens if you go RGB --BT.601--> YCbCr --BT.709--> RGB.

Looking at the code it seems like drawutils.c#L163-L165 converts the RGB values to YCbCr with BT.601.

[Balling]

the RGB values to YCbCr with BT.601.

But then it just should do composition in matrix of the video. It has nothing to do with TV.709 line in ass file.

The bug is there though.

[alwinramirus]

I am way out of my area of experties here, but as I understand it the YCbCr Matrix line is there for historical reasons to maintain backwards compatibility with old subtitles.

What is worse, VSFilter originally always used BT.601, even for BT.709 videos, which means there are many subtitles out there which have RGB colors that match the BT.709 video only if you use BT.601 for the video's YCbCr -> RGB conversion! Very ugly, indeed. Because of this problem the "YCbCr matrix" field was added to the ASS header, which allows interested parties (e.g. the video renderer) to understand which matrix the subtitle was created for, and if that happens to mismatch the video's actual encoded matrix, extra measures could be applied (e.g. by the video renderer) to correct for the mismatch.

​https://github.com/libass/libass/issues/297#issuecomment-360126487

As I understand it this is what is supposed to happen depending on the YCbCr Matrix value:

TV.601 = Output BT.601 with 16-235 YCbCr level range
TV.709 = Output BT.709 with 16-235 YCbCr level range
PC.601 = Output BT.601 with 0-255 YCbCr level range
PC.709 = Output BT.709 with 0-255 YCbCr level range
None = [Decide] based on video
If 'YCbCr Matrix' is missing from a script,[...] use TV.601 for legacy script compatibility

​https://www.videohelp.com/software/xy-VSFilter/version-history

There are also some notes in the libass code: ​https://github.com/libass/libass/blob/49f116ab1fa4386f6a5191f322ac29872279516e/libass/ass_types.h#L105-L156

[Balling]

BTW, there is no way to encode any 8 bit R'G'B' triplet losslessly in YCbCr 8 bit. YCbCr is much bigger than R'G'B', even with full YCbCr and full R'G'B' YCbCr is bigger. So I do not see how it should work.

[Gregory Beauregard]

I just closed ​https://trac.ffmpeg.org/ticket/9682 (duplicate of this). In principle this problem affects all the users of the drawutils function that do conversion from RGB. So there are several other filters where the colors are wrong as well. So all of these filters that do color conversion from RGB have the same bug: ​https://github.com/FFmpeg/FFmpeg/search?q=ff_draw_color

[Balling]

Well, there is now some interesting stuff in 6c3a82f0433de8ff9c35def971a736056cc8ff38.

[nanone]

Well, there is now some interesting stuff in 6c3a82f0433de8ff9c35def971a736056cc8ff38.

I've tested tag 5.1.2 which includes those changes, but without avail. Seen a similar issue on ​https://superuser.com/questions/1757284 also denoting issues with ASS subtitles and incorrect colors.

There has been little activity on this issue for quite some time, but the problem still persist. What's the proper way to deal with this problem?