Opened 10 months ago

Last modified 8 weeks ago

#9491 open defect

ass/subtitle filter incorectly handles colorspaces

Reported by: alwinramirus Owned by:
Priority: normal Component: avfilter
Version: git-master Keywords: ass
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Summary of the bug:
I am trying to burn in ass subtitles on to a video. The resulting video differs in color to what media players and subtitle editors display.
The video is in BT.709 limited and the ass subtitles file contains the header line YCbCr Matrix: TV.709.
What I assume happens:
The ass videofilter incorrectly converts the RGB values to BT.601 instead of BT.709 resulting in shifted colors.
How to reproduce:

% ffmpeg -i video.mp4 -vf ass=subtitles.ass out.mp4
ffmpeg version N-104454-gd92fdc7144-20211031 Copyright (c) 2000-2021 the FFmpeg developers

I have attached the files used for testing.

Attachments (3)

out.mp4 (17.9 KB ) - added by alwinramirus 10 months ago.
subtitles.ass (826 bytes ) - added by alwinramirus 10 months ago.
video.mp4 (18.4 KB ) - added by alwinramirus 10 months ago.

Download all attachments as: .zip

Change History (12)

by alwinramirus, 10 months ago

Attachment: out.mp4 added

by alwinramirus, 10 months ago

Attachment: subtitles.ass added

by alwinramirus, 10 months ago

Attachment: video.mp4 added

comment:1 by Balling, 10 months ago

RGB values to BT.601

using BT.601 matrix, not to BT.601. Anyway, I do not get it, there is nothing in ass file??

comment:2 by alwinramirus, 9 months ago

The ass files just contains a draw command to draw a green rectangle on the left side with the color (0, 245, 0). This is the same color as the video. In every media player this results in just green being displayed. Using ffmpeg (see out.mp4) there are two shades of green. Probably due to using the incorect matrix.

comment:3 by Balling, 9 months ago

0, 245, 0 is R'G'B' already, not YCbCr, so no YCBCr matrix is needed. I suppose that is c&H00F500. Well, it is not that simple then.

Last edited 9 months ago by Balling (previous) (diff)

comment:4 by alwinramirus, 9 months ago

Yes I know that those are RGB values.
This was my guess as to what happens since the resulting rectangle is drawn with the color (0, 206, 0) instead of (0, 245, 0) which is what happens if you go RGB --BT.601--> YCbCr --BT.709--> RGB.

Looking at the code it seems like drawutils.c#L163-L165 converts the RGB values to YCbCr with BT.601.

comment:5 by Balling, 9 months ago

Status: newopen

the RGB values to YCbCr with BT.601.

But then it just should do composition in matrix of the video. It has nothing to do with TV.709 line in ass file.

The bug is there though.

comment:6 by alwinramirus, 9 months ago

I am way out of my area of experties here, but as I understand it the YCbCr Matrix line is there for historical reasons to maintain backwards compatibility with old subtitles.

What is worse, VSFilter originally always used BT.601, even for BT.709 videos, which means there are many subtitles out there which have RGB colors that match the BT.709 video only if you use BT.601 for the video's YCbCr -> RGB conversion! Very ugly, indeed. Because of this problem the "YCbCr matrix" field was added to the ASS header, which allows interested parties (e.g. the video renderer) to understand which matrix the subtitle was created for, and if that happens to mismatch the video's actual encoded matrix, extra measures could be applied (e.g. by the video renderer) to correct for the mismatch.

As I understand it this is what is supposed to happen depending on the YCbCr Matrix value:

TV.601 = Output BT.601 with 16-235 YCbCr level range
TV.709 = Output BT.709 with 16-235 YCbCr level range
PC.601 = Output BT.601 with 0-255 YCbCr level range
PC.709 = Output BT.709 with 0-255 YCbCr level range
None = [Decide] based on video
If 'YCbCr Matrix' is missing from a script,[...] use TV.601 for legacy script compatibility

There are also some notes in the libass code:

comment:7 by Balling, 9 months ago

BTW, there is no way to encode any 8 bit R'G'B' triplet losslessly in YCbCr 8 bit. YCbCr is much bigger than R'G'B', even with full YCbCr and full R'G'B' YCbCr is bigger. So I do not see how it should work.

comment:8 by Gregory Beauregard, 5 months ago

I just closed (duplicate of this). In principle this problem affects all the users of the drawutils function that do conversion from RGB. So there are several other filters where the colors are wrong as well. So all of these filters that do color conversion from RGB have the same bug:

comment:9 by Balling, 8 weeks ago

Well, there is now some interesting stuff in 6c3a82f0433de8ff9c35def971a736056cc8ff38.

Note: See TracTickets for help on using tickets.