ffmpeg not reading color information from JPEG ICC profile

[Cosmin Stejerean]

ffmpeg / ffprobe is not reading color information from the ICC profile in JPEG. Somewhat related to #9672 (where it also fails to write it to JPEG), for proper support of wide color gamut it's important to be able to both read and write color information properly. This ticket focuses on the reading part.

Attached is an example image in P3 color space (which needs a P3 capable display to render properly). ffprobe is not able to read the ICC color information from this, but it can be read by for example exiftool

Steps to reproduce.

1. Download the attached test.jpg

2. Can confirm the ICC profile info with exiftool

exiftool test.jpg

Profile Description             : Display P3

additional output of exiftool is omitted for brevity

2. Check ffprobe color information

ffprobe -show_streams test.jpg | grep color

color_range=pc
color_space=bt470bg
color_transfer=unknown
color_primaries=unknow

Expected values are color primaries smpte432, color trc iec61966-2-1

[haasn]

As of 2cb0cebd11eb90dfcccac5c258af1003bd4f17d2, ffmpeg is capable of parsing embedded ICC profiles and inferring the colorimetry tags. Currently, this is not automatic, but must be manually requested by inserting a -vf iccdetect filter.

Note that this relies on reasonably well-behaved ICC profiles - if the ICC profile deviates too strongly from the set of known primaries and transfer curve, the tags will remain as 'undefined'.

All ICC profiles used in practice should be well-behaved in this regard, but YMMV.

[haasn]

All ICC profiles used in practice should be well-behaved in this regard, but YMMV.

Oh, actually, it's worth pointing out that the test image attached to OP is *not* actually a Display P3 profile, despite the profile comment seemingly claiming otherwise. (It's actually a profile with an sRGB tone curve and custom primaries)

[Balling]

Replying to haasn:

All ICC profiles used in practice should be well-behaved in this regard, but YMMV.

Oh, actually, it's worth pointing out that the test image attached to OP is *not* actually a Display P3 profile, despite the profile comment seemingly claiming otherwise. (It's actually a profile with an sRGB tone curve and custom primaries)

You are absolutely wrong. Display P3 is Apple profile that uses sRGB EOTF and P3-D65 primaries, not custom primaries, of course in ICCv4 adapted to D50, so you need to chromatically adapt the primaries first. They are not custom or whatever.

But then again, you also believed that CLL cannot change in mp4 file, I proved you wrong.

Profile Description : Display P3
Profile Copyright : Copyright Apple Inc., 2015
Media White Point : 0.95045 1 1.08905
Red Matrix Column : 0.51512 0.2412 -0.00105
Green Matrix Column : 0.29198 0.69225 0.04189
Blue Matrix Column : 0.1571 0.06657 0.78407
Chromatic Adaptation : 1.04788 0.02292 -0.0502 0.02959 0.99048 -0.01706 -0.00923 0.01508 0.75168

these primaries perfectly describe P3-D65 after you apply Chromatic Adaptation matrix.

Red Tone Reproduction Curve : (Binary data 32 bytes, use -b option to extract)
Blue Tone Reproduction Curve : (Binary data 32 bytes, use -b option to extract)
Green Tone Reproduction Curve : (Binary data 32 bytes, use -b option to extract)

These values describe sRGB curve using "para" encoding per every component above:

0 2.399994
1 0.947861
2 0.052139
3 0.077393
4 0.040451

This is the same as reference sRGB curve.

[haasn]

See ​https://github.com/mm2/Little-CMS/issues/316

[Balling]

Yes, Apple profile is quite buggy, it has not perfect chromatic adaption matrix and it has negative red Z, which is prohibited and caused all kinds of problems (not on MacOS) in our chromium and firefox, though is correctly done by Little CMS stealing it from Adobe CMS/Apple. It is not like with V-gamut though in mpv code, that has "wrong" negative values even natively, while red Z negative in Display P3 comes from slightly wrong chromatic adapation done by Apple (it is also xy, not XYZ, see your commit ​https://github.com/mpv-player/mpv/commit/247ec0cb841c28b97a186675c0ef923a0ede2f40).

Even sRGB para curve can be made slightly better if only you will round correctly. See repo Compact-ICC-Profiles. It also has correct profiles without negative stuff in red Z.

See my comments here: ​https://github.com/saucecontrol/Compact-ICC-Profiles/issues/7#issuecomment-943567373

Here: ​https://github.com/saucecontrol/Compact-ICC-Profiles/issues/6

And here: ​https://github.com/colour-science/colour/discussions/739

Still you need to understand THAT MEDIA WHITE POINT IS NOT the white point of PROFILE CONNECTION SPACE, though apple did a mistake there as well. PCS is ONLY D50, nothing else is supported untill v5. So you can just ignore it (not for printing, complex relationship is described ​https://github.com/saucecontrol/Compact-ICC-Profiles/issues/7#issuecomment-943567373).

There is also a problem that display profile is used for what should be an input profile, JPEG 2000 part does not allow that, while part two allowed that.

[Cosmin Stejerean]

This is a great start, but I'm curious what it would take to do this automatically. For example it would be useful to have the color information exposed in ffprobe, and having to inject a filter into the process would make this more cumbersome.

[haasn]

This is a great start, but I'm curious what it would take to do this automatically. For example it would be useful to have the color information exposed in ffprobe, and having to inject a filter into the process would make this more cumbersome.

As with the other issue, -flags2 icc_profiles enables this experimental functionality on master:

$ ffprobe -flags2 icc_profiles -show_frames test.jpg | grep color
...
color_range=pc
color_space=bt470bg
color_primaries=unknown
color_transfer=iec61966-2-1

Note that you need to use -show_frames and not -show_streams as a result of the ICC profile and hence color space metadata being internally a property of the frame, not the container. (I'll continue thinking about ways we could possibly improve that)

Let me know if your use case is solved.