JPEG: YCCK/CMYK too green

[Balling]

Summary of the bug:

There are two color spaces that are not supported in JPEG currently, YCCK and, as used in Photo CD (see #5923), PhotoYCC. YCCK is (as said in ​https://web.archive.org/web/20170720100717/http://halicery.com/Image/jpeg/JPEGCMYK.html) YCC --> CMY and then K is added. Also there is this sample: ​https://bugs.chromium.org/p/chromium/issues/detail?id=1115101 (chromium does support it but not icc profiles).
Implementation: ​https://github.com/mozilla/mozjpeg/search?q=YCCK
Note: as said in above link K can be inverted.

Now with PhotoYCC it is a little more complicated. But I fully described it on Wikipedia, fixing the incorrect transfer function (it is the same as in xvYCC format that is IN ffmpeg). The matrix is from BT.601 but without scaling, so 1.402 (2 * (1 - K_r)) and 1.772 (2 * (1 - K_b)) are not applied. Also quantization is different. All is here: ​https://en.wikipedia.org/wiki/Photo_CD#Encoding

The implementation is also here ​https://github.com/DavidGriffith/finx/blob/17825bf28b2351f96867b34a7dc39f7997e2f9ff/nx-X11/programs/Xserver/XIE/mixie/process/mprgb.c#L1004

JPEG 2000 also supports both of them (EnumCS 13 and 9).
This produces very green picture:

% ffplay Channel_digital_image_CMYK_color.jpg

I believe this is a defect, as it is part of the jpeg decoder...
Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Balling]

YCCK file

[Carl Eugen Hoyos]

In this case, you should have mentioned ticket #3426, don't forget to mention related tickets in the future.
To make this a valid ticket, please provide the ffmpeg command line you tested together with the complete, uncut console output.

[Balling]

Well, we can reopen #3426...

ffmpeg -i Channel_digital_image_CMYK_color.jpg colour.png
ffmpeg version N-100815-g37f76c81d6 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9.3-win32 (GCC) 20200320
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --enable-shared --disable-static --disable-w32threads --enable-pthreads --enable-iconv --enable-zlib --enable-libxml2 --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-opencl --enable-libvmaf --enable-vulkan --enable-libvorbis --enable-amf --enable-libaom --enable-avisynth --enable-libdav1d --enable-libdavs2 --enable-ffnvcodec --enable-cuda-llvm --enable-libglslang --enable-libass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvpx --enable-libwebp --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libvidstab --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-libs=-lgomp
  libavutil      56. 63.101 / 56. 63.101
  libavcodec     58.119.100 / 58.119.100
  libavformat    58. 65.101 / 58. 65.101
  libavdevice    58. 11.103 / 58. 11.103
  libavfilter     7. 98.100 /  7. 98.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, image2, from 'Channel_digital_image_CMYK_color.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 146323 kb/s
    Stream #0:0: Video: mjpeg (Baseline), yuva444p(pc, bt470bg/unknown/unknown), 500x333, 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'colour.png':
  Metadata:
    encoder         : Lavf58.65.101
    Stream #0:0: Video: png, rgba(pc, bt470bg/unknown/unknown, progressive), 500x333, q=2-31, 200 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc58.119.100 png
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.267x
video:496kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

This is not cmyk, it is YCCK (also not Adobe CYYK ?)...

The main part here is PhotoYCC for PhotoCD though. ;)

As to Photo CD files you can compare how the files should look and how they look in ffmpeg. ​http://r0k.us/graphics/kodak/ Also some files should be icc managed after that.

[Balling]

I tested in Photoshop 2021: no alternative icc profile produces such insane colours and removing colour management at all also does not produce the colours as in ffmpeg. So it is not ICC.

[Elon Musk]

I compared our output with reference decoder and our output is same or even better. So please refrain from commenting something you have 0% knowledge.

[Balling]

What reference decoder you are comparing with? Also, PhotoYCC is still not supported. And also even if you somehow are saying it is correct, it is not colour managed as ffmpeg is not colour managed, so it is not correct and you know it. Sigh.

[Carl Eugen Hoyos]

What would be the reference decoder for YCCK?
Just compare with any browser of your choice...

[Carl Eugen Hoyos]

Or actually: Any browser on another operating system, sorry!

[Balling]

Replying to cehoyos:

What would be the reference decoder for YCCK?
Just compare with any browser of your choice...

Mozilla that uses ffmpeg (dah!) does open it as green, Chrome and Windows/Photoshop open it as much less green. Dunno.

[Carl Eugen Hoyos]

This shows the intended output:
​https://en.wikipedia.org/wiki/Channel_(digital_image)#/media/File:Channel_digital_image_RGB_color.jpg

[Carl Eugen Hoyos]

This bottle is supposed to be gray, not green at all:
​https://newsgroup.xnview.com/download/file.php?id=2449

[Balling]

Replying to cehoyos:

This shows the intended output:
​https://en.wikipedia.org/wiki/Channel_(digital_image)#/media/File:Channel_digital_image_RGB_color.jpg

Just it is YCC, not YCCK...

​https://newsgroup.xnview.com/download/file.php?id=2449

This file also has FOGRA27 colour profile which does not introduce any such green artifacts.

[Balling]

So, PhotoYCC uses Ci - Component identifier in ASCII 'Y','C','c' and PhotoYCC-Alpha uses CIs 'Y','C','c','a'. Yes, in ASCII, LOL. See ​https://stackoverflow.com/questions/53607147/can-this-library-detect-if-jpg-is-in-rgb-or-cmyk-format/54399410#54399410

[pdr0]

Replying to Balling:

I tested in Photoshop 2021: no alternative icc profile produces such insane colours and removing colour management at all also does not produce the colours as in ffmpeg. So it is not ICC.

Actually this is completely explained by ICC

You can strip the profile in imagemagick or similar, and you get same results as non color managed, or ffmpeg (too green) , or any other non color managed tool

eg.
convert.exe Channel_digital_image_CMYK_color.jpg -strip strip.png

Or you can convert cmyk to srgb and you get same as reference on any sRGB display. (USWebCoatedSWOP.icc is the profile most often used for cmyk)

eg.
convert.exe Channel_digital_image_CMYK_color.jpg -profile "USWebCoatedSWOP.icc" -profile "sRGB2014.icc" cmyk_to_srgb.png

You can read png tags with tweakpng, jpeg tags with jpegsnoop, or imagemagick identify, or exiftool

[pdr0]

strip color profile

[Balling]

Replying to pdr0:

Replying to Balling:

I tested in Photoshop 2021: no alternative icc profile produces such insane colours and removing colour management at all also does not produce the colours as in ffmpeg. So it is not ICC.

Actually this is completely explained by ICC

When you strip the markers it will remove Adobe APP14 marker, no? It signals YCCK. And 4 channels are now recognized as CMYK. Not YCCK. No, this has nothing to do with ICC, we have no ICC support in chrome for CMYK (or more accurately it is utterly broken after we removed Windows color API). Windows 7 color managed (that is old) photo viewer open picture differently from the modern UWP app (non-color managed app), and it looks almost like in Photoshop. And no, it does not look like you even opened Photoshop. Try to select not to color manage there, it looks nothing alike with that green picture. Decoded YCCK should be further converted using LAB (Profile Connection Space) into monitor profile using AFAIK B to AX 3DLUT. Because the color space is CMYK, it should be YCCK --> CMYK converted.

Color Space Data                : CMYK
Profile Connection Space        : Lab
B To A1                         : (Binary data 145588 bytes, use -b option to extract)
B To A2                         : (Binary data 145588 bytes, use -b option to extract)

Also you could have tried to remove icc profile from jpeg file. Exiftool should be able to do it without reencoding.

[Balling]

And I know I should have done that from the very beginning, i.e. remove the icc profile!

It can be done just by very simple

exiftool -icc_profile= FILE

Unfortunately does not work with jpeg 2000 :-(

Please download the attached file and open in old or modern Windows 10 app. BTW, ICC is giant there! It is very hard indeed to color manage correctly this stuff. So I will attach it too. You can use ICC profile inspector to see its stuff. It has correct parsing of all the stuff: ​http://www.color.org/profileinspector.xalter

[pdr0]

You're right about APP14, and it is YCCK

ISO/IEC 10918-6:2013 (E), section 6.1:

ColorTransform = 2 [YCCK]

See post nine
​https://stackoverflow.com/questions/50798014/determining-color-space-for-jpeg/50861048

Why are the photo CD PhotoYCC equations on wikipedia
​https://en.wikipedia.org/wiki/Photo_CD#Encoding

different than intel's PhotoYCC Color Model equations ?
​https://scc.ustc.edu.cn/zlsc/sugon/intel/ipp/ipp_manual/IPPI/ippi_ch6/ch6_color_models.htm

[Balling]

Replying to pdr0:

You're right about APP14, and it is YCCK

ISO/IEC 10918-6:2013 (E), section 6.1:

ColorTransform = 2 [YCCK]

See post nine
​https://stackoverflow.com/questions/50798014/determining-color-space-for-jpeg/50861048

Why are the photo CD PhotoYCC equations on wikipedia
​https://en.wikipedia.org/wiki/Photo_CD#Encoding

different than intel's PhotoYCC Color Model equations ?
​https://scc.ustc.edu.cn/zlsc/sugon/intel/ipp/ipp_manual/IPPI/ippi_ch6/ch6_color_models.htm

There is also a flag in APP14 which says whether K is inverted or not! Afaik, inverted is used in EPostscript. ISO/IEC omits that.

Now, that stackoverflow managed to omit the fact that there are also legacy YCbCr formats, that Microsoft managed to incorectly define (and thus deprecate) in their corporate version of IJG JPEG library.

As to why those are different in Intel... RGB are not between 0 and 1, that is why. Nonlinear RGB' in PhotoYCC are from -0.43357 to 1.402278, while Linear RGB values are from -0.20 to 2.00.

The decoding equations are different, because a) sRGB/(previously NIF) is EOTF in by itself, there is no EOTF for PhotoYCC. THERE also exists a possibility to further color manage it with Kodak ICC profiles, I can attach them here. But that was in .pcd files, JPEG used already defined viewing conditions as specified in ​http://www.graphcomp.com/info/specs/livepicture/fpx.pdf

There are NIF and PhotoYCC there.

[pdr0]

You can apply those equations, but you'd still get wrong colors unless you convert the ICC profile CMYK to sRGB. That sample "Channel_digital_image_CMYK_color.jpg" has a CMYK ICC profile attached

As a workaround , you can pipe imagemagick to to ffmpeg (or ffplay) , and get close results to photoshop's CMYK to sRGB conversion. There are slight differences in the conversion, slightly different sRGB icc profiles you could use. For example, you could use sRGB2014.icc vs. sRGB_v4_ICC_preference.icc vs. sRGB_ICC_v4_Appearance.icc, etc...

IM can read image sequences, but I believe it's memory bound when piped - entire sequence must fit in RAM (or at least it used to be , not sure if newer versions are any different), but it works. Or use batch conversion to individual sRGB png sequence first

eg. img_000.jpg, img_001.jpg,...img_010.jpg

convert.exe "img_%03d.jpg[0-10]" -profile "USWebCoatedSWOP.icc" -profile "sRGB2014.icc" rgb:- | ffmpeg -f rawvideo -s 500x333 -pix_fmt rgb24 -r 15 -i - -c:v utvideo -an outsequence.avi

I don't see any workarounds within ffmpeg only , and I don't see color management coming to ffmpeg any time soon

[Balling]

Replying to pdr0:

You can apply those equations

Wait a second why are you applying PhotoYCC equations? I did not attach any PhotoYCC samples. That is YCCK, it is different from PhotoYCC, completely different.

As for color management, ffmpeg is already color managed in that part, because the standard does say to convert YCCK to CMYK. Right here, but there is some bug there, I suppose. Maybe yuva internally destroys the picture, I dunno. ​https://github.com/FFmpeg/FFmpeg/blame/2c6f532e0a29527347418d2d8c4ccfe57a6ace0e/libavcodec/mjpegdec.c#L2833

[Balling]

And I was right. As you can see here: ​https://github.com/MediaArea/MediaInfo/issues/127#issuecomment-280676691

There is a regression!! Previously YCCK was recognized as yuvj444p. Now it is recognized as yuva444p! Ha. Maybe e51cc7ed856aa3d5e14c50a46d8156c79d483367 or other.

[Carl Eugen Hoyos]

You can test that the issue is not related to yuvj vs yuva:

diff --git a/libavcodec/mjpegdec.c b/libavcodec/mjpegdec.c
index 20f310fd70..5a873dfebe 100644
--- a/libavcodec/mjpegdec.c
+++ b/libavcodec/mjpegdec.c
@@ -533,8 +533,9 @@ int ff_mjpeg_decode_sof(MJpegDecodeContext *s)
                 if (s->adobe_transform == 0 && s->bits <= 8) {
                     s->avctx->pix_fmt = AV_PIX_FMT_GBRAP;
                 } else {
-                    s->avctx->pix_fmt = s->bits <= 8 ? AV_PIX_FMT_YUVA444P : AV_PIX_FMT_YUVA444P16;
-                    s->avctx->color_range = s->cs_itu601 ? AVCOL_RANGE_MPEG : AVCOL_RANGE_JPEG;
+                    s->avctx->pix_fmt = s->bits <= 8 ? AV_PIX_FMT_YUVJ444P : AV_PIX_FMT_YUVA444P16;
+//                    s->avctx->color_range = s->cs_itu601 ? AVCOL_RANGE_MPEG : AVCOL_RANGE_JPEG;
+s->avctx->color_range = AVCOL_RANGE_JPEG;
                 }
             }
             av_assert0(s->nb_components == 4);
@@ -2794,7 +2795,7 @@ the_end:
             }
         }
     }
-    if (s->adobe_transform == 2 && s->avctx->pix_fmt == AV_PIX_FMT_YUVA444P) {
+    if (s->adobe_transform == 2 && s->avctx->pix_fmt == AV_PIX_FMT_YUVJ444P) {
         int w = s->picture_ptr->width;
         int h = s->picture_ptr->height;
         av_assert0(s->nb_components == 4);
diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c
index 2a919461a5..1c192454e6 100644
--- a/libavutil/pixdesc.c
+++ b/libavutil/pixdesc.c
@@ -403,13 +403,14 @@ static const AVPixFmtDescriptor av_pix_fmt_descriptors[AV_PIX_FMT_NB] = {
     },
     [AV_PIX_FMT_YUVJ444P] = {
         .name = "yuvj444p",
-        .nb_components = 3,
+        .nb_components = 4,
         .log2_chroma_w = 0,
         .log2_chroma_h = 0,
         .comp = {
             { 0, 1, 0, 0, 8, 0, 7, 1 },        /* Y */
             { 1, 1, 0, 0, 8, 0, 7, 1 },        /* U */
             { 2, 1, 0, 0, 8, 0, 7, 1 },        /* V */
+{ 3, 1, 0, 0, 8, 0, 7, 1 },
         },
         .flags = AV_PIX_FMT_FLAG_PLANAR,
     },

[Balling]

Okay my latest idea is that it is subsampling issue, subsampled CMYK and YCCK (yeah, CMYK is also insanely affected, at least that uses in Ci endcoding, see #4772) are good and not subsampled are bad. That is regression of d56c373391d334d3dc29c752955fd17f0cef7ff9.

Franky speaking I am starting to fear that this is just a problem of not limiting CMYK green values... CMYK is not a triange or prism (in XYZ) and so we get that problem. Green should be limited to much lower values.

I also found good test here: ​https://github.com/mozilla/mozjpeg/issues/23

Also, we will try to fix this in Chromium, and if it will work with our native jpeg decoder, we will try to get to fallback (that is ffmpeg).

[Balling]

BTW, I found YCCK with inverted colors as promised. Extracted from PDF with poppler's

pdfimages -all

There is an "open" issue in poppler: ​https://gitlab.freedesktop.org/cairo/cairo/-/issues/156

So, pdf uses "/Decode [ 1 0 1 0 1 0 1 0 ]" to signal not normal K component.

The images should look like here on the cover: ​https://www.amazon.com/Lewins-CELLS-George-Plopper-ebook/dp/B00JKPGLEA

There is nothing in APP14 that can show that, unfortunately, I though it should be flags1, but nope. Sigh. Acrobat extracts them by converting to YCbCr.
You know where to find a pdf ebook, right? D:)

You know I think it is just wrong sRGB conversion if it is what FFmpeg is targeting. See: ​https://graphicdesign.stackexchange.com/questions/130658/conversion-from-cmyk-to-rgb-everything-becomes-greener-sometimes?rq=1 and ​https://graphicdesign.stackexchange.com/questions/61294/why-do-professional-print-houses-use-gamut-limiting-cmyk

As always download and open in normal viewer. I also renaming the issue since that sample is just CMYK.

[Balling]

Interesting moment here: mpv with gpu-next warns that Little CMS cannot get it:

mpv.com --vo=gpu-next McNett-mirazyme.jpg

[vo/gpu-next] lcms2: [9] Wrong input color space on transform
[vo/gpu-next] Failed creating CMS transform!

[Balling]

I found how the inverted K is signalled: ​https://bugzilla.mozilla.org/show_bug.cgi?id=674619#c7

APP12 'EMBED\x00' (in hex: FF EC 00 08 45 4D 42 45 44 00)

​https://wiki.mozilla.org/PDF.js/EMBED

P.S. Apparently the normal K is in pdf and eps, standalone has inverted K, see ​https://github.com/mm2/Little-CMS/blob/1f6a2adbc885b326a8fabe96ac0aa7e73369413c/utils/jpgicc/jpgicc.c#L662-L663