libx264 ignores color range flag for gray10 input

[Diederick Niehorster]

When transcoding a file containing gray10 video data in full (pc) color range with libx264, the output seems to be in limited (tv) color range (i.e. image got a lot darker), despite encoder correctly identifying the pc color range. Feel free to add this file to FATE if its an interesting test case.

This is with ffmpeg-2021-12-27-git-617452ce2c-full_build from gyan.dev, but has been present for much longer. libx264 version: core 164 r3079 d9a19f0.

ffmpeg -y -i test.mkv -c:v libx264 -preset veryfast -crf 0 test.mp4

Same thing happens with crf 17, and also when adding -color_range 2 to the command.

relevant part of ffmpeg output (i trimmed it):

Input #0, matroska,webm, from 'test.mkv':
  Metadata:
    ENCODER         : Lavf59.10.100
  Duration: 00:00:00.02, start: 0.000000, bitrate: 1142412 kb/s
  Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gray10le(pc), 1152x390, 500 fps, 500 tbr, 1k tbn (default)
    Metadata:
      DURATION        : 00:00:00.018000000
Stream mapping:
  Stream #0:0 -> #0:0 (ffv1 (native) -> h264 (libx264))
[libx264 @ 00000237f24afe40] profile High 10, level 5.1, 4:0:0, 10-bit
[libx264 @ 00000237f24afe40] Output #0, mp4, to 'test.mp4':
  Metadata:
    encoder         : Lavf59.10.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), gray10le(pc, progressive), 1152x390, q=2-31, 500 fps, 16k tbn (default)
    Metadata:
      DURATION        : 00:00:00.018000000
      encoder         : Lavc59.15.101 libx264

When probing the resulting file, i get (note that pixel format is identified as yuv420p10le, not gray10le):

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.10.100
  Duration: 00:00:00.02, start: 0.000000, bitrate: 427913 kb/s
  Stream #0:0[0x1](und): Video: h264 (High 10) (avc1 / 0x31637661), yuv420p10le(pc, progressive), 1152x390, 427483 kb/s, 500 fps, 500 tbr, 16k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]

ffmpeg banner:

ffmpeg version 2021-12-27-git-617452ce2c-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11.2.0 (Rev2, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      57. 13.100 / 57. 13.100
  libavcodec     59. 15.101 / 59. 15.101
  libavformat    59. 10.100 / 59. 10.100
  libavdevice    59.  1.100 / 59.  1.100
  libavfilter     8. 21.100 /  8. 21.100
  libswscale      6.  1.102 /  6.  1.102
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100

[Diederick Niehorster]

to reproduce report

[Diederick Niehorster]

Since ffprobe identifies the encoded pixel format as yuv420p10le, i decided to see if things do look fine if that the input i provide to the encoder. Command:

ffmpeg -y -i test.mkv -vf "format=yuv420p10" -c:v libx264 -preset veryfast -crf 0 test.mp4

Then things perceptually look fine (but see below), despite the color range flag apparently being lost, according to ffmpeg output:

Output #0, mp4, to 'test.mp4':
  Metadata:
    encoder         : Lavf59.10.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p10le(tv, progressive), 1152x390, q=2-31, 500 fps, 16k tbn (default)
    Metadata:
      DURATION        : 00:00:00.018000000
      encoder         : Lavc59.15.101 libx264

However, encoding yuv420p10le (or yuv444p10le for that matter) does not seem to be lossless despite crf 0. The conversion gray10le->yuv420p10le (or gray10le->yuv444p10le) is not perfect, but leads to pixel intensity errors of only +1 or -1 value on 14.11% of pixels in my example input (when total conversion actually is gray10le->yuv420p10le->gray10le, so not really a problem, some roundoff error i guess). In the encoded file however 68.66% of pixels have the wrong intensity value, with errors ranging from -5 to +5 intensity values. I assume this is not expected? Should i post a separate ticket for that?

[pdr0]

Replying to Diederick Niehorster:

Since ffprobe identifies the encoded pixel format as yuv420p10le, i decided to see if things do look fine if that the input i provide to the encoder. Command:

ffmpeg -y -i test.mkv -vf "format=yuv420p10" -c:v libx264 -preset veryfast -crf 0 test.mp4

Then things perceptually look fine (but see below), despite the color range flag apparently being lost, according to ffmpeg output:

Output #0, mp4, to 'test.mp4':
  Metadata:
    encoder         : Lavf59.10.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p10le(tv, progressive), 1152x390, q=2-31, 500 fps, 16k tbn (default)
    Metadata:
      DURATION        : 00:00:00.018000000
      encoder         : Lavc59.15.101 libx264

However, encoding yuv420p10le (or yuv444p10le for that matter) does not seem to be lossless despite crf 0. The conversion gray10le->yuv420p10le (or gray10le->yuv444p10le) is not perfect, but leads to pixel intensity errors of only +1 or -1 value on 14.11% of pixels in my example input (when total conversion actually is gray10le->yuv420p10le->gray10le, so not really a problem, some roundoff error i guess). In the encoded file however 68.66% of pixels have the wrong intensity value, with errors ranging from -5 to +5 intensity values. I assume this is not expected? Should i post a separate ticket for that?

use full range

(gray uses full range, but your yuv420p10 conversion uses limited range)

ffmpeg -y -i test.mkv -vf "scale=in_range=pc:out_range=pc,format=yuv420p10" -c:v libx264 -preset veryfast -crf 0 test2.mp4

[Diederick Niehorster]

Replying to pdr0:

Replying to Diederick Niehorster:

However, encoding yuv420p10le (or yuv444p10le for that matter) does not seem to be lossless despite crf 0. The conversion gray10le->yuv420p10le (or gray10le->yuv444p10le) is not perfect, but leads to pixel intensity errors of only +1 or -1 value on 14.11% of pixels in my example input (when total conversion actually is gray10le->yuv420p10le->gray10le, so not really a problem, some roundoff error i guess). In the encoded file however 68.66% of pixels have the wrong intensity value, with errors ranging from -5 to +5 intensity values. I assume this is not expected? Should i post a separate ticket for that?

use full range

(gray uses full range, but your yuv420p10 conversion uses limited range)

ffmpeg -y -i test.mkv -vf "scale=in_range=pc:out_range=pc,format=yuv420p10" -c:v libx264 -preset veryfast -crf 0 test2.mp4

Thanks. Using that command, both the output stream of ffmpeg and ffprobe now correctly identify the data as full range. However, the output in terms of pixel values is identical (both conversions apparently were full range, even if that info did not survive the filter chain without your addition). My problems thus remain:

1. libx264 with full range gray10 input seems to ignore the full range flag (also when adding a filter -vf scale=in_range=pc:out_range=pc to the command, i just tried)
2. libx264 with crf 0 and pix_fmt yuv420p10le is not lossless, but shows many pixels that are off by up to 5 intensity values (should this be reported separately?)

[Balling]

This is a duplicate of that issue about gray being always full range, it was a fix in ffmpeg actually.

[Diederick Niehorster]

Replying to Balling:

This is a duplicate of that issue about gray being always full range, it was a fix in ffmpeg actually. See my closing comment there ​https://trac.ffmpeg.org/ticket/2452#comment:20

I do not see how it is a duplicate. Here a source is actually full range, but treated as limited range by the encoder regardless.
Note that it appears without using ffmpeg, also when i directly use the API (which is how i came across it).

I have an input that is actually full-range, and identified as such in the AVFrame. Full range is set also in the encoder context. Yet luminance is shifted down. To be fair, i do not know if this in the libx264 encoder, or in the h264 decoder which i have to use to read the file back in. But with ffv1 stored in an mkv i can do lossless roundtrip with ffmpeg and with the API, suggesting its something specific to libx264/h264. Note also that ffprobe does recognize the stream as full range (see output in problem report).

[pdr0]

Replying to Diederick Niehorster:

1. libx264 with full range gray10 input seems to ignore the full range flag (also when adding a filter -vf scale=in_range=pc:out_range=pc to the command, i just tried)
2. libx264 with crf 0 and pix_fmt yuv420p10le is not lossless, but shows many pixels that are off by up to 5 intensity values (should this be reported separately?)

test2.mp4 (and test3.mp4) has full range flag - ffmpeg yuv420p10le(pc) . Mediainfo also specifies full range flag.

It looks like something changed with libx264; -qp 0 is lossless, but -crf 0 is not

Even the filesize is almost 3x larger with -qp 0 vs. -crf 0

test3.mp4 is lossless, psnr inf

ffmpeg -y -i test.mkv -vf "scale=in_range=pc:out_range=pc,format=yuv420p10" -c:v libx264 -preset veryfast -qp 0 test3.mp4 -y

ffmpeg -i test3.mp4 -i test.mkv -lavfi  "[0:v]scale=in_range=pc:out_range=pc,format=gray10le,settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

[Parsed_psnr_6 @ 000000da162e95c0] PSNR y:inf average:inf min:inf max:inf

If you are using ffplay, you need to force full range rgb or gbrp playback with swscale or zscale to be consistent. It doesn't necessarily automatically acknowlege the full range flag for the RGB playback conversion

ffplay -vf "scale=in_range=pc, format=gbrp" -i test3.mp4

[pdr0]

extractplanes=y is probably more "elegant" than scale with format conversion . The 10bit Y plane is equivalent to the gray10 (just discarding CbCr planes)

ffmpeg -i test3.mp4 -i test.mkv -lavfi  "[0:v]extractplanes=y,settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

PSNR y:inf average:inf min:inf max:inf

ffplay -vf extractplanes=y -i test3.mp4

[Balling]

-crf 0 was never supposed to be lossless with x264, only for 8 bit that is true! For 10 bit -qp 0 is needed. I thought they fixed it in some version of x264 but apparently not. See ​https://github.com/BroneKot/xvid4psp/issues/23#issuecomment-160890507

CRF of minus 12 (-12) is lossless for 10 bit AFAIK. Again, they should have fixed that long time ago...

[Diederick Niehorster]

Thanks both!

It seems that the docs here ​https://trac.ffmpeg.org/wiki/Encode/H.264 are wrong then, it says crf 0 is lossless.

I can indeed verify that

ffmpeg -y -i test.mkv -vf "scale=in_range=pc:out_range=pc,format=yuv420p10" -c:v libx264 -preset veryfast -qp 0 test3.mp4

leads to a losslessly encoded file (yields zero pixel intensity value errors), super!

Using your commands i have furthermore found that the workaround is not necessary. full-range gray10 is encoded correctly, it was a playback issue:

ffmpeg -y -i test.mkv -c:v libx264 -preset veryfast -qp 0 test3.mp4

followed by reading in the file with either "-vf extractplanes=y" or "-vf scale=in_range=pc" yields zero pixel intensity value errors in the output: input and output are identical.

Note here that with reading in, i mean directly through the api (demux+decode). Its annoying that the full range info is lost by the automatically inserted filter turning the yuv420p10 back into gray10 / some rgb format and "-vf scale=in_range=pc" is thus needed. I'll dig into this to see if its set on the AVFrame delivered by the decoder, etc. Seems to me like this info should not get lost if its available.

[Balling]

This is still strange. I thought that crf of -12 is mapped to 0 in ffmpeg for 10 bit from 0 to 63, compared to 0 to 51 for 8 bit. See ​https://code.videolan.org/videolan/x264/-/blob/master/x264.c#L733 so it is from -12 to 51 there.

Okay, I got it: lossless mode is only High 4:4:4 Predictive@L5.1 (-qp 0), High 10@L5.1 (-crf 0) is not lossless. So that is why! Hahaha. ​https://code.videolan.org/videolan/x264/-/blob/master/x264.c#L579

[Balling]

I fixed wiki in ​https://trac.ffmpeg.org/wiki/Encode/H.264?action=diff&version=81 and also "Note that usage of -profile:v is incompatible with lossless encoding and setting -profile:v high444 does not work." Why is that??? Is this a bug?

[Balling]

This is a duplicate of #8862, use mpv or ffplay.exe -i test.mp4 -vf scale=in_color_matrix=auto,format=gbrp

I.e. there is no bug in any range flags or conversions, ffplay just cannot read full range BT.601 on 4:4:4 file result without some help. It is better to just use mpv. (It does not matter whether it is BT.601 or BT.709 for gray BTW, since both are decoded the same for Y only component, but since it is not clarified ffmpeg selects BT.601 and here we get ffplay bug with 4:4:4 #8862.)

Also I will point out that -vf extractplanes does not appear to really work with this file:

ffmpeg.exe -i test.mkv -vf extractplanes=y -an -frames:v 1 nhcvbqaw.png
produces garbage.

[Diederick Niehorster]

Thanks! Seems you do not have to set a profile, the right one is automatically selected:

ffmpeg -y -i test.mkv -c:v libx264 -preset veryfast -qp 0 test3.mp4

results in:

[libx264 @ 0000021d6b51fdc0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0000021d6b51fdc0] profile High 4:4:4 Predictive, level 5.1, 4:0:0, 10-bit
[libx264 @ 0000021d6b51fdc0] 264 - core 164 r3079 d9a19f0 - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=2 psy=0 mixed_ref=0 me_range=16 chroma_me=0 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=12 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=1 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0
Output #0, mp4, to 'test3.mp4':
  Metadata:
    encoder         : Lavf59.10.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), gray10le(pc, progressive), 1152x390, q=2-31, 500 fps, 16k tbn (default)
    Metadata:
      DURATION        : 00:00:00.018000000
      encoder         : Lavc59.15.101 libx264

note that profile High 4:4:4 Predictive, level 5.1, 4:0:0, 10-bit is selected automatically.

Anyway, its clear now that this is not in libx264, or on the encoding side, at all. It also has nothing to do with mkv (problem shows itself in mp4 also). Nor is it even in the decoding side, i have confirmed that the h264 decoder correctly delivers AVFrames with color_range=2 (2 is JPEG/pc) set on them (and also on the decoder context). The problem is with the auto-inserted scale filter used for format conversion (yuv420p10->gray10, used since data is stored in yuv420p10 in the file) that does not honor the input color_range, unless told what it is with scale=in_range=pc (selecting gray10 as output automatically sets out_range to pc, no need to specify that in the command). I'm not sure if thats a bug or something that can be requested as an enhancement. I'll check out the ticket you linked to, and add to that or post a new one.

It interesting that
ffmpeg.exe -y -i test.mkv -vf extractplanes=y -frames:v 1 nhcvbqaw.png
produces garbage. The data in there is after all gray10, and thus in a sense only contains a y plane. Extracting a y plane when there only is a y plane is perhaps a bit non-sensical, but probably should work. I'll file a ticket.

[Balling]

Seems you do not have to set a profile, the right one is automatically selected:

That is what I meant! Profile high444 predictive is only forced by qp 0, it cannot be forced by -profile:v high444, also crf 0 does not force high444 profile and thus is not lossless. Maybe the cause is deprecation of high 444 itself in ITU spec in favour of predictive.

Extracting a y plane when there only is a y plane is perhaps a bit non-sensical

Well, actually you're not supposed to get black for white and white for black at the very least. That is to you Paul, FFV1 does not work with your extractplanes.

what it is with scale=in_range=pc

No, the bug is that SDL2 used in ffplay cannot play YCbCr in 4:4:4 variant in full range. It does respect full range and sees it in mkv metadata and does correctly convert and set output metadata. Unfortunately for you, ffplay does not play the result correctly.

[Diederick Niehorster]

Replying to Balling:

what it is with scale=in_range=pc

No, the bug is that SDL2 used in ffplay cannot play YCbCr in 4:4:4 variant in full range. It does respect full range and sees it in mkv metadata and does correctly convert and set output metadata. Unfortunately for you, ffplay does not play the result correctly.

I see the exact same thing when using ffmpeg with a full range mp4, or when using the API directly, so its not an ffplay problem (though there may be other problems there, don't know about that).

[Balling]

-crf 0 was never supposed to be lossless with x264 [and ffmpeg too], only for 8 bit that is true! For 10 bit -qp 0 is needed

I found out where I saw it first quite long time ago: ​https://forum.videohelp.com/threads/382367-X265-Lossless-not-really-lossless-and-I-can-prove-it#post2475951 Just will leave it here for posterity.