lutrgb not clipping all pixels

[chris319]

I am trying to convert yuv420p pixels to rgb24 and confine the range of the rgb24 pixels to 16 - 235 using lutrgb.

It works semi successfully but lutrgb leaves behind some errant pixels which are > 235 or < 16.

Command line:

ffmpeg -y -i "D:\Videos\SonyAX53\C0059.mp4" -pix_fmt rgb24 -c:v libx264 -vf "lutrgb=r='clip(val,16,235)':g='clip(val,16,235)':b='clip(val,16,235)'" -c:a pcm_s16be RGB.mov

=================================================================================================
Console output:

D:\Programs\ffmpeg\Batch_files>ffmpeg -y -i "D:\Videos\SonyAX53\C0059.mp4" -pix_fmt rgb24 -c:v libx264 -vf "lutrgb=r='clip(val,16,235)':g='clip(val,16,235)':b='clip(val,16,235)'" -c:a pcm_s16be RGB.mov
ffmpeg version 2022-02-10-git-b6bb6b9f22-essentials_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers

built with gcc 11.2.0 (Rev7, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
libavutil 57. 21.100 / 57. 21.100
libavcodec 59. 20.100 / 59. 20.100
libavformat 59. 17.101 / 59. 17.101
libavdevice 59. 5.100 / 59. 5.100
libavfilter 8. 26.101 / 8. 26.101
libswscale 6. 5.100 / 6. 5.100
libswresample 4. 4.100 / 4. 4.100
libpostproc 56. 4.100 / 56. 4.100

Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'D:\Videos\SonyAX53\C0059.mp4':

Metadata:

major_brand : XAVC
minor_version : 16785407
compatible_brands: XAVCmp42iso2
creation_time : 2018-10-18T21:02:07.000000Z

Duration: 00:00:14.51, start: 0.000000, bitrate: 51416 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 49386 kb/s, 59.94 fps, 59.94 tbr, 60k tbn (default)

Metadata:

creation_time : 2018-10-18T21:02:07.000000Z
handler_name : Video Media Handler
vendor_id : [0][0][0][0]
encoder : AVC Coding

Stream #0:1[0x2](und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)

Metadata:

creation_time : 2018-10-18T21:02:07.000000Z
handler_name : Sound Media Handler
vendor_id : [0][0][0][0]

Stream #0:2[0x3](und): Data: none (rtmd / 0x646D7472), 491 kb/s (default)

Metadata:

creation_time : 2018-10-18T21:02:07.000000Z
handler_name : Timed Metadata Media Handler
timecode : 00:41:30:58

Stream mapping:

Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (pcm_s16be (native) -> pcm_s16be (native))

Press [q] to stop, ? for help
Incompatible pixel format 'rgb24' for codec 'libx264', auto-selecting format 'yuv444p'
[libx264 @ 000001d8799a0340] using SAR=1/1
[libx264 @ 000001d8799a0340] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 000001d8799a0340] profile High 4:4:4 Predictive, level 4.2, 4:4:4, 8-bit
[libx264 @ 000001d8799a0340] 264 - core 164 r3086 5585eaf - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - ​http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mov, to 'RGB.mov':

Metadata:

major_brand : XAVC
minor_version : 16785407
compatible_brands: XAVCmp42iso2
encoder : Lavf59.17.101

Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv444p(tv, unknown/bt709/iec61966-2-4, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 59.94 fps, 60k tbn (default)

Metadata:

creation_time : 2018-10-18T21:02:07.000000Z
handler_name : Video Media Handler
vendor_id : [0][0][0][0]
encoder : Lavc59.20.100 libx264

Side data:

cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A

Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)

Metadata:

creation_time : 2018-10-18T21:02:07.000000Z
handler_name : Sound Media Handler
vendor_id : [0][0][0][0]
encoder : Lavc59.20.100 pcm_s16be

frame= 870 fps= 32 q=-1.0 Lsize= 17855kB time=00:00:14.51 bitrate=10080.2kbits/s speed=0.532x
video:15113kB audio:2721kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.121322%
[libx264 @ 000001d8799a0340] frame I:4 Avg QP:22.93 size:296530
[libx264 @ 000001d8799a0340] frame P:335 Avg QP:25.58 size: 38933
[libx264 @ 000001d8799a0340] frame B:531 Avg QP:30.86 size: 2347
[libx264 @ 000001d8799a0340] consecutive B-frames: 1.6% 50.8% 0.7% 46.9%
[libx264 @ 000001d8799a0340] mb I I16..4: 10.9% 48.9% 40.1%
[libx264 @ 000001d8799a0340] mb P I16..4: 0.4% 0.8% 0.2% P16..4: 30.2% 5.8% 6.8% 0.0% 0.0% skip:55.7%
[libx264 @ 000001d8799a0340] mb B I16..4: 0.0% 0.1% 0.0% B16..8: 21.1% 0.5% 0.1% direct: 0.1% skip:77.9% L0:46.7% L1:47.1% BI: 6.2%
[libx264 @ 000001d8799a0340] 8x8 transform intra:52.7% inter:50.0%
[libx264 @ 000001d8799a0340] coded y,u,v intra: 50.0% 22.7% 20.4% inter: 8.7% 1.2% 1.0%
[libx264 @ 000001d8799a0340] i16 v,h,dc,p: 27% 38% 18% 17%
[libx264 @ 000001d8799a0340] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 24% 34% 4% 3% 3% 5% 3% 7%
[libx264 @ 000001d8799a0340] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 28% 15% 5% 6% 6% 5% 5% 6%
[libx264 @ 000001d8799a0340] Weighted P-Frames: Y:2.1% UV:1.2%
[libx264 @ 000001d8799a0340] ref P L0: 69.5% 15.5% 11.4% 3.5% 0.1%
[libx264 @ 000001d8799a0340] ref B L0: 90.6% 8.1% 1.3%
[libx264 @ 000001d8799a0340] ref B L1: 95.6% 4.4%
[libx264 @ 000001d8799a0340] kb/s:8529.23

[Balling]

We do not support limited range RGB, it is an open issue. #1851

(It is prohibited in the code.)

confine the range of the rgb24 pixels to 16 - 235 using lutrgb.

Why do you do this anyway?

Second of all, lossy x264 will create files not perfect and containing out of the needed range values. Use lossless x264 with -qp 0.

[chris319]

To answer the question of "why":

EBU R 103 states the following:

Signals shall not exceed the “Total Video Signal Range”

"Total video signal range" is given as 1 - 254 in 8-bit video.

​https://tech.ebu.ch/docs/r/r103.pdf

Per ITU-R BT.709-6, the 8-bit values of 0 and 255 are reserved for sync.

​https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.709-6-201506-I!!PDF-E.pdf

A file which does not conform to these specs could fail to pass a broadcaster's QC.

The following code leaves behind pixels outside the 8-bit range of 16 - 235, including pixels of 0 and 255.

ffmpeg -y -i "D:\Videos\SonyAX53\C0059.mp4" -pix_fmt rgb24 -c:v libx264 -qp 0 -vf "lutrgb=r='clip(val,16,235)':g='clip(val,16,235)':b='clip(val,16,235)'" -c:a pcm_s16be RGB.mkv

[Elon Musk]

Filter works fine. Please refrain from opening this ticket or new tickets on this subject.
First check facts before spreading misinformation.

[Balling]

"Total video signal range" is given as 1 - 254 in 8-bit video.

This is only limited range, both RGB and YCbCr, full range YCbCr and full range RGB do not have this limitation. They have no reserved values, as sync in HDMI for them happens outside the band, not in-band. Moreover EBU R 103 allows 1% of pixels to be out-of-gamut, because you are supposed to preserve highlights that may be superwhite. Superwhite works on all normal flagship TVs.

-qp 0 The following code leaves behind pixels outside the 8-bit range of 16 - 235, including pixels of 0 and 255.

How can you be sure? Do you use -vf extractplanes=r to check it? What is the pattern? As I said we do not support limited range RGB, there is no way to get what you want.

BTW. Looks like this is from here:
​https://forum.videohelp.com/threads/395939-ffmpeg-Color-Range/page9