Screen blending regression

[antmarot]

Summary of the bug:

Incorrect resulting colors when using screen blending.
Looks like a regression since it works well in ffmpeg version 3.2.10 but does not work in ffmpeg version 4.2.1.

How to reproduce:

% ffmpeg -filter_complex "color=green:size=hd720,format=rgba[a]; color=black:size=hd720,format=rgba[b]; [a][b] blend=all_mode=screen,format=yuv422p10le" -pix_fmt yuv420p -t 1 /tmp/test.mp4

The previous command should produce a green video but the result is purple (v4.2.1).

v4.2.1 output:

ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.8)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.1_1 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-13.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-13.jdk/Contents/Home/include/darwin -fno-stack-check' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Stream mapping:
  format -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[libx264 @ 0x7ff359001200] using SAR=1/1
[libx264 @ 0x7ff359001200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7ff359001200] profile High, level 3.1
[libx264 @ 0x7ff359001200] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=22 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/tmp/test.mp4':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.54.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=   25 fps=0.0 q=-1.0 Lsize=       5kB time=00:00:00.88 bitrate=  41.9kbits/s speed=2.52x    
video:3kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 33.352619%
[libx264 @ 0x7ff359001200] frame I:1     Avg QP:16.00  size:   672
[libx264 @ 0x7ff359001200] frame P:6     Avg QP:17.00  size:   237
[libx264 @ 0x7ff359001200] frame B:18    Avg QP:19.67  size:    37
[libx264 @ 0x7ff359001200] consecutive B-frames:  4.0%  0.0%  0.0% 96.0%
[libx264 @ 0x7ff359001200] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x7ff359001200] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4: 16.3%  0.0%  0.0%  0.0%  0.0%    skip:83.7%
[libx264 @ 0x7ff359001200] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%
[libx264 @ 0x7ff359001200] 8x8 transform intra:0.0%
[libx264 @ 0x7ff359001200] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 4.1% 0.0%
[libx264 @ 0x7ff359001200] i16 v,h,dc,p: 98%  0%  2%  0%
[libx264 @ 0x7ff359001200] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0x7ff359001200] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7ff359001200] kb/s:22.14

v3.2.10 output:

ffmpeg version 3.2.10-1~deb9u1~bpo8+1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.9.2 (Debian 4.9.2-10)
  configuration: --prefix=/usr --extra-version='1~deb9u1~bpo8+1' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --disable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
File '/tmp/test.mp4' already exists. Overwrite ? [y/N] y
[libx264 @ 0x55cab3d60cc0] using SAR=1/1
[libx264 @ 0x55cab3d60cc0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 @ 0x55cab3d60cc0] profile High, level 3.1
[libx264 @ 0x55cab3d60cc0] 264 - core 142 r2431 a5831aa - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/tmp/test.mp4':
  Metadata:
    encoder         : Lavf57.56.101
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc57.64.101 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
Stream mapping:
  format -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
frame=   25 fps= 16 q=-1.0 Lsize=       3kB time=00:00:00.88 bitrate=  31.3kbits/s speed=0.572x
video:2kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 50.349346%
[libx264 @ 0x55cab3d60cc0] frame I:1     Avg QP:15.00  size:   672
[libx264 @ 0x55cab3d60cc0] frame P:6     Avg QP:16.83  size:    43
[libx264 @ 0x55cab3d60cc0] frame B:18    Avg QP:19.67  size:    37
[libx264 @ 0x55cab3d60cc0] consecutive B-frames:  4.0%  0.0%  0.0% 96.0%
[libx264 @ 0x55cab3d60cc0] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x55cab3d60cc0] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 0x55cab3d60cc0] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%
[libx264 @ 0x55cab3d60cc0] 8x8 transform intra:0.0%
[libx264 @ 0x55cab3d60cc0] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 0x55cab3d60cc0] i16 v,h,dc,p: 98%  0%  2%  0%
[libx264 @ 0x55cab3d60cc0] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0x55cab3d60cc0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x55cab3d60cc0] kb/s:12.81

[Carl Eugen Hoyos]

Is the issue reproducible with current FFmpeg git head, the only version supported here?

[antmarot]

Just checked and yes, it's reproducible with current FFmpeg git head.

[pdr0]

The blend operation in RGB is ok. The problem is inaccurate RGB => YUV conversion

If you test output RGB24, the output is "green" or the expected 0,128,0

ffmpeg -filter_complex "color=green:size=hd720,format=rgba[a]; color=black:size=hd720,format=rgba[b]; [a][b] blend=all_mode=screen,format=rgb24" -vframes 1 rgb24.png -y

To increase accuracy of RGB<=>YUV conversions, either use swsflags with -vf scale, or zscale, with appropriate matrix (709 for HD is convention)

eg. swsflags

ffmpeg -filter_complex "color=green:size=hd720,format=rgba[a]; color=black:size=hd720,format=rgba[b]; [a][b] blend=all_mode=screen,scale=out_color_matrix=bt709:flags=bicubic+accurate_rnd+full_chroma_int+full_chroma_inp,format=yuv420p" -t 1 test_swsflags.mp4 -y

eg. zscale

ffmpeg -filter_complex "color=green:size=hd720,format=rgba[a]; color=black:size=hd720,format=rgba[b]; [a][b] blend=all_mode=screen,zscale=matrix=709,format=yuv420p" -t 1 test_zscale.mp4 -y

[antmarot]

Thank you pdr0. I confirm it now produces the expected outputs.
Out of curiosity, does it mean that it is the RGB => YUV conversion that has regressed or it's the expected behavior?

[Marton Balint]

It is not a precision issue. The result depends on which pixel format the blend filter use, RGB or YUV. Blending in YUV gets you purple, blending in RGB gets you green. The automagically selected format can change (depending on what formats the filters support) between versions.

So this is not necessarily a bug. Admittedly some blending modes makes little sense for YUV, and the filter gives you too much freedom here. Maybe an option should be added to the filter so you can limit the formats to RGB only to always get the expected results?