Blockiness and ringing in prores4444 encoded video with alpha

[stib]

Summary of the bug:
How to reproduce:
Transcode a file with an alpha channel to ProRes4444 using prores_ks encoder

% ffmpeg -i 'input.mov'  -c:v prores_ks -profile:v 4 -quant_mat 'hq' -pix_fmt yuva444p10le 'output.mov'

ffmpeg version N-82664-g801b5c1 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.108 / 57. 66.108
  libavformat    57. 58.101 / 57. 58.101
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 67.100 /  6. 67.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\alpha test.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2016-12-12T03:17:14.000000Z
  Duration: 00:00:10.00, start: 0.000000, bitrate: 406275 kb/s
    Stream #0:0(eng): Video: qtrle (rle  / 0x20656C72), bgra(progressive), 1920x1080, 405436 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      encoder         : Animation
      timecode        : 00:00:00:00
    Stream #0:1(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
File 'prorestest2.mov' already exists. Overwrite ? [y/N] y
Output #0, mov, to 'prorestest2.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    encoder         : Lavf57.58.101
    Stream #0:0(eng): Video: prores (prores_ks) (ap4h / 0x68347061), yuva444p10le, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
      encoder         : Lavc57.66.108 prores_ks
Stream mapping:
  Stream #0:0 -> #0:0 (qtrle (native) -> prores (prores_ks))
Press [q] to stop, [?] for help
frame=  250 fps=5.5 q=-0.0 Lsize=  186122kB time=00:00:09.96 bitrate=153082.2kbits/s speed=0.218x
video:186119kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001761%

Expected
Something similar to the original. This is a png from the test file, which was created with Adobe AFter Effects, encoded as QTRLE bgra-8 in a .mov container:

Observed
Blockiness, ringing and noise, particularly in areas with partial alpha. This is a png saved from the transcoded result, which was created using the command above. The artefacts appear in areas where the alpha channel is partially transparent, although the alpha channel itself seems to be ok. The problem is worse when the -alpha_bits encoding option is set to 16:

The effect is most noticeable when viewing one channel against black. This is a screenshot of the Red channel video playing in DJV View:

And here is the Alpha channel:

The machine used to encode was running windows7 x64

[Carl Eugen Hoyos]

Did you provide input.mov?
Are you reporting an issue with the prores encoder or with libswscale?

[stib]

png file which can be used as a source file to test the problems with the prores encoder

[stib]

I tried to ftp the movie, but I couldn't connect to the server. connecting using
ftp ftp://anonymous@upload.ffmpeg.org timed out, and when I connected with sftp ftp://anonymous@upload.ffmpeg.org it asked for a password and wouldn't take an empty one or 'anonymous'.

Anyway the problem still occurs when the source is a png image, so I've attached one.

This is the command I've been using with the png source:
ffmpeg -i source.png -c:v prores_ks -pix_fmt yuva444p10le -quant_mat 'hq' -profile:v 4 output.mov

[stib]

Replying to cehoyos:

Are you reporting an issue with the prores encoder or with libswscale?

I'm not sure… The problem only occurs when encoding in prores444 codec, using the prores_ks encoder, and with yuva444p10le pixel format.

[Carl Eugen Hoyos]

Do any of the following commands allow to reproduce the issue you see?

$ ffmpeg -i source.png -pix_fmt yuvj444p out.jpg

$ ffmpeg -i source.png -vf format=yuv444p out.png

$ ffmpeg -i source.png -vcodec prores_ks -pix_fmt yuv444p10 out.mov

[Mulvya]

I believe this is an issue specific to the prores encoding to a pixel format with alpha. See this ​thread at Video Stack Exchange.

[stib]

None of those commands produce output with an alpha channel, there is no noise or blockiness, but the alpha blending is gone; it looks like it is just ignoring the alpha in the input image:

Replying to cehoyos:

Do any of the following commands allow to reproduce the issue you see?

$ ffmpeg -i source.png -pix_fmt yuvj444p out.jpg

$ ffmpeg -i source.png -vf format=yuv444p out.png

$ ffmpeg -i source.png -vcodec prores_ks -pix_fmt yuv444p10 out.mov

[Carl Eugen Hoyos]

Not a regression.

[Carl Eugen Hoyos]

May be related to ticket #3846.

[Elon Musk]

Should be fixed in 315f51128a95ca34ac3212c86b2a938330ba6b6e.