gradfun errors

[shekh]

I see several issues.

1. bad x86 asm

Resulting images are obviously different with/without asm optimization.

How to reproduce:

ffmpeg.exe -cpuflags 0 -i test.png -vf gradfun=40:28 test_c.png
ffmpeg version N-91152-g7c333dc6a7 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 18.102 / 56. 18.102
  libavcodec     58. 19.102 / 58. 19.102
  libavformat    58. 16.100 / 58. 16.100
  libavdevice    58.  4.100 / 58.  4.100
  libavfilter     7. 24.100 /  7. 24.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
  libpostproc    55.  2.100 / 55.  2.100
Input #0, png_pipe, from 'test.png':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, gray(pc), 400x200 [SAR 2835:2835 DAR 2:1], 25 tbr, 25 tbn, 25 tbc
File 'test_c.png' already exists. Overwrite ? [y/N] Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'test_c.png':
  Metadata:
    encoder         : Lavf58.16.100
    Stream #0:0: Video: png, gray, 400x200 [SAR 2835:2835 DAR 2:1], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.19.102 png
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=2.67x    
video:36kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

ffmpeg.exe -i test.png -vf gradfun=40:28 test_mmx.png
ffmpeg version N-91152-g7c333dc6a7 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 18.102 / 56. 18.102
  libavcodec     58. 19.102 / 58. 19.102
  libavformat    58. 16.100 / 58. 16.100
  libavdevice    58.  4.100 / 58.  4.100
  libavfilter     7. 24.100 /  7. 24.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
  libpostproc    55.  2.100 / 55.  2.100
Input #0, png_pipe, from 'test.png':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, gray(pc), 400x200 [SAR 2835:2835 DAR 2:1], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'test_mmx.png':
  Metadata:
    encoder         : Lavf58.16.100
    Stream #0:0: Video: png, gray, 400x200 [SAR 2835:2835 DAR 2:1], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.19.102 png
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=2.67x    
video:36kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

2. Output is quantized in 2x2 blocks. This is far lower quality than even 8-bit buffer would allow, but it is using 16-bit buffer. Something must be wrong.

Visible on same test images.

3. Out of bounds read, cannot provide test

​https://github.com/FFmpeg/FFmpeg/blob/7c333dc6a7fe1dc991f484c70ed7990d6ee62724/libavfilter/vf_gradfun.c#L95-L100

        if (y < height - r) {
            int mod = ((y + r) / 2) % r;
            uint16_t *buf0 = buf + mod * bstride;
            uint16_t *buf1 = buf + (mod ? mod - 1 : r - 1) * bstride;
            int x, v;
            ctx->blur_line(dc, buf0, buf1, src + (y + r) * src_linesize, src_linesize, width / 2);
...

Looks like it should be

{{{
        if (y < height - r - 1) {
...
}}}

[shekh]

After looking more into code I realize it downsizes buffer by design, so the origin of 4x quantization is clear now. This could be mentioned in description.

[shekh]

Idea for asm defect.

index 3581f89fe8..a13b5c3442 100644
--- a/libavfilter/x86/vf_gradfun.asm
+++ b/libavfilter/x86/vf_gradfun.asm
@@ -39,9 +39,9 @@ SECTION .text
     psubw      m2, m6
     pminsw     m2, m7
     pmullw     m2, m2
-    psllw      m1, 2
     paddw      m0, %1
     pmulhw     m1, m2
+    psllw      m1, 2
     paddw      m0, m1
     psraw      m0, 7
     packuswb   m0, m0

[Carl Eugen Hoyos]

The overflows are still reproducible.