vidstabtransform bicubic interpolation issue

[Mihaly]

Summary of the bug:

Using vidstabtransform with bicubic interpolation causes white atrtefacts on low exposure pixels. (no, linear, bilinear interpolation works well!)

How to reproduce:

$ ffmpeg -y -i ./VI000290.mp4 -vf vidstabtransform=optalgo=gauss:maxangle=0.12566370614:interpol=bicubic:maxshift=54:smoothing=30:optzoom=0:zoom=0:input=./VI000290-stab.trf:crop=keep,pp=al:f -vcodec utvideo ./VI000290-stab.avi

ffmpeg version N-87306-g6743351 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 7.2.0 (GCC)
configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib
libavutil      55. 75.100 / 55. 75.100
libavcodec     57.106.101 / 57.106.101
libavformat    57. 82.100 / 57. 82.100
libavdevice    57.  8.101 / 57.  8.101
libavfilter     6.105.100 /  6.105.100
libswscale      4.  7.103 /  4.  7.103
libswresample   2.  8.100 /  2.  8.100
libpostproc    54.  6.100 / 54.  6.100

[ablinne]

I have just encountered this exact bug. Those white pixels are probably due to an integer underflow in the interpolation routine. A bit strange that this happens, because my source material is in the limited 16-235 luma range. Maybe this should be reported upstream at ​https://github.com/georgmartius/vid.stab?. I work around the issue by compressing the luma range a bit further before applying the filter and expanding it back afterwards. However, this introduces a little bit of quantization noise.