Introduction
This page shows the results from the student projects that were part of GSoC 2018.
Projects
Color Constancy Filter
Description: Color constancy is the ability of the human visual system that ensures the perception of the color of objects to remain relatively constant under varying illumination conditions. A color constancy filter would basically be applied on images with various degrees of damage to restore original colors as close as possible.
Applications:
- Image restoring filter by photographers.
- Pre-processing step for computer vision algorithms that utilizes image colors.
Outcome: Low-level image statistics Grey Edge algorithm was implemented alongside with Weighted Grey Edge improvement.
Commit Links:
- Grey Edge
- Weighted Grey Edge(review in progress)
Future work: Adding Color Constancy Using CNNs and Color Dog Voting Algorithm
Mentor: Thilo Borgmann (thilo.borgmann AT mail DOT de)
Student: Mina Sami (minas.gorgy AT gmail DOT com)
Samples:
Heavily damaged sample:
Corrected by Grey Edge:
Corrected by Weighted Grey Edge:
Super Resolution filter
Description: Super resolution methods aim to recover high-resolution images from low-resolution images. These methods find direct application in many areas such as medical imaging, security and surveillance imaging, HDTV, satellite imaging. This project goal was to implement machine-learning-based super-resolution upscaling filter.
Results: The basis of this project was the result of the qualification task implementing SRCNN model for image super-resolution ( commit). Contributions of the GSoC period include:
- Introduction of DNN inference module ( commit). This module includes interface for model loading and execution as well as simple backend, which supports layers required for the super resolution filter.
- Introduction of TensorFlow backend to DNN inference module ( commit). It implements backend executing binary TensorFlow models using TensorFlow framework.
- Implementation of ESPCN model for super resolution ( commit). This model shows better results in terms of restoration accuracy and speed.
- Various improvements and fixes of the DNN module and the super resolution filter. Commits: 1, 2, 3, 4, 5.
Scripts for model training and evaluation as well as generation of model files, suitable for the super resolution filter, are provided in the repository. In addition to SRCNN and ESPCN models, VESPCN and VSRNet models were trained and evaluated, but they were not implemented in the super resolution filter due to low increase in restoration accuracy.
Future work: After the GSoC period resnet-based models will be evaluated. If their upscaling quality is good, one of them will be added to the super resolution filter.
Mentor: Pedro Souza (bygrandao AT gmail DOT com)
Student: Sergey Lavrushkin (dualfal AT gmail DOT com)
OpenCL support for Libavfilter
Description: FFmpeg has good support for offloading decoding and encoding from the CPU, but not such good support for doing anything else with video without using the CPU. Therefore it would be beneficial to increase libavfilter's support for OpenCL filters, which would allow to offload more operations to the GPU, making the CPU less busy. The project aimed to add more OpenCL filters to libavfilter.
Results: During the GSoC period, the following contributions to the project have been submitted:
- boxblur_opencl filter. Status: commit. This filter applies BoxBlur algorithm to the input video.
- sobel_opencl filter. Status: commit. This filter applies Sobel operator to the input video.
- prewitt_opencl filter. Status: commit. This filter applies Prewitt operator to the input video.
- roberts_opencl filter. Status: commit. This filter applies Roberts cross operator to the input video.
- lumakey_opencl filter. Status: review in progress. This filter turns certain luma values into transparency.
- dilation_opencl filter. Status: commit. This filter applies Dilation effect to the input video.
- erosion_opencl filter. Status: commit. This filter applies Erosion effect to the input video.
- Infrastructural patch. Status: commit. This infrastructural patch contains macro, which significantly reduce boilerplate code when setting OpenCL arguments to specific kernel.
- Documentation for the whole OpenCL subsection, including not only newly added filters, but also for existing OpenCL filters. Status: review in progress
- Additional minor improvements of the documentation. Status: commit.
- Fixing ticket of Coverity Scan. Status: commit.
Average CPU busy for CPU version of filters is ~68.034%.
Average CPU busy for OpenCL version of filters is ~28.734%.
Graph, which showing the average CPU busy when applying CPU filters and OpenCL filters:
Graph with results was obtained on computer with that characteristics:
- CPU: Intel Core i5-4200U
- GPU: AMD Radeon Graphics HD 8750M
Future work: Seeing the patches that are still in review to the end, implementing additional OpenCL support and optimizing the already existing OpenCL filters.
Mentor: Mark Thompson (sw AT jkqxz DOT net)
Student: Danil Iashchenko (danyaschenko AT gmail DOT com)
FFserver replacement
Description: This year FFserver has been dropped from the project because it was hard to maintain and to use. This project aims to replace it with an easier to use and easier to maintain codebase.
Results: A working version with some issues still remaining is (for the time being) hosted here: https://github.com/klaxa/ffserver. The supported formats are matroska, HLS and DASH, while HLS and DASH may require more work to become more robust. A flexible httpd-interface is present and was used to seamlessly switch from the libavformat http-server to the more mature libmicrohttpd http-server. Advantages over the old ffserver include an easier to read and write lua configuration files and simpler design due to exclusive public API usage.
Future work: There are still some issues to be solved, especially regarding memory leaks and more investigation and "care" for HLS and DASH. More protocols can be added as well, most notably RTP, which was on the roadmap, but was pushed back by switching to libmicrohttpd.
Mentor: Rostislav Pehlivanov (atomnuker [at] gmail [dot] com)
Student: Stephan Holljes (klaxa1337 [at] googlemail [dot] com)
Cineform Decoder Improvements
Description: The existing reverse engineered Cineform decoder present in libavcodec could not decode the 3d transform and interlaced samples. This project aimed to add the said functionality to the decoder and further implement frame threading to it.
Results: The decoder has been improved with the added feature of being able to decode the 3d transform and interlaced samples. Also frame threading has been implemented for all the samples that can be decoded.
Commits during GSOC Period
- Commits : http://git.videolan.org/?p=ffmpeg.git;a=search;s=Gagandeep+Singh;st=author
- Under review : http://ffmpeg.org/pipermail/ffmpeg-devel/2018-August/233181.html
- Under review : http://ffmpeg.org/pipermail/ffmpeg-devel/2018-August/233182.html
- Under review : http://ffmpeg.org/pipermail/ffmpeg-devel/2018-August/233183.html
Future work: As much time was used in reading the cineform-sdk and then integrating frame threading for the 3d transform samples, work on assembly instructions could not be done. LUTs are also to be implemented for definite speedup. Also a bug with the old codebook (table 9) is still to be figured out.
Mentor: Kieran Kunhya ((kierank in #ffmpeg-devel on Freenode IRC)
Student: Gagandeep Singh (deepgagan231197 [at] gmail [dot] com)
