wiki:

SponsoringPrograms

/

GSoC

/

2018

/

Results

Introduction

This page shows the results from the student projects that were part of GSoC 2018.


Projects

Color Constancy Filter

Description: Color constancy is the ability of the human visual system that ensures the perception of the color of objects to remain relatively constant under varying illumination conditions. A color constancy filter would basically be applied on images with various degrees of damage to restore original colors as close as possible.

Applications:

  • Image restoring filter by photographers.
  • Pre-processing step for computer vision algorithms that utilizes image colors.

Outcome: Low-level image statistics Grey Edge algorithm was implemented alongside with Weighted Grey Edge improvement.

Commit Links:

Future work: Adding Color Constancy Using CNNs and Color Dog Voting Algorithm

Mentor: Thilo Borgmann (thilo.borgmann AT mail DOT de)

Student: Mina Sami (minas.gorgy AT gmail DOT com)

Samples:

Heavily damaged sample:
https://firebasestorage.googleapis.com/v0/b/ffmpeg-color-constancy.appspot.com/o/original.jpg

Corrected by Grey Edge:
https://firebasestorage.googleapis.com/v0/b/ffmpeg-color-constancy.appspot.com/o/grey_edge.png

Corrected by Weighted Grey Edge:
https://firebasestorage.googleapis.com/v0/b/ffmpeg-color-constancy.appspot.com/o/weighted_grey_edge.png

Super Resolution filter

Description: Super resolution methods aim to recover high-resolution images from low-resolution images. These methods find direct application in many areas such as medical imaging, security and surveillance imaging, HDTV, satellite imaging. This project goal was to implement machine-learning-based super-resolution upscaling filter.

Results: The basis of this project was the result of the qualification task implementing SRCNN model for image super-resolution ( commit). Contributions of the GSoC period include:

  • Introduction of DNN inference module ( commit). This module includes interface for model loading and execution as well as simple backend, which supports layers required for the super resolution filter.
  • Introduction of TensorFlow backend to DNN inference module ( commit). It implements backend executing binary TensorFlow models using TensorFlow framework.
  • Implementation of ESPCN model for super resolution ( commit). This model shows better results in terms of restoration accuracy and speed.
  • Various improvements and fixes of the DNN module and the super resolution filter. Commits: 1, 2, 3, 4, 5.

Scripts for model training and evaluation as well as generation of model files, suitable for the super resolution filter, are provided in the repository. In addition to SRCNN and ESPCN models, VESPCN and VSRNet models were trained and evaluated, but they were not implemented in the super resolution filter due to low increase in restoration accuracy.

Future work: After the GSoC period resnet-based models will be evaluated. If their upscaling quality is good, one of them will be added to the super resolution filter.

Mentor: Pedro Souza (bygrandao AT gmail DOT com)

Student: Sergey Lavrushkin (dualfal AT gmail DOT com)

OpenCL support for Libavfilter

Description: FFmpeg has good support for offloading decoding and encoding from the CPU, but not such good support for doing anything else with video without using the CPU. Therefore it would be beneficial to increase libavfilter's support for OpenCL filters, which would allow to offload more operations to the GPU, making the CPU less busy. The project aimed to add more OpenCL filters to libavfilter.

Results: During the GSoC period, the following contributions to the project have been submitted:

Average CPU busy for CPU version of filters is ~68.034%.
Average CPU busy for OpenCL version of filters is ~28.734%.

Graph, which showing the average CPU busy when applying CPU filters and OpenCL filters:

Graph with results was obtained on computer with that characteristics:

  • CPU: Intel Core i5-4200U
  • GPU: AMD Radeon Graphics HD 8750M

https://drive.google.com/uc

Future work: Seeing the patches that are still in review to the end, implementing additional OpenCL support and optimizing the already existing OpenCL filters.

Mentor: Mark Thompson (sw AT jkqxz DOT net)

Student: Danil Iashchenko (danyaschenko AT gmail DOT com)

FFserver replacement

Description: This year FFserver has been dropped from the project because it was hard to maintain and to use. This project aims to replace it with an easier to use and easier to maintain codebase.

Results: A working version with some issues still remaining is (for the time being) hosted here: https://github.com/klaxa/ffserver. The supported formats are matroska, HLS and DASH, while HLS and DASH may require more work to become more robust. A flexible httpd-interface is present and was used to seamlessly switch from the libavformat http-server to the more mature libmicrohttpd http-server. Advantages over the old ffserver include an easier to read and write lua configuration files and simpler design due to exclusive public API usage.

Future work: There are still some issues to be solved, especially regarding memory leaks and more investigation and "care" for HLS and DASH. More protocols can be added as well, most notably RTP, which was on the roadmap, but was pushed back by switching to libmicrohttpd.

Mentor: Rostislav Pehlivanov (atomnuker [at] gmail [dot] com)

Student: Stephan Holljes (klaxa1337 [at] googlemail [dot] com)

Cineform Decoder Improvements

Description: The existing reverse engineered Cineform decoder present in libavcodec could not decode the 3d transform and interlaced samples. This project aimed to add the said functionality to the decoder and further implement frame threading to it.

Results: The decoder has been improved with the added feature of being able to decode the 3d transform and interlaced samples. Also frame threading has been implemented for all the samples that can be decoded.

Commits during GSOC Period

Future work: As much time was used in reading the cineform-sdk and then integrating frame threading for the 3d transform samples, work on assembly instructions could not be done. LUTs are also to be implemented for definite speedup. Also a bug with the old codebook (table 9) is still to be figured out.

Mentor: Kieran Kunhya ((kierank in #ffmpeg-devel on Freenode IRC)

Student: Gagandeep Singh (deepgagan231197 [at] gmail [dot] com)

Last modified 2 months ago Last modified on Aug 17, 2018, 3:48:38 PM