Support Core Image of MacOSX

[rusghon]

Hello,
I would like to thank the whole FFMPEG team.

I would like to support Core Image of MacOSX.
Core Image allow us to use GPU acceleration video filtering.
This can reduce burden of CPU and realize high speed encode.
In specific, CICategorySharpen, CICategoryBlur, CICategoryColorAdjustment is useful.
One can know this effect with Core Image Fun House of developer tool of MacOSX.

Sincerely thankful for your attention

Core Image Reference Collection
​https://developer.apple.com/library/mac/documentation/graphicsimaging/Reference/CoreImagingRef/_index.html#//apple_ref/doc/uid/TP40001171

Core Image Programming Guide
​https://developer.apple.com/library/mac/documentation/graphicsimaging/Conceptual/CoreImaging/ci_intro/ci_intro.html#//apple_ref/doc/uid/TP30001185

Core Image Filter Reference
​https://developer.apple.com/library/mac/documentation/graphicsimaging/Reference/CoreImageFilterReference/index.html

[Carl Eugen Hoyos]

Patch welcome.

[rusghon]

Also, CICategoryGeometryAdjustmen is useful.
It has Crop and LanczosScale, etc.

[ponpon]

i have found Core Image Filter Plug-In for Quick Time.
the source of it is is included in CoreVF.dmg of x264Encoder.dmg.
though i dont know if it is useful for ffmoeg, i attach its url.
​http://download.cnet.com/x264Encoder/3000-2170_4-185068.html

[ponpon]

+1 for this enhancement
i am rooting for you and looking forward to your work

[Thilo Borgmann]

Core Image filtering patch sent to ffmpeg-devel:

​http://article.gmane.org/gmane.comp.video.ffmpeg.devel/211493

MACs are rare around FFmpeg devs. So I would appreciate more testing on this patch, especially from people who want to use it once it hits FFmpeg/HEAD. Please stay with the features already provided for testing. Additional feature requests should be addressed in seperate tickets after the patch hits FFmpeg/HEAD. Bugs regarding this patch should be posted directly to ffmpeg-devel, please (I might miss them here).

Thanks!

[ponpon]

Replying to thilo.borgmann:

i report the result of build with the recent patch under osx1095 and clang37 because i am not a member of devel mailing list.

failure

./configure --prefix=${TARGET} --enable-pthreads --cc=clang-mp-3.7 --cxx=clang++-mp-3.7 --optflags="-Ofast" --target-os=darwin --arch=x86_64 --cpu=corei7-avx && make -j 8

doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffmpeg-filters.html] Error 1
make: *** Waiting for unfinished jobs....
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffplay-all.html] Error 1
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffprobe-all.html] Error 1
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffmpeg-all.html] Error 1
STRIP	libavcodec/x86/vp9itxfm.o
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffserver-all.html] Error 1

success with --disable-htmlpages

./configure --prefix=${TARGET} --enable-pthreads --cc=clang-mp-3.7 --cxx=clang++-mp-3.7 --optflags="-Ofast" --target-os=darwin --arch=x86_64 --cpu=corei7-avx   --disable-htmlpages && make -j 8

[Carl Eugen Hoyos]

From a clang manpage:

	   -Ofast
	       Enables all the optimizations from -O3 along with other
	       aggressive optimizations	that may violate strict	compliance
	       with language standards.

If this is true, it sounds like a very bad idea. Did you benchmark it?

--enable-pthreads --target-os=darwin --arch=x86_64

These have no effect (not even the effect you believe they have), please remove them.

[ponpon]

thank you for your pointing out.
the result is the same. the filter isn't tried still.

./configure --prefix=${TARGET} --cc=clang-mp-3.7 --cxx=clang++-mp-3.7 && make -j 8 

CC	libavutil/xtea.o
HTML	doc/ffmpeg.html
HTML	doc/ffplay.html
HTML	doc/ffprobe.html
HTML	doc/ffserver.html
HTML	doc/ffmpeg-all.html
HTML	doc/ffplay-all.html
HTML	doc/ffprobe-all.html
HTML	doc/ffserver-all.html
HTML	doc/ffmpeg-filters.html
HTML	doc/libswscale.html
HTML	doc/libswresample.html
HTML	doc/libavcodec.html
HTML	doc/libavformat.html
HTML	doc/libavdevice.html
HTML	doc/libavfilter.html
HTML	doc/developer.html
HTML	doc/faq.html
HTML	doc/fate.html
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffmpeg-filters.html] Error 1
make: *** Waiting for unfinished jobs....
HTML	doc/general.html
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffplay-all.html] Error 1
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffprobe-all.html] Error 1
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffmpeg-all.html] Error 1
doc/filters.texi:4958: misplaced {
doc/filters.texi:4958: misplaced }
doc/filters.texi:4969: @option expected braces
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:4985: unexpected @
doc/filters.texi:5021: unknown command `inputCorrectionLevel'
make: *** [doc/ffserver-all.html] Error 1

man page

 man ffmpeg-filters
   copy
       Copy the input source unchanged to the output. This is mainly useful for testing purposes.

        at section coreimage

       Video filtering on GPU using Apple's CoreImage API on OSX.

       Hardware acceleration is based on an OpenGL context. Usually, this means it is processed by video hardware. However, software-based OpenGL implementations exist which means there is no guarantee for hardware
       processing. It depends on the respective OSX.

       There are many filters and image generators provided by Apple that come with a large variety of options. The filter has to be referenced by its name along with its options.

       The coreimage filter accepts the following options:
        at table @option
        at item list_filters List all available filters along with all their respective options as well as possible minimum and maximum values along with the default values.
        at example
           coreimage=list_filters=true
        at end example

        at item filter
       Specifiy all filters by their respective name and options.
       Use B<list_filters> to determine all valid filter names and options.
       Numerical options are specified by a float value and are automatically clamped to their respective value range.
       Vector and color options have to be specified by a list of space separated float values. Character escaping has to be done.
       A special option name C<default> is available to use default options for a filter.
       It is required to specify either C<default> or at least one of the filter options.
       All omitted options are used with their default values.
       The syntax of the filter string is as follows:
        at example
       filter=E<lt>NAMEE<gt>@E<lt>OPTIONE<gt>=E<lt>VALUEE<gt>[@E<lt>OPTIONE<gt>=E<lt>VALUEE<gt>][...][#E<lt>NAMEE<gt>@E<lt>OPTIONE<gt>=E<lt>VALUEE<gt>[@E<lt>OPTIONE<gt>=E<lt>VALUEE<gt>][...]]
        at end example
        at end table

       Several filters can be chained for successive processing without GPU-HOST transfers allowing for fast processing of complex filter chains.  Currently, only filters with zero (generators) or exactly one (filters)
       input image and one output image are supported.  Also, transition filters are not yet usable as intended.

       Some filters generate output images with additional padding depending on the respective filter kernel. The padding is automatically removed to ensure the filter output has the same size as the input image.  For image
       generators, the size of the output image is determined by the given input image. The generators do not use the pixel information of the input image to generate their output. However, the generated output is blended
       onto the input image, resulting in partial or complete coverage of the output image.

        at subsection Examples

        at itemize

        at item
       List all filters available:
        at example
       coreimage=list_filters=true
        at end example

        at item
       Use the CIBoxBlur filter with default options to blur an image:
        at example
       coreimage=filter=CIBoxBlur at default
        at end example

        at item
       Use a filter chain with CISepiaTone at default values and CIVignetteEffect with its center at 100x100 and a radius of 50 pixels:
        at example
       coreimage=filter=CIBoxBlur at default#CIVignetteEffect at inputCenter=100\ 100 at inputRadius=50
        at end example

        at item
       Use nullsrc and CIQRCodeGenerator to create a QR code for the FFmpeg homepage, given as complete and escaped command-line for Apple's standard bash shell:
        at example
       ./ffmpeg -f lavfi -i nullsrc=s=100x100,coreimage=filter=CIQRCodeGenerator at inputMessage=https\\\\\://FFmpeg.org/@inputCorrectionLevel=H -frames:v 1 QRCode.png
        at end example
        at end itemize

   crop
       Crop the input video to given dimensions.

[Carl Eugen Hoyos]

Replying to ponpon:

the result is the same. the filter isn't tried still.

You can use the configure option --disable-doc or make ffmpeg instead of make as workarounds.

./configure --prefix=${TARGET} --cc=clang-mp-3.7 --cxx=clang++-mp-3.7 && make -j 8 

If forgot: I don't think --cxx=clang++-mp-3.7 has any effect in your configure line (but I may miss something).

[Thilo Borgmann]

Thanks for pointing this out! However, if you had posted a reply on ffmpeg-devel, I would have found this earlier.

I think I've just missed to properly escape @ and {} in the texinfo file.

See ffmpeg-devel for an updated patch.

[ponpon]

Replying to thilo.borgmann:
i tried the new patch (Date: 2016-03-13 20:09:39 GMT).
but

patch -p1 < lavf_Add_coreimage_filter_for_GPU_based_image_filtering_on_OSX.patch
patch: **** Only garbage was found in the patch input.

patch version

patch -v
patch 2.5.8
Copyright (C) 1988 Larry Wall
Copyright (C) 2002 Free Software Foundation, Inc.

This program comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of this program
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

written by Larry Wall and Paul Eggert

in the patch, there is no @. only <at>. i understand differences are marked with @@. original diff? replace error?

[Thilo Borgmann]

What I see in my inbox from ffmpeg-devel it is correct. If you're copy-pasting from gmane.org or similar, you'll have @ replaced by <at>.

I'll attach it here again, you should consider subscribing to ffmpeg-devel...

[Thilo Borgmann]

Preliminary patch from ffmpeg-devel

[ponpon]

Replying to thilo.borgmann:
thank you for the attach.
i can build and the filter functions.
but slow and it seems the filter uses dGPU from gfxcardstatu.
is there a penalty in deta transher by using dGPU?

though graphic conveter use coreimage, it seems graphic conveter uses iGPU.

sorry. i hesitated. this is because junk mails are too many since my mail addresss is open to public inspection by work-related matters.
i am preparing to subscribe to ffmpeg-devel temporarily. i can test and root for you.

[Thilo Borgmann]

So I assume you use a MAC with two or more GPUs. Unfortunately I don't have such a MAC available. Without access to such a machine I can't really tell what happens in your case nor can I find out how to select a specific GPU.

If you mean that you can grant me access to such a machine by saying "I can test and root for you", then I would appreciate that very much and would try to make the GPU selctable and/or autoselected. If that's the case you can mail me directly at ​thilo.borgmann@mail.de.

If subscribing to ffmpeg-devel is a problem for you, you could clone my github at:
​https://github.com/thiloborgmann/FFmpeg/
I try to keep the coreimage branch of my repo up-to-date with what I post on the mailing list.

p.s. just found this on developer.apple.com:
"By default, once your application creates an OpenGL context (by either calling OpenGL directly or an API that relies on OpenGL such as Core Animation, Core Image, etc), the MacBook Pro automatically switches to the higher-end discrete GPU for performance concerns and won't switch back until the application quits."

This means using the dGPU (discrete GPU) is what you would want to do anyway. However, making the GPU selectable is still desirable!

When you say slow, what exactly have you measured and compared to what exactly?
Of course, uploading before processing and downloading afterwards is a speed penalty. The bigger the filter chain on GPU the less important this penalty becomes. For very simple filtering the overall processing time on the CPU can be better.

[Thilo Borgmann]

Supported with commit 4d251723.