More efficient deshake filter

[Clément Bœsch]

An interesting de-shake filter article: ​http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Our deshake filter isn't very efficient with the provided sample (at least with the default settings)

clive -f best 'http://www.youtube.com/watch?v=627MqC6E5Yo'

ffplay 'Sam and Cocoa shaky original.webm' \
    -vf 'split[a][b]; [a]pad=iw*2:ih[src]; [b]deshake[filt]; [src][filt]overlay=w'

It would be nice to improve the filter to handle such sample.

[Carl Eugen Hoyos]

Please provide the sample, your url is 403 here.

[Bernardo]

Sam and Cocoa shaky original (bitrate reduced to fit 2.5MB)

[Carl Eugen Hoyos]

I don't know much about the deshake filter but I wonder if a high bitrate sample wouldn't make more sense - please consider uploading the larger sample to ​http://www1.datafilehost.com/ and post the link here.

[Clément Bœsch]

Original uploaded here: ​http://lucy.pkh.me/samples/Sam%20and%20Cocoa%20shaky%20original.webm

[Carl Eugen Hoyos]

Works better here with "-vf deshake=rx=48:ry=64"

[Carl Eugen Hoyos]

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

[Bernardo]

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows. Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result. This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab (​https://github.com/georgmartius/vid.stab and ​http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

[Carl Eugen Hoyos]

Replying to jbvsmo:

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows.

Sorry, which link do you mean?

Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result.

Of course, I don't understand the above argument...

This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab (​https://github.com/georgmartius/vid.stab and ​http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

I'd like to know the answer as well, please ask the developer!

[Bernardo]

Replying to cehoyos:

Sorry, which link do you mean?

The article in the ticket description:

​http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Of course, I don't understand the above argument...

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better. That's what I'm saying

[Carl Eugen Hoyos]

Replying to jbvsmo:

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better.

That is likely true, thank you for explaining.

[Georg Martius]

We are working on adding video stabilization using vid.stab library (​http://public.hronopik.de/vid.stab/). The sample video can still not dealt with very well because it has massive rolling shutter which is not compensated by vid.stab yet, but I am working on it.

[Carl Eugen Hoyos]

What's wrong with improving the existing filter?

[Georg Martius]

Well its architecture is too simple. A single pass cannot work without the possibility for the filter to look ahead for a few frames (not supported by the AVFilter API).
Deshake only uses compensates translations no rotations, and many more small features are missing which vid.stab offers, e.g. contrast based selection of relevant parts of the frame.

[Carl Eugen Hoyos]

An external library is simply no solution and I don't understand how the API does not work for the internal filter but works for an external one. (and why shouldn't deshake buffer some frames?)

[Georg Martius]

Why is an external library no solution? ffmpeg relies on many external libs!
The external lib is a 2-pass version. (I also have a single pass one for transcode at the moment, but it suffers from the same problem that you cannot look into the future and thus the corrections are worse).
Buffering is not problem, but you have to produce frames for every frame you read so you cannot wait for say 10 frames before you output anything.

[Andrew]

In case anyone else stumbles across this like me you can build ffmpeg with vid.stab just using --enable-libvidstab

Of course you first need to download and install the vid.stab codebase

Also commented on in ffmpeg-users list: ​http://ffmpeg.org/pipermail/ffmpeg-user/2013-May/014948.html