Opened 10 years ago

Last modified 9 years ago

#1430 open enhancement

More efficient deshake filter

Reported by: Clément Bœsch Owned by: Georg Martius
Priority: wish Component: avfilter
Version: unspecified Keywords: deshake
Cc: georg.martius@web.de Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by Clément Bœsch)

An interesting de-shake filter article: http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Our deshake filter isn't very efficient with the provided sample (at least with the default settings)

clive -f best 'http://www.youtube.com/watch?v=627MqC6E5Yo'
ffplay 'Sam and Cocoa shaky original.webm' \
    -vf 'split[a][b]; [a]pad=iw*2:ih[src]; [b]deshake[filt]; [src][filt]overlay=w'

It would be nice to improve the filter to handle such sample.

Attachments (1)

Sam_and_Cocoa_shaky_original.mp4 (2.1 MB ) - added by Bernardo 9 years ago.
Sam and Cocoa shaky original (bitrate reduced to fit 2.5MB)

Change History (21)

comment:1 by Clément Bœsch, 10 years ago

Description: modified (diff)

comment:2 by Clément Bœsch, 10 years ago

Description: modified (diff)

comment:3 by Carl Eugen Hoyos, 10 years ago

Keywords: deshake added
Priority: normalwish

comment:4 by Carl Eugen Hoyos, 9 years ago

Please provide the sample, your url is 403 here.

by Bernardo, 9 years ago

Sam and Cocoa shaky original (bitrate reduced to fit 2.5MB)

comment:5 by Carl Eugen Hoyos, 9 years ago

I don't know much about the deshake filter but I wonder if a high bitrate sample wouldn't make more sense - please consider uploading the larger sample to http://www1.datafilehost.com/ and post the link here.

comment:7 by Carl Eugen Hoyos, 9 years ago

Works better here with "-vf deshake=rx=48:ry=64"

comment:8 by Carl Eugen Hoyos, 9 years ago

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

in reply to:  8 ; comment:9 by Bernardo, 9 years ago

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows. Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result. This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab (https://github.com/georgmartius/vid.stab and http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

in reply to:  9 ; comment:10 by Carl Eugen Hoyos, 9 years ago

Replying to jbvsmo:

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows.

Sorry, which link do you mean?

Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result.

Of course, I don't understand the above argument...

This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab (https://github.com/georgmartius/vid.stab and http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

I'd like to know the answer as well, please ask the developer!

in reply to:  10 ; comment:11 by Bernardo, 9 years ago

Replying to cehoyos:

Sorry, which link do you mean?

The article in the ticket description:

http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Of course, I don't understand the above argument...

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better. That's what I'm saying

in reply to:  11 comment:12 by Carl Eugen Hoyos, 9 years ago

Replying to jbvsmo:

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better.

That is likely true, thank you for explaining.

comment:13 by Georg Martius, 9 years ago

Owner: set to Georg Martius
Status: newopen

comment:14 by Georg Martius, 9 years ago

We are working on adding video stabilization using vid.stab library (http://public.hronopik.de/vid.stab/). The sample video can still not dealt with very well because it has massive rolling shutter which is not compensated by vid.stab yet, but I am working on it.

comment:15 by Carl Eugen Hoyos, 9 years ago

What's wrong with improving the existing filter?

comment:16 by Georg Martius, 9 years ago

Cc: georg.martius@web.de added

comment:17 by Georg Martius, 9 years ago

Well its architecture is too simple. A single pass cannot work without the possibility for the filter to look ahead for a few frames (not supported by the AVFilter API).
Deshake only uses compensates translations no rotations, and many more small features are missing which vid.stab offers, e.g. contrast based selection of relevant parts of the frame.

comment:18 by Carl Eugen Hoyos, 9 years ago

An external library is simply no solution and I don't understand how the API does not work for the internal filter but works for an external one. (and why shouldn't deshake buffer some frames?)

comment:19 by Georg Martius, 9 years ago

Why is an external library no solution? ffmpeg relies on many external libs!
The external lib is a 2-pass version. (I also have a single pass one for transcode at the moment, but it suffers from the same problem that you cannot look into the future and thus the corrections are worse).
Buffering is not problem, but you have to produce frames for every frame you read so you cannot wait for say 10 frames before you output anything.

comment:20 by Andrew, 9 years ago

In case anyone else stumbles across this like me you can build ffmpeg with vid.stab just using --enable-libvidstab

Of course you first need to download and install the vid.stab codebase

Also commented on in ffmpeg-users list: http://ffmpeg.org/pipermail/ffmpeg-user/2013-May/014948.html

Note: See TracTickets for help on using tickets.