Opened 4 years ago

Last modified 3 years ago

#1430 open enhancement

More efficient deshake filter

Reported by: ubitux Owned by: flexman
Priority: wish Component: avfilter
Version: unspecified Keywords: deshake
Cc: georg.martius@web.de Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by ubitux)

An interesting de-shake filter article: http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Our deshake filter isn't very efficient with the provided sample (at least with the default settings)

clive -f best 'http://www.youtube.com/watch?v=627MqC6E5Yo'
ffplay 'Sam and Cocoa shaky original.webm' \
    -vf 'split[a][b]; [a]pad=iw*2:ih[src]; [b]deshake[filt]; [src][filt]overlay=w'

It would be nice to improve the filter to handle such sample.

Attachments (1)

Sam_and_Cocoa_shaky_original.mp4 (2.1 MB) - added by jbvsmo 3 years ago.
Sam and Cocoa shaky original (bitrate reduced to fit 2.5MB)

Change History (21)

comment:1 Changed 4 years ago by ubitux

  • Description modified (diff)

comment:2 Changed 4 years ago by ubitux

  • Description modified (diff)

comment:3 Changed 4 years ago by cehoyos

  • Keywords deshake added
  • Priority changed from normal to wish

comment:4 Changed 3 years ago by cehoyos

Please provide the sample, your url is 403 here.

Changed 3 years ago by jbvsmo

Sam and Cocoa shaky original (bitrate reduced to fit 2.5MB)

comment:5 Changed 3 years ago by cehoyos

I don't know much about the deshake filter but I wonder if a high bitrate sample wouldn't make more sense - please consider uploading the larger sample to http://www1.datafilehost.com/ and post the link here.

comment:7 Changed 3 years ago by cehoyos

Works better here with "-vf deshake=rx=48:ry=64"

comment:8 follow-up: Changed 3 years ago by cehoyos

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

comment:9 in reply to: ↑ 8 ; follow-up: Changed 3 years ago by jbvsmo

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows. Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result. This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab (https://github.com/georgmartius/vid.stab and http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

comment:10 in reply to: ↑ 9 ; follow-up: Changed 3 years ago by cehoyos

Replying to jbvsmo:

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows.

Sorry, which link do you mean?

Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result.

Of course, I don't understand the above argument...

This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab (https://github.com/georgmartius/vid.stab and http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

I'd like to know the answer as well, please ask the developer!

comment:11 in reply to: ↑ 10 ; follow-up: Changed 3 years ago by jbvsmo

Replying to cehoyos:

Sorry, which link do you mean?

The article in the ticket description:

http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Of course, I don't understand the above argument...

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better. That's what I'm saying

comment:12 in reply to: ↑ 11 Changed 3 years ago by cehoyos

Replying to jbvsmo:

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better.

That is likely true, thank you for explaining.

comment:13 Changed 3 years ago by flexman

  • Owner set to flexman
  • Status changed from new to open

comment:14 Changed 3 years ago by flexman

We are working on adding video stabilization using vid.stab library (http://public.hronopik.de/vid.stab/). The sample video can still not dealt with very well because it has massive rolling shutter which is not compensated by vid.stab yet, but I am working on it.

comment:15 Changed 3 years ago by cehoyos

What's wrong with improving the existing filter?

comment:16 Changed 3 years ago by flexman

  • Cc georg.martius@web.de added

comment:17 Changed 3 years ago by flexman

Well its architecture is too simple. A single pass cannot work without the possibility for the filter to look ahead for a few frames (not supported by the AVFilter API).
Deshake only uses compensates translations no rotations, and many more small features are missing which vid.stab offers, e.g. contrast based selection of relevant parts of the frame.

comment:18 Changed 3 years ago by cehoyos

An external library is simply no solution and I don't understand how the API does not work for the internal filter but works for an external one. (and why shouldn't deshake buffer some frames?)

comment:19 Changed 3 years ago by flexman

Why is an external library no solution? ffmpeg relies on many external libs!
The external lib is a 2-pass version. (I also have a single pass one for transcode at the moment, but it suffers from the same problem that you cannot look into the future and thus the corrections are worse).
Buffering is not problem, but you have to produce frames for every frame you read so you cannot wait for say 10 frames before you output anything.

comment:20 Changed 3 years ago by sinkers

In case anyone else stumbles across this like me you can build ffmpeg with vid.stab just using --enable-libvidstab

Of course you first need to download and install the vid.stab codebase

Also commented on in ffmpeg-users list: http://ffmpeg.org/pipermail/ffmpeg-user/2013-May/014948.html

Note: See TracTickets for help on using tickets.