Opened 5 years ago

Closed 5 years ago

#442 closed enhancement (fixed)

scene change detection

Reported by: lovelove Owned by: ubitux
Priority: wish Component: avfilter
Version: unspecified Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

suggestion: scene change detection

status: Isn't this possibly already implemented "under the hood" in the form of the -sc_threshold parameter?

key benefits / fields of application:

(1) Exporting first frame of each scene as an image effectively creates generates a storyboard ...
(2) ... which in turn would be an enormous help in identifying added and deleted scenes when comparing the content of two videos.
(3) Alternatively, when storing the timestamps in a text file, this text file could then be fed to a video editor to automatically split a long recording into individual scenes.
(4) Using I-frame upon scene change for improving compression ratio.

review of the situation: Here is a scholar paper reviewing existing scene detection method and comparing their performances: http://publik.tuwien.ac.at/files/pub-et_9739.pdf

proof-of-concept:
http://www.youtube.com/watch?v=4988BhKUa9Q&NR=1
http://www.youtube.com/watch?v=4esP2nO1_kY

open source implementations of scene change detection algorithms possibly easily reusable in FFmpeg:

(1) BBC R&D's open source (LGPL) Video Shot Change Detector: http://www.bbc.co.uk/opensource/projects/shot_change/ ( = http://sourceforge.net/projects/shot-change/ )

(2) Shotdetect (LGPL): http://shotdetect.nonutc.fr/

(3) yuvdiff (GNU GPL): http://www.silicontrip.net/~mark/lavtools/yuvdiff.c

(4) http://www.mathworks.com/products/computer-vision/demos.html?file=/products/demos/shipping/vision/videoscenechange.html

(5) http://www.catenary.com/howto/motion.html

(6) lav2yuv -S list.el ("Output a scene list with scene detection")
http://manpages.unixforum.co.uk/man-pages/linux/suse-linux-10.1/1/lav2yuv-man-page.html

(7) SCSelect (filter, which distinguishes between scene begins, scene ends and global motion):
http://home.arcor.de/kassandro/RemoveDirt/RemoveDirt.htm

(8) sourcecode of Virtualdub' VCR mod:
http://forum.doom9.org/showthread.php?p=268534#post268534

(9) SCXvid: http://forum.doom9.org/showthread.php?p=1192526#post1192526

(10) Marc FDs and neuron2's filters:
http://forum.doom9.org/showthread.php?p=269425#post269425

Change History (8)

comment:1 Changed 5 years ago by lovelove

Here is the (latest of several) FFmpeg-users threads discussing this issue:
http://ffmpeg-users.933282.n4.nabble.com/Scene-detection-td941845.html

I will try to concisely sum up the key suggestions of the thread:


"You could store the various first frames of each scene in order to get a sort of photographic storyline of a movie, or you could store the timestamps somewhere in order to use that to automatically cut every movie 'shot' with a video editor. Both options should be possible with FFmpeg+libavfilter." (Stefano Sabatini)


"It should be not too hard to implement a scene detection filter in libavfilter. The fake syntax for a lavfi filter (BIG WARNING, libavfilter is not still properly integrated into FFmpeg, check the soc repo) would be something of the kind:

ffmpeg -i in.avi -vfilters "[IN] split [SC_DETECT] [OUT], [SC_DETECT] sc_detect=PARAMS, process_sc_frames=PARAMS" /dev/null

As you see you would need two filters, one detecting scene changes and outputting the detected scene change frames to the output, and a filter to furtherly processes the information of the frames, for example it could print to a file the timestamp of the file or to render as an image file the processed frames.

The sc_detect filter implementation is the tricky part, but a naive implementation shouldn't be too hard to accomplish." (Stefano Sabatini)


"Scene-detection looks like a much requested feature, please file a feature request on the issue tracker." (Stefano Sabatini)


"On the technical side: this has been already discussed on ffmpeg-devel, an underkill solution may be as simple as tweaking the select filter and adopt a simple pixel-per-pixel average difference as metric.

For images with moving objects that's not a proper solution, and you need motion estimation code, which is already integrated in libavcodec, so the ideal solution would be to refactor the ME code in libavcodec and move it to libavutil. By doing this you may also improve the code itself, and thus improve the codecs using it at the same time." (Stefano Sabatini)


"For ideas/details about how to implement a scene-cut detection, I would suggest looking at the code of shotdetect (Johan Mathe is really nice, and the license is LGPL by the way). His approach is actually really simple and straightforward, and we've now been using it in our long-term archive solution for quite a while and could provide information based on hands-on experience about what to expect from that straight-forwards algorithm.

I'm mentioning this, because I think Johan's algorithm is merely a variation of a "pixel-per-pixel average difference" and it works surprisingly well - on really different kinds of content." (Peter B.)

Version 2, edited 5 years ago by lovelove (previous) (next) (diff)

comment:2 Changed 5 years ago by lovelove

cross-reference: #92 ("Extract all key frames from a video using ffmpeg tool")

comment:3 Changed 5 years ago by saste

  • Component changed from undetermined to avfilter

comment:4 Changed 5 years ago by cehoyos

  • Cc lovelove removed
  • Keywords scene change detection removed
  • Priority changed from normal to wish
  • Status changed from new to open

comment:5 Changed 5 years ago by lovelove

@cehoyos: why did you remove me from CC ? Please re-add me. thank you

comment:6 Changed 5 years ago by cehoyos

  • Reproduced by developer set

comment:7 Changed 5 years ago by ubitux

  • Owner set to ubitux

Work in progress, patch on the mailing list. Preview: http://imgur.com/a/icpfk

comment:8 Changed 5 years ago by ubitux

  • Resolution set to fixed
  • Status changed from open to closed

Added in 7286814.

Note: See TracTickets for help on using tickets.