Opened 2 months ago

Last modified 2 months ago

#6816 new enhancement

Combine "vidstabdetect" or "vidstabtransform" with scene detection

Reported by: JouMxyzptlk Owned by:
Priority: wish Component: undetermined
Version: unspecified Keywords: libvidstab
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Currently "vidstabdetect" does not detect a scene change.
The result is a very well visible jump at every scene change since it detects it as "extreme camera movement".

It is quite easy to detect in the .trf file due to their nonsense values:

Frame 66 (List 91 [(LM 0 0 833 213 112 0.717225 0.478954) <cut>
Frame 67 (List 47 [(LM -118 -138 1081 376 112 0.840303 21.377870) <cut>
Frame 68 (List 106 [(LM 0 4 957 213 112 0.846433 1.043925)

Frame 670 (List 97 [(LM 0 0 709 213 112 0.749554 1.124203) <cut>
Frame 671 (List 46 [(LM 86 16 833 213 112 0.646211 11.540179) <cut>
Frame 672 (List 105 [(LM 0 1 833 213 112 0.646211 0.761001) <cut>

Replacing them in this example with:
Frame 67 (List 0 [])
Frame 671 (List 0 [])
improves it a lot.

"killing" a few frames before and after improves it further.

Could this be automated either in "detect" to discard those nonsense values beyond "LM +-30" (as option), or in "vidstabtransform" the same way? During detection would be better IMHO.

playing around with the .trf also exploited a bug:

Deleting the offending line instead of replacing them with the dummy entry causes ffmpeg version 3.4 (Windows x64) to crash with:

[vid.stab - serialization @ 000000a4d63fdcb8] VID.STAB file: index of frames is not continuous 324 -< 330VID.STAB file: index of frames is not continuous 668 -< 674VID.STAB file: index of frames is not continuous 756 -< 762Assertion failed!

Program: c:\prog\ffmpeg\bin\ffmpeg.exe
File: /Users/kyle/software/vidstab/src/vidstab-20170830-afc8ea9/src/vsvector.c, Line 76

Expression: V

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

I'll try to implement a workaround now, automatically scan the .trf file for those bad lines and replace them.

Change History (3)

comment:1 Changed 2 months ago by cehoyos

  • Keywords libvidstab added; deshake vidstabdetect scene change removed
  • Priority changed from normal to wish

comment:2 Changed 2 months ago by JouMxyzptlk

The script workaround for now:

Replaces one "out of whack" line, matching any three digit LM block value is a quite cheap approach, but it works:

sed -re "s/(Frame [0-9]+ ).*LM\ [0-9][0-9][0-9].*/\1\(List 0 \[\]\)/g" transform1.trf > transform2.trf

Replaces three lines, the one before and the one after:

sed -re "1N;$!N;s/(Frame [0-9]+ )(.*)(Frame [0-9]+ ).*LM\ [0-9][0-9][0-9].*(Frame [0-9]+ )(.*)/\1(List 0 \[\]\)\n\3\(List 0 \[\]\)\n\4\(List 0 \[\]\)/;P;D" transform1.trf > transform2.trf

Different approach for three lines:

sed -re ":a; $q; N; s/(Frame [0-9]+ )(.*)(Frame [0-9]+ ).*LM\ [0-9][0-9][0-9].*(Frame [0-9]+ )(.*)/\1(List 0 \[\]\)\n\3\(List 0 \[\]\)\n\4\(List 0 \[\]\)/; 3,${P;D}; ba" transform1.trf > transform2.trf

Latter version allows going insane to replace 5 lines, two before and two after...

sed -re ":a; $q; N; s/(Frame [0-9]+ ).*(Frame [0-9]+ ).*(Frame [0-9]+ ).*LM\ [0-9][0-9][0-9].*(Frame [0-9]+ ).*(Frame [0-9]+ ).*/\1(List 0 \[\]\)\n\2\(List 0 \[\]\)\n\3\(List 0 \[\]\)\n\4\(List 0 \[\]\)\n\5\(List 0 \[\]\)/; 5,${P;D}; ba" transform1.trf > transform2.trf

It's been quite a while since I played with sed so deeply...

Last edited 2 months ago by JouMxyzptlk (previous) (diff)

comment:3 Changed 2 months ago by JouMxyzptlk

The hacking of the transform.trf does not really solve the issue, it just makes the jumps a bit less. Phase two of deshaking should get a "scene change thresold".
And an option to ignore frames when the camera moves left, but a person near the camera moves right, i.e. when two areas move in different directions, ignore those frames else the deshaking may take care of the wrong region.

So options would be:
"scenethresold=50" which sets it to ignore frames (i.e. treat them as scene change) with "LM +-50 +-50", and as second option like "blockthresold=30" to discard block which have "LM +-30 +-30" if there is no scene change detected.
Which value might be right for each video would be to the user test out.

Note: See TracTickets for help on using tickets.