Opened 4 years ago

Last modified 4 years ago

#1339 new enhancement

Auto insert fps filter as adapter for overlay filter inputs with different framerates

Reported by: annorax Owned by:
Priority: wish Component: avfilter
Version: git-master Keywords: fps overlay filter
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Now that the fps filter has been added (you're my heroes for adding it), it would be great if an fps filter would be auto-inserted before the overlay filter, where framerate conversion is required if the framerates of the two input vary (the way a scale filter is often auto-inserted wherever it's required).

To do this, whenever an overlay filter is used in a filter chain, the framerate of each of its inputs should be detected (using the same function used by ffprobe to populate the avg_frame_rate property), then insert an fps filter to upsample the input with the lower framerate to match the framerate of the other one.

Currently I do this manually: run ffprobe on each video file before performing an overlay, then insert an fps filter between the lower-fps file and the overlay filter. This means 3 procedure calls. It would be great if it could all be done in 1.

Change History (10)

comment:1 Changed 4 years ago by reimar

This does not sound like a good general solution. IMHO the overlay filter should create a new output frame whenever any of its inputs changes. I don't know if that would work for your use-case though?

comment:2 Changed 4 years ago by annorax

I'm not sure I know what you mean by "whenever any of its inputs change" but I don't think that's the current behavior anyway.

Here is the behavior I observed:

  • The overlay filter takes every frame of the first (main) input in turn and copies it to the output with the same timestamp while overlaying the next frame from the second input (ignoring the timestamps / framerate of the second input).
  • If it runs out of frames for the second input before the first one, it simply overlays the last frame of the second input on each remaining frame of the first input.
  • If it runs out of frames for the first input before the second one, the rest of the frames of the second input are ignored.

Therefore, the framerate and duration of the output is always equal to the framerate and duration of the first input. This can lead to a situation where the framerate of the second input (the overlaid video) is not the same as the framerate of the output file, so it appears to be running in slow/fast motion.

Why wouldn't it be a good general solution to equalize the frame rates? Do you think a user might want one of the input streams to appear slowed down / sped up in the output just because it's been overlaid on top of another stream?

At the very least, I think this would be useful to many users if it were included as an optional behavior (i.e. triggered by a config parameter).

Last edited 4 years ago by annorax (previous) (diff)

comment:3 Changed 4 years ago by reimar

The most obvious reason why it is not a general solution is that not all inputs have a (fixed) framerate or something that is a suitable substitute.
Even in a very simple case of overlaying a 24 fps video with a 25 fps one, "equalizing" the fps will not lead to a good result. If you change the 24 fps one to 25 fps that will result in noticeable stutter once a second, for exact results you'd have to go to an insane 600 fps.
If the overlay filter were taking both timestamps into account you could get an _exact_ (though variable fps) output with at most 49 frames per second (though those frames would be at very variable intervals and a lot of players would have problems displaying it accurately I admit, so the win may be questionable in real-world use).

comment:4 Changed 4 years ago by annorax

Fair enough, but as I said even now the overlay filter isn't taking both streams' timestamps into account, so what I'm suggesting is the most easily obtainable "fix" for this use case, and I've seen many postings by people using the overlay filter the way I am.

The current default behavior of the overlay filter (ending up with a fast-motion/slow-motion overlay) is, to me, not as good as what I'm suggesting, and can be very confusing.

Last edited 4 years ago by annorax (previous) (diff)

comment:5 Changed 4 years ago by annorax

BTW, I'm using the overlay filter for overlaying (PiP style) two recorded streams of a video conference (one stream per party).

Version 0, edited 4 years ago by annorax (next)

comment:6 Changed 4 years ago by Cigaes

I think Reimar explained pretty well why your suggestion is not a good idea and is, in fact, not properly defined in the first place. Please remember that filters must be designed for general use, not for any specific use case.

Currently, the overlay filters produces one output frame for each input frame on its "main" input. It uses the timestamps to determiner which overlay frame must go on the main frame.

Adding an option to output a frame also when the overlay changes, like Reimar suggests, is of course possible.

As far as I can tell your claim that it ignores the timestamps on the overlay input is just not true, and I can confirm that two video at different frame rates with visible synchronization stay on sync. The worst that happens is an off-by-one bug that is already underway of being fixed.

Please, be more specific about the actual problems you experience and are trying to solve.

comment:7 Changed 4 years ago by rogerdpack

A few notes here:

1) the "main" video fps always wins for the output.
2) according to what I can see in https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_overlay.c what it does is, for each "main" frame coming in, it looks for the overlay frame that "overlaps" that one in PTS (or matches it). So it basically outputs the original video, with a synchronized overlay on it. So order really matters here. Maybe the documentation could be updated a bit to make this more clear? It seems like reasonable functionally...unless you have 2 inputs with variable frame rates, so couldn't decide which one is the "main" one but I'd imagine that's pretty rare...

comment:8 Changed 4 years ago by cehoyos

What happens if a higher frame-rate was manually specified? Does ffmpeg just repeat the same combination of main and overlay or does it also look for every repeated main frame which overlay frame is used?

comment:9 Changed 4 years ago by Cigaes

Replying to cehoyos:

What happens if a higher frame-rate was manually specified? Does ffmpeg just repeat the same combination of main and overlay or does it also look for every repeated main frame which overlay frame is used?

You mean with -i ... -i ... -filter_complex ...overlay... -r something output?

AFAIK, the effect of -r comes completely after the effects of filters, so depending on the value of -vsync it will result on the frame being duplicated as you describe.

IMHO, there are too many possible circumstances to be able to "Do What I Mean" always, so it is best to keep everything simple so accurate results can be achieved with the correct combination of filters without bad surprises.

comment:10 Changed 4 years ago by rogerdpack

I can (apparently) confirm that if the "overlay" is 30 fps and you use -r 30 for the output, it doesn't show all of the overlay frames (it basically has a "master slave" relationship, with the master--the under frame--controlling frequency of output frames. However I suppose one could get around this by (as the OP said) determining the frame rate, and increasing/decreasing it first before applying the filter, so the current behavior might be all right. Either that or insert an fps filter automatically, as mentioned, but I'd almost prefer the current way (more explicit)...

Note: See TracTickets for help on using tickets.