When using video filter fps, provide an option to retain actual pts, and pts_time for frames

[vishal733]

Summary of the bug:
How to reproduce:

% ffmpeg -i input -filter_complex "showinfo,fps=20,showinfo" out_%03d.png
ffmpeg version: master (current head: ed3b64402ef770097f81e4f1e17f1fca253159aa, avfilter/af_anlmdn: ignore too small values
)
built on macOS

Output:

[Parsed_showinfo_0 @ 0x...] n:   0 pts:      0 pts_time:0       pos:    93846 ...
[Parsed_showinfo_0 @ 0x...] n:   1 pts:   3000 pts_time:0.0333333 pos:    95321 ...
[Parsed_showinfo_2 @ 0x...] n:   0 pts:      0 pts_time:0       pos:    93846 ...
[Parsed_showinfo_0 @ 0x...] n:   2 pts:   6000 pts_time:0.0666667 pos:    95358 ...
[Parsed_showinfo_0 @ 0x...] n:   3 pts:   9000 pts_time:0.1     pos:    95398 ...
[Parsed_showinfo_2 @ 0x...] n:   1 pts:      1 pts_time:0.05    pos:    95358 ...

In the output, showinfo_0 refers to timestamps before action of fps. and showinfo_2 refers to timestamps after action of fps.
Clearly you can see that pts is artificially alterered for frames obtained after fps filter. (Ex: pts of frame at pos 95358 is altered from 0.0666667 to 0.05)
Can we not have a pts_bypass mode which allows to have the pts of frames unaltered.
This will be really useful for applications using ffmpeg to decode frames, and then having accurate back-correspondence.

[vishal733]

Note that adding a constant offset (the option provided by fps video filter) to pts of frames output from fps is still not accurate to the millisecond level.

[Cigaes]

How would it be supposed to work? Changing the timestamps of the frames is exactly how the filter can alter their rate. Preserving the PTS means preserving the initial frame rate.

[vishal733]

I understand your point now, and got the rationale behind fps filter changing the pts values.

Btw, just to correct this statement:

Preserving the PTS means preserving the initial frame rate.

Not really. If you don't change the pts, it will just be a vfr(variable frame rate) output, at the desired output fps. Isn't it?
Whereas, by changing pts values, we get a cfr(constant frame rate) at the desired output fps.

I guess my request here could be better worded as "Ability to get vfr at desired fps from fps filter".

Actually my usecase is, I want to allow frames to pass through at 2fps without altering their pts. And then apply more video filters on top of it.
I guess -r option could have worked for me, if it was possible to apply filters after doing -r.

I hope I am making sense?

[Cigaes]

Sorry, no, you are not making any sense. The timestamps of the frames are exactly what is governing their rate. If you want 2 FPS, that means the frames are spaced by exactly 0.5s, and therefore their timestamps are 0.5s apart.

In other words, you are asking for a framerate that is both constant equal to 2 and variable.

[vishal733]

So here is what I'm asking.
Say fps=2 is set

(1) Original frames (all of them):

[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   0 pts:  12012 pts_time:0.125125 pos:  2753847 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   1 pts:  16016 pts_time:0.166833 pos:  2754781 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   2 pts:  20020 pts_time:0.208542 pos:  2754812 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   3 pts:  24024 pts_time:0.25025 pos:  2754843 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   4 pts:  28028 pts_time:0.291958 pos:  2754874 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   5 pts:  32032 pts_time:0.333667 pos:  2754905 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   6 pts:  36036 pts_time:0.375375 pos:  2754936 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   7 pts:  40040 pts_time:0.417083 pos:  2754967 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   8 pts:  44044 pts_time:0.458792 pos:  2754998 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:   9 pts:  48048 pts_time:0.5005  pos:  2754750 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  10 pts:  52052 pts_time:0.542208 pos:  2755029 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  11 pts:  56056 pts_time:0.583917 pos:  2755345 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  12 pts:  60060 pts_time:0.625625 pos:  2755376 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  13 pts:  64064 pts_time:0.667333 pos:  2755407 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  14 pts:  68068 pts_time:0.709042 pos:  2755438 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  15 pts:  72072 pts_time:0.75075 pos:  2755469 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  16 pts:  76076 pts_time:0.792458 pos:  2755500 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  17 pts:  80080 pts_time:0.834167 pos:  2754718
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  18 pts:  84084 pts_time:0.875875 pos:  2755600 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  19 pts:  88088 pts_time:0.917583 pos:  2755631 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  20 pts:  92092 pts_time:0.959292 pos:  2755662 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  21 pts:  96096 pts_time:1.001   pos:  2755933 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  22 pts: 100100 pts_time:1.04271 pos:  2755964 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  23 pts: 104104 pts_time:1.08442 pos:  2755995 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  24 pts: 108108 pts_time:1.12613 pos:  2756026 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  25 pts: 112112 pts_time:1.16783 pos:  2756057 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  26 pts: 116116 pts_time:1.20954 pos:  2755568 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  27 pts: 120120 pts_time:1.25125 pos:  2756088 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  28 pts: 124124 pts_time:1.29296 pos:  2756119 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  29 pts: 128128 pts_time:1.33467 pos:  2756150 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  30 pts: 132132 pts_time:1.37637 pos:  2756181 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  31 pts: 136136 pts_time:1.41808 pos:  2756212 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  32 pts: 140140 pts_time:1.45979 pos:  2756243 
[Parsed_showinfo_2 @ 0x7f9cbd601380] n:  33 pts: 144144 pts_time:1.5015  pos:  2756516 

(2) Current output after passing through fps=2 filter:

[Parsed_showinfo_4 @ 0x7f9cbd601c00] n:   0 pts:      0 pts_time:0       pos:  2754812 
[Parsed_showinfo_4 @ 0x7f9cbd601c00] n:   1 pts:      1 pts_time:0.5     pos:  2755438 
[Parsed_showinfo_4 @ 0x7f9cbd601c00] n:   2 pts:      2 pts_time:1       pos:  2755568 

(3) Desired out I'm asking for (via some configuration) fps=2 filter :

[Parsed_showinfo_4 @ 0x7f9cbd601c00] n:   0 pts:      20020    pts_time:0.208542       pos:  2754812 
[Parsed_showinfo_4 @ 0x7f9cbd601c00] n:   1 pts:      68068    pts_time:0.709042     pos:  2755438 
[Parsed_showinfo_4 @ 0x7f9cbd601c00] n:   2 pts:      116116    pts_time:1.20954       pos:  2755568 

(2) Is the current output. And is at 2fps(cfr)
(3) is my desired output. And what I'm calling 2fps(vfr).
Difference between (2), and (3) is that in (3) the pts_time is unaltered. But (3) is effectively still at 2fps (vfr).

While (3) can be achieved even now (-r option), but it's not possible to apply any video filters afer doing a -r.

Let me know what you think now..

[Carl Eugen Hoyos]

This is not how the fps filter is supposed to work, feel free to test the select filter and please continue this discussion on the user mailing list, this is a bug tracker.

[Cigaes]

pts_time:0.125125
pts_time:0.166833

→

pts_time:0.208542
pts_time:0.709042

How is it unaltered?

[vishal733]

How is it unaltered?

Basically I meant the pts_time in (3) is unaltered compared to (1).

Anyways.

[vishal733]

Apologies for posting this in the bug tracking section..

[Cigaes]

How is it unaltered when 0.125125 becomes 0.208542? Unaltered means that 0.125125 stays 0.125125.

[vishal733]

That's not the way to look at it..

Steps:
i) First skip frames at 0.125125, 0.166833
ii) Pick frame at 0.208542 (Frame n=0 output from fps filter)
iii) Skip frames at 0.25025, 0.291958, 0.333667, 0.375375, 0.417083, 0.458792, 0.5005, 0.542208, 0.583917, 0.625625, 0.667333
iv) Pick frame at 0.709042 (Frame n=1 output from fps filter)

The frames skipped are based upon the fps value given to fps filter.

Now fps filteradditionally resets the pts_time in step (ii), and step(iv) to 0, and 0.5 respectively (which qualifies as a cfr output from fps filter, as intended by this filter)
Whereas what I need is for fps filter to not reset the pts_time in step(ii), and step(iv). That's what I meant by unaltered pts_time. (maybe even this will qualify as a cfr, and not really a vfr!)

[Hendrik]

You should look into the select filter. It looks flexible enough to design an expression that does what you want. Its specifically designed to only select frames given a certain criteria.

[Cigaes]

What you are asking will not give a constant frame rate, which is the role of the fps filter.

[vishal733]

Thanks Cigaes. Yes, after looking at the details of select filter, I think it should suffice!