Output "-ss" memory consumption regression

[Bryce Chester Newman]

Summary of the bug:

There appears to be a behavior change or bug introduced between FFmpeg 6.1 and 7.0/7.1 when using the “-ss” argument that is causing a substantial memory increase.
Running the command below using FFmpeg 6.1 memory usage is about 500MB – which is expected.
Running the command below using FFmpeg 7.0/7.1 memory usage is about 3GB – which is not expected.
If I remove the “-ss” argument when running on FFmepg 7.0 or 7.1 the memory footprint is the same as 6.1. I can reproduce this on Linux(Ubuntu) and Apple M2. I can reproduce the memory increase using .mxf, mov and .mp4 as an input file.

How to reproduce:

ffmpeg -report -v 9 -loglevel 99 -y -i 1951279840.mov  \
-filter_complex "[0:v:0]scale='768:432':force_original_aspect_ratio=decrease:force_divisible_by=2[jpg-600];[0:v:0]scale='1280:640'[mp4-640]" \
-map "[jpg-600]" -ss 67 -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg \
-map "[mp4-640]" -map_metadata -1 -f mp4 -vcodec libx264 -profile:v main -level 3.1 -crf 23 -b:v 2500K -movflags faststart -refs 4 -pix_fmt yuv420p -color_primaries bt709 -color_trc bt709 -colorspace bt709 -preset medium /tmp/mp4-640-23456.mp4

[Bryce Chester Newman]

FFmpeg report log

[Bryce Chester Newman]

I think you should be able to reproduce this using just about any video file, but please let me know if you need an example file.

[Marton Balint]

Are you using the -ss after the -i intentionally? Because the parameter position controls if ffmpeg seeks in the input file, or decodes the whole input up to the seek point.

[Bryce Chester Newman]

Replying to Marton Balint:

Are you using the -ss after the -i intentionally? Because the parameter position controls if ffmpeg seeks in the input file, or decodes the whole input up to the seek point.

Yes. We don't want to use the -ss before the -i, otherwise as you have stated, the behavior will skip to the position the in video stream that is after the start of the video and that will cause the video stream output -map "[mp4-640]" to start wherever -ss value was set. We have been using the -ss arg in "-map "[jpg-600]" -ss 67 -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg" and the full example command in the same position for years, so something has changed in 7.x that is causing a 500% increase memory allocation.

[Bryce Chester Newman]

For now, we can't upgrade to 7.x until this issue is fixed.

[MasterQuestionable]

͏    Is it reproducible with?
͏    ffmpeg -y -v debug -hide_banner -nostdin -nostats -i "1951279840.mov" -ss 67 -vframes 1 "!.jpg"

[Gyan]

A workaround for now is to ingest the input twice, once with a seek for the image output, and once without for the video, and drop the output ss, like this,

ffmpeg -i 1951279840.mov -ss 67 -i 1951279840.mov \
-filter_complex "[1:v:0]scale='768:432':force_original_aspect_ratio=decrease:force_divisible_by=2[jpg-600];[0:v:0]scale='1280:640'[mp4-640]" \
-map "[jpg-600]" -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg \
-map "[mp4-640]" ... /tmp/mp4-640-23456.mp4

[MasterQuestionable]

͏    It should also be possible to use "select" for the extraction of after 67 s:
͏    Though potentially less efficient than input "-ss".
͏    And in some cases less precise. ("-ss" alike shall match the nearest eligible frame)

͏    E.g. "select='gte( t, 67 )'"
͏    See also: ​https://trac.ffmpeg.org/ticket/11211#comment:1

[Bryce Chester Newman]

I appreciate the workaround Gyan, but let's say that -ss is 3 seconds and the video was 30 minutes in duration, wouldn't ffmpeg try to read the rest of the remainder of the file? This is also seems like it would not be a good option for other workflows we have that stream the file.

As for using "select='gte( t, 67 )'", what is "less efficient", cpu usage, memory, time, etc? Also, what are the cases that select would be less precise?

Is using the -ss arg something that is planned to be fixed? I can wait for an upgrade to 7.x if -ss's usage is fixed.

[MasterQuestionable]

͏    Try with the debug output: which should report the input reading statistics.
͏    "select" by definition may have to evaluate every input frame:
͏    Though didn't cause the trouble in my test case.

͏    Seeking by timestamp (instead of frame number) invariably encounters 1 problem:
͏    No frame at the exact time.
͏    Thus the most sensible approach would be "round" to the nearest frame.
͏    (similar to rounding in math)
͏    .
͏    Which the simple "gte" logic may misalign.

͏    On the seeking things:
͏    ​https://trac.ffmpeg.org/ticket/11060
͏    ; ongoing, but troublesome.

͏    ----

͏    Specifically on your reading the same file concerns:
͏    OS, file system (at times even the hardware) may handle it somewhat.

͏    The frequently accessed data may be cached.
͏    So reloading may be less expensive than expected.
͏    But this is highly implementation dependent.
͏    So preferably to be avoided right from programming.

[Gyan]

Replying to Bryce Chester Newman:

I appreciate the workaround Gyan, but let's say that -ss is 3 seconds and the video was 30 minutes in duration, wouldn't ffmpeg try to read the rest of the remainder of the file?

No. Once the output(s) which consume that input have terminated, that input is closed.

[Bryce Chester Newman]

Any chance this might be fixed in a future release or in master anytime soon?

[MasterQuestionable]

͏    Need more precise regression window...

͏    Also please confirm?
͏    ​https://trac.ffmpeg.org/ticket/11217#comment:5

[Bryce Chester Newman]

Is it reproducible with ffmpeg -y -v debug -hide_banner -nostdin -nostats -i "1951279840.mov" -ss 67 -vframes 1 "!.jpg"? No, but that is not the issue. That command has always worked as expected in FFmpeg 6 and 7. The issue is reproducible with `ffmpeg -report -v 9 -loglevel 99 -y -i 1951279840.mov \
-filter_complex "[0:v:0]scale='768:432':force_original_aspect_ratio=decrease:force_divisible_by=2[jpg-600];[0:v:0]scale='1280:640'[mp4-640]" \
-map "[jpg-600]" -ss 67 -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg \
-map "[mp4-640]" -map_metadata -1 -f mp4 -vcodec libx264 -profile:v main -level 3.1 -crf 23 -b:v 2500K -movflags faststart -refs 4 -pix_fmt yuv420p -color_primaries bt709 -color_trc bt709 -colorspace bt709 -preset medium /tmp/mp4-640-23456.mp4`

The work around is not a viable work around as this comes with an additional cost for us to request the file two or more times when using streaming.

[Bryce Chester Newman]

Using the work around as shown here...

`time ffmpeg -y -i 1951279840.mov -i 1951279840.mov \
-filter_complex "[0:v:0]scale='768:432':force_original_aspect_ratio=decrease:force_divisible_by=2[jpg-600];[1:v:0]scale='1280:640'[mp4-640]" \
-map "[jpg-600]" -ss 67 -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg \
-map "[mp4-640]" -map_metadata -1 -f mp4 -vcodec libx264 -profile:v main -level 3.1 -crf 23 -b:v 2500K -movflags faststart -refs 4 -pix_fmt yuv420p -color_primaries bt709 -color_trc bt709 -colorspace bt709 -preset medium /tmp/mp4-640-23456.mp4`

...there are some observations worth noting.

1) Memory is as expected when using the above command, but like I mentioned accessing the file more than once has a cost for us.
2) User Time. The CPU time outside the kernel increased by 35% compared to the original command in this issue.
3) System Time. Increased by 3% compared to the original command in this issue.
4) Time to execute increased by 30%(or 3 seconds) compared to the original command in this issue.

[MasterQuestionable]

͏    That's the minimal reproducer derived from your origin command.
͏    Compare the behavior between FFmpeg 6, 7: is alike anomaly present?

[Bryce Chester Newman]

Are you asking if the substantial memory increase occurs in FFmpeg 6 given the original FFmpeg command and args from this issue? If so, then the answer is no, the substantial memory increase does not occur in FFmpeg version 6 or less.

Or are you asking to run the command below on FFmpeg 6 and report on the observations?

`time ffmpeg -y -i 1951279840.mov -i 1951279840.mov \
-filter_complex "[0:v:0]scale='768:432':force_original_aspect_ratio=decrease:force_divisible_by=2[jpg-600];[1:v:0]scale='1280:640'[mp4-640]" \
-map "[jpg-600]" -ss 67 -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg \
-map "[mp4-640]" -map_metadata -1 -f mp4 -vcodec libx264 -profile:v main -level 3.1 -crf 23 -b:v 2500K -movflags faststart -refs 4 -pix_fmt yuv420p -color_primaries bt709 -color_trc bt709 -colorspace bt709 -preset medium /tmp/mp4-640-23456.mp4`

[MasterQuestionable]

͏    I mean:
͏    If the command in comment:5, would cause alike resource consumption increase alike?
͏    (comparing FFmpeg 6 and 7)

͏    The minimal command shall of course consume much less resource than your original.
͏    But if alike problems present, there shall be alike resource consumption anomaly.

[Bryce Chester Newman]

I ran the command three times on 6.1 and 7.1.
Notice FFmpeg 7.1 uses 24% more memory than 6.1.
Notice FFmpeg 7.1 uses less real/user cpu than 6.1.
Notice FFmpeg 6.1 is very slightly faster than 7.1.

FFMPEG 7.1
ffmpeg -y -v debug -hide_banner -nostdin -nostats -i "1951279840.mov" -ss 67 -vframes 1 "\!.jpg"

real 0m9.410s
user 0m43.076s
sys 0m2.538s
mem 392.7MB

real 0m9.617s
user 0m43.191s
sys 0m2.689s
mem 392.5MB

real 0m9.534s
user 0m43.498s
sys 0m2.688s
mem 392.2MB

FFMPEG 6.1
ffmpeg -y -v debug -hide_banner -nostdin -nostats -i "1951279840.mov" -ss 67 -vframes 1 "\!.jpg"

real 0m12.018s
user 0m53.972s
sys 0m2.495s
mem 315.2MB

real 0m11.919s
user 0m54.095s
sys 0m2.426s
mem 315.6MB

real 0m11.986s
user 0m53.905s
sys 0m2.461s
mem 315.3MB

[Gyan]

Notice FFmpeg 6.1 is very slightly faster than 7.1.

7.1 takes an average of ~9.5s whereas 6.1 takes ~11.96s

[Bryce Chester Newman]

wrt "7.1 takes an average of ~9.5s whereas 6.1 takes ~11.96s" that - in my opinion - is not at all important and is not relevant to the issue I reported when I opened this issue. I feel like the original issue I reported is somehow being contorted. I am really not sure what is being proven here by the observation or the latest comments about time while executing ffmpeg -y -v debug -hide_banner -nostdin -nostats -i "1951279840.mov" -ss 67 -vframes 1 "\!.jpg". The original FFmpeg command I posted to this issue is the problem with memory and is very much reproducible. Where do we go from here?

[MasterQuestionable]

͏    The memory consumption bloat. Alike observed. (though not as drastic)
͏    Which should more clearly hint the cause.

͏    Your original filter-chain probably somehow amplified the problem.