Output "-ss" memory consumption regression

[Bryce Chester Newman]

Summary of the bug:

There appears to be a behavior change or bug introduced between FFmpeg 6.1 and 7.0/7.1 when using the “-ss” argument that is causing a substantial memory increase.
Running the command below using FFmpeg 6.1 memory usage is about 500MB – which is expected.
Running the command below using FFmpeg 7.0/7.1 memory usage is about 3GB – which is not expected.
If I remove the “-ss” argument when running on FFmepg 7.0 or 7.1 the memory footprint is the same as 6.1. I can reproduce this on Linux(Ubuntu) and Apple M2. I can reproduce the memory increase using .mxf, mov and .mp4 as an input file.

How to reproduce:

ffmpeg -report -v 9 -loglevel 99 -y -i 1951279840.mov  \
-filter_complex "[0:v:0]scale='768:432':force_original_aspect_ratio=decrease:force_divisible_by=2[jpg-600];[0:v:0]scale='1280:640'[mp4-640]" \
-map "[jpg-600]" -ss 67 -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg \
-map "[mp4-640]" -map_metadata -1 -f mp4 -vcodec libx264 -profile:v main -level 3.1 -crf 23 -b:v 2500K -movflags faststart -refs 4 -pix_fmt yuv420p -color_primaries bt709 -color_trc bt709 -colorspace bt709 -preset medium /tmp/mp4-640-23456.mp4

[Bryce Chester Newman]

FFmpeg report log

[Bryce Chester Newman]

I think you should be able to reproduce this using just about any video file, but please let me know if you need an example file.

[Marton Balint]

Are you using the -ss after the -i intentionally? Because the parameter position controls if ffmpeg seeks in the input file, or decodes the whole input up to the seek point.

[Bryce Chester Newman]

Replying to Marton Balint:

Are you using the -ss after the -i intentionally? Because the parameter position controls if ffmpeg seeks in the input file, or decodes the whole input up to the seek point.

Yes. We don't want to use the -ss before the -i, otherwise as you have stated, the behavior will skip to the position the in video stream that is after the start of the video and that will cause the video stream output -map "[mp4-640]" to start wherever -ss value was set. We have been using the -ss arg in "-map "[jpg-600]" -ss 67 -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg" and the full example command in the same position for years, so something has changed in 7.x that is causing a 500% increase memory allocation.

[Bryce Chester Newman]

For now, we can't upgrade to 7.x until this issue is fixed.

[MasterQuestionable]

͏    Is it reproducible with?
͏    ffmpeg -y -v debug -hide_banner -nostdin -nostats -i "1951279840.mov" -ss 67 -vframes 1 "!.jpg"

[Gyan]

A workaround for now is to ingest the input twice, once with a seek for the image output, and once without for the video, and drop the output ss, like this,

ffmpeg -i 1951279840.mov -ss 67 -i 1951279840.mov \
-filter_complex "[1:v:0]scale='768:432':force_original_aspect_ratio=decrease:force_divisible_by=2[jpg-600];[0:v:0]scale='1280:640'[mp4-640]" \
-map "[jpg-600]" -f image2 -q:v 5 -update 1 -frames:v 1 /tmp/jpg-600-123456.jpg \
-map "[mp4-640]" ... /tmp/mp4-640-23456.mp4

[MasterQuestionable]

͏    It should also be possible to use "select" for the extraction of after 67 s:
͏    Though potentially less efficient than input "-ss".
͏    And in some cases less precise. ("-ss" alike shall match the nearest eligible frame)

͏    E.g. "select='gte( t, 67 )'"
͏    See also: ​https://trac.ffmpeg.org/ticket/11211#comment:1

[Bryce Chester Newman]

I appreciate the workaround Gyan, but let's say that -ss is 3 seconds and the video was 30 minutes in duration, wouldn't ffmpeg try to read the rest of the remainder of the file? This is also seems like it would not be a good option for other workflows we have that stream the file.

As for using "select='gte( t, 67 )'", what is "less efficient", cpu usage, memory, time, etc? Also, what are the cases that select would be less precise?

Is using the -ss arg something that is planned to be fixed? I can wait for an upgrade to 7.x if -ss's usage is fixed.

[MasterQuestionable]

͏    Try with the debug output: which should report the input reading statistics.
͏    "select" by definition may have to evaluate every input frame:
͏    Though didn't cause the trouble in my test case.

͏    Seeking by timestamp (instead of frame number) invariably encounters 1 problem:
͏    No frame at the exact time.
͏    Thus the most sensible approach would be "round" to the nearest frame.
͏    (similar to rounding in math)
͏    .
͏    Which the simple "gte" logic may misalign.

͏    On the seeking things:
͏    ​https://trac.ffmpeg.org/ticket/11060
͏    ; ongoing, but troublesome.

͏    ----

͏    Specifically on your reading the same file concerns:
͏    OS, file system (at times even the hardware) may handle it somewhat.

͏    The frequently accessed data may be cached.
͏    So reloading may be less expensive than expected.
͏    But this is highly implementation dependent.
͏    So preferably to be avoided right from programming.

[Gyan]

Replying to Bryce Chester Newman:

I appreciate the workaround Gyan, but let's say that -ss is 3 seconds and the video was 30 minutes in duration, wouldn't ffmpeg try to read the rest of the remainder of the file?

No. Once the output(s) which consume that input have terminated, that input is closed.