ffmpeg_demux: readrate plays "catch up" if output is blocked, then later resumed

[Wallboy]

Summary of the bug:

Certain muxers such as FIFO can block the input if the output is not currently writable. Such is the case if the FIFO muxer has drop_pkts_on_overflow 0 and attempt_recovery 1, and you are outputting to something like RTMP, and the RTMP server goes down temporarily.

Assuming the "-re" option was used. aka readrate = 1.

Once the RTMP server reconnects after sometime, the FIFO muxer recovers and can now continue. Now when ffmpeg starts reading from the input and calls readrate_sleep(Demuxer *d), it essentially sees we've fallen behind and needs to play "catch up", and does not sleep and sends packets as quick as possible until it makes up the time it lost.

This means the output you are expecting to be sending at native input rate of 1x instead gets blasted with a surge of packets, which is worse depending on the amount of time the passed before a reconnect was successful. An RTMP server that has a rate limit on ingest would most certainly just disconnect you at this point.

How to reproduce:

We can show this through using ffmpeg running it's own RTMP server:

ffmpeg -f flv -listen 1 -i rtmp://127.0.0.1/app/path -vf showinfo -f null /dev/null

Create a testsrc to send to this server:

ffmpeg -re -f lavfi -i "testsrc=duration=600:size=640x360:rate=30" -c:v libx264 -pix_fmt yuv420p -f fifo -fifo_format flv -attempt_recovery 1 -drop_pkts_on_overflow 0 -map 0 -flags +global_header rtmp://127.0.0.1/app/path

In the server output, take note of the speed of frames from the showinfo filter (the n: value). Now disconnect the ffmpeg RTMP server (Ctrl+C) and wait for some time to pass (30 seconds to a minute). Then start the server back up and wait for the FIFO command to recovery and reconnect.

Now take note of how fast the incoming frames are compared to before. This quickly continues to climb up before it slows back down once the readrate_sleep() function determines it has caught up.

Seems this function would need to be aware of when the output has stalled. Not sure how simple of a fix this would be.

EDIT: The realtime filter could potentially be a solution, but this wouldn't work for if you wanted to use streamcopy.

[Gyan]

Will look into it.

[Wallboy]

I've been messing around with a potential hacky solution.

Just before the following condition:

if (pts - burst_until > now)...

Add the following check to see if we've fallen behind, and then offset wallclock_start and recalculate now:

if (now - pts > 500000) {  // arbitrary 0.5 sec threshold
    d->wallclock_start += now - (pts - burst_until);
    now = (av_gettime_relative() - d->wallclock_start) * d->readrate + stream_ts_offset;
}

This seems to do the job for my use case scenarios, but I'm sure there are some edge cases to be aware of. At least it's some sort of idea for a potential real solution.

[Gyan]

Patch posted:
​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20250218094649.6619-1-ffmpeg@gyani.pro/

[Gyan Doshi <ffmpeg@gyani.pro>]

In 6232f416/ffmpeg:

ffmpeg_demux: add option readrate_catchup

At present, if reading from a readrate-limited input is stalled,
then upon resumption, ffmpeg will read the input without any
throttle till the average readrate matches the specified readrate.

This new option allows to set a speed limit when reading is resumed
until the average readrate matches the primary readrate.

Fixes #11469

[jb_alvarado]

@Gyan, this patch breaks -re and -readrate 1. When I use now -readrate 1 or -re I get a speed around ~0.5. Here is a full output:

./ffmpeg -re -i '/home/jb/Videos/Punkt Licht Trailer ASI HD FINAL.mp4' -f null -
ffmpeg version N-118534-g6232f416b1 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 14 (GCC)
  configuration: --prefix=/home/jb/dev/github/compile-ffmpeg-osx-linux/local --disable-debug --disable-shared --disable-doc --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avfilter
  libavutil      59. 57.100 / 59. 57.100
  libavcodec     61. 33.102 / 61. 33.102
  libavformat    61.  9.107 / 61.  9.107
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  9.100 / 10.  9.100
  libswscale      8. 13.100 /  8. 13.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/jb/Videos/Punkt Licht Trailer ASI HD FINAL.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41
    creation_time   : 2021-10-28T07:40:48.000000Z
  Duration: 00:01:19.08, start: 0.000000, bitrate: 21036 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 20691 kb/s, 25 fps, 25 tbr, 25k tbn (default)
    Metadata:
      creation_time   : 2021-10-28T07:40:48.000000Z
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : AVC Coding
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 317 kb/s (default)
    Metadata:
      creation_time   : 2021-10-28T07:40:48.000000Z
      handler_name    : #Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41
    encoder         : Lavf61.9.107
  Stream #0:0(eng): Video: wrapped_avframe, yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn (default)
    Metadata:
      encoder         : Lavc61.33.102 wrapped_avframe
      creation_time   : 2021-10-28T07:40:48.000000Z
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
  Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      encoder         : Lavc61.33.102 pcm_s16le
      creation_time   : 2021-10-28T07:40:48.000000Z
      handler_name    : #Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
[out#0/null @ 0x2e29a240] video:849KiB audio:14828KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 1977 fps= 14 q=-0.0 Lsize=N/A time=00:01:19.08 bitrate=N/A speed=0.556x    

If I add -readrate_catchup to this command with a minimum of 1.2, I get a final rate of 1. But I would suspect that without -readrate_catchup I would get this too.

[Gyan]

Tested -re with a HD file and I cannot reproduce this.

Test with streamcopy, other files, other machines ..etc

[jb_alvarado]

Have you try a 25 fps video? By me I get different rates, when I use files with different fps.

[Gyan]

I tested with a couple of other files and I did get a lower speed with a high bitrate file.

I think I know what's happening. The default for catchup is the same as the primary readrate. If there are micro-lags during reading which accumulate so that catchup kicks in, then ffmpeg can never catch up with the default catchup rate as it is the same as the primary rate. So it may be prudent to set the catchup rate to a slightly higher value to lower the lag. Remember that the behaviour before this patch was effectively a catchup rate of INFINITY, which is why this ticket was opened.

[jb_alvarado]

Yes I understand. But when I'm not using the catchup parameter I would suspect something like catchup rate of INFINITY.

When I played a file with 1 minute duration, with the old behavior and -re/-readrate 1 it was played almost exactly 1 minute long. Now it takes ~1:39.

Edit:I tested now time ffmpeg -re -f lavfi -i "testsrc=duration=60:size=640x360:rate=25" -f null - and interestingly this is slightly more accurate with the new behavior. 59,974 vs 59,488. So some how with files as input, it works a bit different.

[Gyan]

I've posted a patch to set a higher default for readrate_catchup, so except for very short inputs, the primary readrate should be achieved.

[jb_alvarado]

Sounds good, thank you!

[Gyan]

Pushed as 323cb8c61ea157e01ae98f5a2e3acb8208cfc84a