Ogg Vorbis HTTP Stream Excess Memory/CPU Usage

[Kyra]

Summary of the bug:

HTTP streams featuring Ogg Vorbis content appear to leak memory with the resident size increasing in pace with the input data. Eventually constant mmap and munmap calls in the main ffmpeg thread cause an increase in CPU usage well beyond reason.

This does not occur with ffmpeg 4.2.3, nor does it seem to occur with other codecs (MP3 and AAC at least).

How to reproduce:

ffmpeg -i http://sharprouter.sharp-stream.com:8000/totallyradio.ogg -f null -                                                                          
ffmpeg version 4.3 Copyright (c) 2000-2020 the FFmpeg developers                                                                                                       
  built with gcc 9.3.0 (Gentoo 9.3.0-r1 p3)

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Carl Eugen Hoyos]

Please test current FFmpeg git head and please provide valgrind output for memory leaks.

[Kyra]

Latest HEAD tested and a run through Valgrind yielded little benefit.

Bisecting commits points to e983197cbc93420b67aa7e811be47d7278c2c8a2 being the apparent cause.

[Kyra]

Specifically the call to ffio_ensure_seekback seems to be key.

[Elon Musk]

How much more memory is used?

[Kyra]

Progressively more as the stream continues (i.e. as much as the input data), along with CPU usage as this is constantly reallocated.

[Elon Musk]

When usage of memory/cpu becomes noticeable? After how many minutes of input stream?

[Kyra]

Increased memory usage should be apparent right away (watch RES in htop or such), increased CPU should be apparent after about an hour. Running strace on the main process should show the reason for the increased CPU.

[Carl Eugen Hoyos]

Replying to Kyra:

Increased memory usage should be apparent right away (watch RES in htop or such)

Of course.

increased CPU should be apparent after about an hour.

(As also asked on irc)
How can I reproduce this: Which systems shows CPU usage when allocating memory?

[Kyra]

It's more the constant reallocation of memory which greatly increases CPU usage. For example on a Rock Pi S (with only 512MiB of RAM) it jumps to 46MiB resident using ~10% of the CPU (RK3308 at 1.3GHz) after five minutes.

getsockopt(3, SOL_SOCKET, SO_RCVBUF, [367360], [4]) = 0
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 1 ([{fd=3, revents=POLLIN}], left {tv_sec=0, tv_nsec=99992127})
recvfrom(3, "n\335;\311\345\333I\362\301\200;z\262\333+w\326\24STV\225\7]\220d-8\362\250\374\323"..., 32768, 0, NULL, NULL) = 4194
pselect6(1, [0], NULL, NULL, {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
mmap(NULL, 7061504, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f80d04000
munmap(0x7f813c0000, 7057408)           = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [367360], [4]) = 0
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 0 (Timeout)
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 1 ([{fd=3, revents=POLLIN}], left {tv_sec=0, tv_nsec=64956041})
recvfrom(3, " hS\254\351\204\371#\361\33\315\362Q\22\366s\246\216\263\332\320R\234C\352\326\7=\313\375\t\257"..., 32768, 0, NULL, NULL) = 236
pselect6(1, [0], NULL, NULL, {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 0 (Timeout)
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 1 ([{fd=3, revents=POLLIN}], left {tv_sec=0, tv_nsec=57274531})
recvfrom(3, "OggS\0\0\0\241N\0\0\0\0\0\356\4@Kr\1\0\0L\304\372\200\34\3772\3776\377"..., 32768, 0, NULL, NULL) = 2896
mmap(NULL, 7065600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f81a78000
munmap(0x7f80d04000, 7061504)           = 0

[Carl Eugen Hoyos]

Not sure why but the issue is reproducible.

[Elon Musk]

Fixed my Marton Balint.