Opened 3 months ago

Last modified 3 months ago

#8778 open defect

Ogg Vorbis HTTP Stream Excess Memory/CPU Usage

Reported by: Kyra Owned by:
Priority: important Component: avformat
Version: git-master Keywords: ogg regression
Cc: Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

Summary of the bug:

HTTP streams featuring Ogg Vorbis content appear to leak memory with the resident size increasing in pace with the input data. Eventually constant mmap and munmap calls in the main ffmpeg thread cause an increase in CPU usage well beyond reason.

This does not occur with ffmpeg 4.2.3, nor does it seem to occur with other codecs (MP3 and AAC at least).

How to reproduce:

ffmpeg -i http://sharprouter.sharp-stream.com:8000/totallyradio.ogg -f null -                                                                          
ffmpeg version 4.3 Copyright (c) 2000-2020 the FFmpeg developers                                                                                                       
  built with gcc 9.3.0 (Gentoo 9.3.0-r1 p3)

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

Change History (11)

comment:1 Changed 3 months ago by cehoyos

  • Resolution set to needs_more_info
  • Status changed from new to closed

Please test current FFmpeg git head and please provide valgrind output for memory leaks.

comment:2 Changed 3 months ago by Kyra

  • Component changed from undetermined to avformat
  • Resolution needs_more_info deleted
  • Status changed from closed to reopened
  • Version changed from unspecified to git-master

Latest HEAD tested and a run through Valgrind yielded little benefit.

Bisecting commits points to e983197cbc93420b67aa7e811be47d7278c2c8a2 being the apparent cause.

Last edited 3 months ago by cehoyos (previous) (diff)

comment:3 Changed 3 months ago by Kyra

Specifically the call to ffio_ensure_seekback seems to be key.

comment:4 Changed 3 months ago by richardpl

How much more memory is used?

comment:5 Changed 3 months ago by Kyra

Progressively more as the stream continues (i.e. as much as the input data), along with CPU usage as this is constantly reallocated.

comment:6 Changed 3 months ago by Kyra

  • Summary changed from Memory Leak With Ogg Vorbis HTTP Streams to Ogg Vorbis HTTP Stream Excess Memory/CPU Usage

comment:7 Changed 3 months ago by richardpl

When usage of memory/cpu becomes noticeable? After how many minutes of input stream?

comment:8 follow-up: Changed 3 months ago by Kyra

Increased memory usage should be apparent right away (watch RES in htop or such), increased CPU should be apparent after about an hour. Running strace on the main process should show the reason for the increased CPU.

comment:9 in reply to: ↑ 8 Changed 3 months ago by cehoyos

  • Keywords ogg regression added

Replying to Kyra:

Increased memory usage should be apparent right away (watch RES in htop or such)

Of course.

increased CPU should be apparent after about an hour.

(As also asked on irc)
How can I reproduce this: Which systems shows CPU usage when allocating memory?

comment:10 Changed 3 months ago by Kyra

It's more the constant reallocation of memory which greatly increases CPU usage. For example on a Rock Pi S (with only 512MiB of RAM) it jumps to 46MiB resident using ~10% of the CPU (RK3308 at 1.3GHz) after five minutes.

getsockopt(3, SOL_SOCKET, SO_RCVBUF, [367360], [4]) = 0
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 1 ([{fd=3, revents=POLLIN}], left {tv_sec=0, tv_nsec=99992127})
recvfrom(3, "n\335;\311\345\333I\362\301\200;z\262\333+w\326\24STV\225\7]\220d-8\362\250\374\323"..., 32768, 0, NULL, NULL) = 4194
pselect6(1, [0], NULL, NULL, {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
mmap(NULL, 7061504, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f80d04000
munmap(0x7f813c0000, 7057408)           = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [367360], [4]) = 0
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 0 (Timeout)
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 1 ([{fd=3, revents=POLLIN}], left {tv_sec=0, tv_nsec=64956041})
recvfrom(3, " hS\254\351\204\371#\361\33\315\362Q\22\366s\246\216\263\332\320R\234C\352\326\7=\313\375\t\257"..., 32768, 0, NULL, NULL) = 236
pselect6(1, [0], NULL, NULL, {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 0 (Timeout)
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=0, tv_nsec=100000000}, NULL, 0) = 1 ([{fd=3, revents=POLLIN}], left {tv_sec=0, tv_nsec=57274531})
recvfrom(3, "OggS\0\0\0\241N\0\0\0\0\0\356\4@Kr\1\0\0L\304\372\200\34\3772\3776\377"..., 32768, 0, NULL, NULL) = 2896
mmap(NULL, 7065600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f81a78000
munmap(0x7f80d04000, 7061504)           = 0
Last edited 3 months ago by Kyra (previous) (diff)

comment:11 Changed 3 months ago by cehoyos

  • Priority changed from normal to important
  • Reproduced by developer set
  • Status changed from reopened to open

Not sure why but the issue is reproducible.

Note: See TracTickets for help on using tickets.