Opened 2 years ago

Closed 21 months ago

Last modified 21 months ago

#5735 closed defect (fixed)

CPU usage increases over time (possibly caused by loudnorm)

Reported by: yatahaze Owned by:
Priority: normal Component: avfilter
Version: git-master Keywords: loudnorm
Cc: cus@passwd.hu Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

I am taking an RTP stream and adding loudnorm and rebroadcasting it. ffmpeg will use the cpu more and more over time. See this graph: http://puu.sh/qeg9W/8529b8a5b5.png (over a time span of 3 hours).

When I took out the loudnorm filter, it seems to not be increasing slowly over time, or just a lot less. I need more time to verify it's not going up anymore. If it is, it's a lot slower and a lot less dramatic.

My command:

ExecStart?=/root/bin/ffmpeg -loglevel panic -i rtp://127.0.0.1:5001 -c:a libmp3lame -af loudnorm=I=-15 -f tee -map 0:a "[f=rtp]rtp://224.3.3.3:1235|[f=mpegts]udp://192.168.0.113:1235/"

More Info:

# ffmpeg -v 9 -loglevel 99 -i rtp://127.0.0.1:5001
ffmpeg version 3.1.1-1~xenial Copyright (c) 2000-2016 the FFmpeg developers

built with gcc 5.3.1 (Ubuntu 5.3.1-14ubuntu2.1) 20160413
configuration: --prefix=/usr --extra-version='1~xenial' --libdir=/usr/lib/ffmpeg --shlibdir=/usr/lib/ffmpeg --disable-static --disable-debug --toolchain=hardened --enable-pthreads --enable-runtime-cpudetect --enable-gpl --enable-shared --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv --enable-libkvazaar --enable-libopenh264 --enable-nonfree --enable-libfdk-aac --enable-libfaac
libavutil 55. 28.100 / 55. 28.100
libavcodec 57. 48.101 / 57. 48.101
libavformat 57. 41.100 / 57. 41.100
libavdevice 57. 0.101 / 57. 0.101
libavfilter 6. 47.100 / 6. 47.100
libavresample 3. 0. 0 / 3. 0. 0
libswscale 4. 1.100 / 4. 1.100
libswresample 2. 1.100 / 2. 1.100
libpostproc 54. 0.100 / 54. 0.100

Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input file with argument 'rtp://127.0.0.1:5001'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input file rtp://127.0.0.1:5001.
Successfully parsed a group of options.
Opening an input file: rtp://127.0.0.1:5001.
Probing rtp score:100 size:0
[rtp @ 0x25c89a0] No default whitelist set
[udp @ 0x25c8e60] No default whitelist set
[udp @ 0x25c8e60] end receive buffer size reported is 131072
[udp @ 0x25d9140] No default whitelist set
[udp @ 0x25d9140] end receive buffer size reported is 131072
[rtp @ 0x25c69e0] SDP:
v=0
c=IN IP4 127.0.0.1
m=application 5001 RTP/AVP 33

[rtp @ 0x25c69e0] sdp: v='0'
[rtp @ 0x25c69e0] sdp: c='IN IP4 127.0.0.1'
[rtp @ 0x25c69e0] sdp: m='application 5001 RTP/AVP 33'
[rtp @ 0x25c69e0] Filter: pid=0x11 type=1
[rtp @ 0x25c69e0] Filter: pid=0x0 type=1
[rtp @ 0x25c89c0] No default whitelist set
[udp @ 0x25c90e0] No default whitelist set
[udp @ 0x25c90e0] end receive buffer size reported is 131072
[udp @ 0x25c8e80] No default whitelist set
[udp @ 0x25c8e80] end receive buffer size reported is 131072
[rtp @ 0x25c69e0] setting jitter buffer size to 500
[rtp @ 0x25c69e0] PAT:
[rtp @ 0x25c69e0] sid=0x1 pid=0x42
[rtp @ 0x25c69e0] new_program: id=0x0001
[rtp @ 0x25c69e0] Filter: pid=0x42 type=1
[rtp @ 0x25c69e0] PMT: len 27
[rtp @ 0x25c69e0] sid=0x1 sec_num=0/0 version=20 tid=2
[rtp @ 0x25c69e0] pcr_pid=0x47
[rtp @ 0x25c69e0] Filter: pid=0x47 type=0
[rtp @ 0x25c69e0] stream=0 stream_type=3 pid=47 prog_reg_desc=
[rtp @ 0x25c69e0] tag: 0x0a len=4
[rtp @ 0x25c69e0] pid=47 pes_code=0x1
(repeats, works fine except for my cpu usage problem)

Change History (13)

comment:1 Changed 2 years ago by cehoyos

Is network input required to reproduce the issue or is it also reproducible with file or lavfi input?
Is network output required to reproduce?
In any case please remove the loglevel option and provide the complete, uncut console output to make this a valid ticket.

comment:2 Changed 2 years ago by yatahaze

I'm in an office situation, so I need some time to test with a non-network stream. I could try non-rtp easily, but that's not what you asked for.

Here is the full command output without the log level option:

# /root/bin/ffmpeg -i rtp://127.0.0.1:5001 -c:a libmp3lame -af loudnorm=I=-15 -f tee -map 0:a "[f=rtp]rtp://224.3.3.3:1235|[f=mpegts]udp://192.168.0.113:1235/"
ffmpeg version N-81026-gb701ec4 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.1) 20160609
  configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --bindir=/root/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libx264 --enable-nonfree --enable-libebur128
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 50.100 / 57. 50.100
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 47.100 /  6. 47.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
Input #0, rtp, from 'rtp://127.0.0.1:5001':
  Duration: N/A, start: 86568.446233, bitrate: 192 kb/s
  Program 1
    Stream #0:0: Audio: mp2 ([3][0][0][0] / 0x0003), 44100 Hz, stereo, s16p, 192 kb/s
[tee @ 0x2286cc0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[mpegts @ 0x231f600] frame size not set
Output #0, tee, to '[f=rtp]rtp://224.3.3.3:1235|[f=mpegts]udp://192.168.0.113:1235/':
  Metadata:
    encoder         : Lavf57.41.100
    Stream #0:0: Audio: mp3 (libmp3lame), 48000 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc57.50.100 libmp3lame
Stream mapping:
  Stream #0:0 -> #0:0 (mp2 (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
size=N/A time=00:00:22.89 bitrate=N/A speed=1.09x

comment:3 Changed 2 years ago by yatahaze

  • Disabling the loudnorm filter fixes the CPU issue, so I believe that is the component at fault.
  • CPU usage seems to raise by 1% every 10 minutes (or maybe a bit less)
  • I have a graph of before/after loudnorm:

http://puu.sh/qfdHr/2009c29e8e.png

  • I will attempt with a non-network input next.

comment:4 Changed 2 years ago by yatahaze

I'm an ffmpeg noob and have no idea how to stream out a local mp3 file. It plays back at high speed and uses 100% CPU. Can you/someone help me get the command right?

My command is:

# /root/bin/ffmpeg -i /root/music2.mp3 -c:a libmp3lame -af loudnorm=I=-15 -f tee -map 0:a "[f=rtp]rtp://224.3.3.3:1235|[f=mpegts]udp://192.168.0.113:1235/"
ffmpeg version N-81026-gb701ec4 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.1) 20160609
  configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --bindir=/root/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libx264 --enable-nonfree --enable-libebur128
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 50.100 / 57. 50.100
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 47.100 /  6. 47.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mp3, from '/root/music2.mp3':
  Metadata:
    title           : Maschine Sounds (Proton Radio)
    artist          : Oliver Lieb
    album           : Maschine Sounds (Proton Radio)
    date            : 2007
    comment         : TALiON
    genre           : Electronic
  Duration: 02:27:13.18, start: 0.025057, bitrate: 191 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 192 kb/s
    Metadata:
      encoder         : LAME3.97
    Side data:
      replaygain: track gain - -2.500000, track peak - unknown, album gain - unknown, album peak - unknown,
[tee @ 0x23811e0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[mpegts @ 0x2422800] frame size not set
Output #0, tee, to '[f=rtp]rtp://224.3.3.3:1235|[f=mpegts]udp://192.168.0.113:1235/':
  Metadata:
    title           : Maschine Sounds (Proton Radio)
    artist          : Oliver Lieb
    album           : Maschine Sounds (Proton Radio)
    date            : 2007
    comment         : TALiON
    genre           : Electronic
    encoder         : Lavf57.41.100
    Stream #0:0: Audio: mp3 (libmp3lame), 48000 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc57.50.100 libmp3lame
Stream mapping:
  Stream #0:0 -> #0:0 (mp3 (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
size=N/A time=00:07:52.89 bitrate=N/A speed=5.72x

comment:5 Changed 2 years ago by cehoyos

Are you searching for the -re option?

comment:6 Changed 2 years ago by yatahaze

Yes thank you. Testing with that now.

Update on current situation:

  • Running ffmpeg without loudnorm for 1.5 hours: CPU usage started at 16% and finished off at 28%... so it is not loudnorm.
  • Running VLC for the same time period: Started at 3% and ended up at 159%. Wow.

I have no idea why both ffmpeg and vlc would have the same symptoms. They both started rising up at the same time together, which is strange. Something fishy is going on...

http://puu.sh/qfmUR/b276738a25.png

comment:7 Changed 2 years ago by yatahaze

I'm seeing the gradual increase of CPU usage while streaming a local file too.

My command is

root/bin/ffmpeg -re -i /root/music2.mp3 -c:a libmp3lame -af loudnorm=I=-15 -f tee -map 0:a "[f=rtp]rtp://224.3.3.3:1235|[f=mpegts]udp://192.168.0.113:1235/"

comment:8 Changed 2 years ago by cehoyos

Is an mp3 file required as input or is it also reproducible with -f lavfi -i sine?
Is the issue also reproducible with only rtp output? Or only mpegts output? What about mpegts file output (if this is about mpegts)?

comment:9 follow-up: Changed 22 months ago by leonnortje

Can confirm that this behaviour is present in ffmpeg 3.1.2 and is definitely related to the loudnorm filter.

CPU escalates by ±5% every hour until 100% CPU is consumed. (On a 1 CPU Digital Ocean droplet)

Reproducing is as as simple as:

ffmpeg -i -ANYLIVESTREAM- -af loudnorm -f null -

Input format does not seem to make a difference, tried reading from a local Icecast AAC stream, a remote MP3 stream and even a local named pipe receiving WAV, all exhibiting the same behaviour.

Nor does output format seem to make a difference. Pushed to AAC, WAV, even null output causes issue to arise.

Specific loudnorm settings does not seem to make a difference, even using the default values causes issue.

Used as -af switch and in a -filter_complex both with same results.

Full output:

/usr/local/bin/ffmpeg -i http://amber.hydrosila.com:8000/record.mp3 -af loudnorm -f null -
ffmpeg version 3.1.2 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
  configuration: --pkg-config-flags=--static --extra-cflags=-I/home/master/ffmpeg_build/include --extra-ldflags=-L/home/master/ffmpeg_build/lib --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libebur128 --enable-nonfree
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 47.100 /  6. 47.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[mp3 @ 0x3ac63a0] Skipping 218 bytes of junk at 0.
Input #0, mp3, from 'http://amber.hydrosila.com:8000/record.mp3':
  Metadata:
    icy-br          : 128
    icy-description : Hot Dance Music
    icy-genre       : Misc
    icy-name        : Radio Record
    icy-pub         : 1
    icy-url         : http://www.radiorecord.ru
    StreamTitle     : FEDER/ALEX AIONO - Lordly (Record Mix)
  Duration: N/A, start: 0.000000, bitrate: 128 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
[null @ 0x3acdaa0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Output #0, null, to 'pipe:':
  Metadata:
    icy-br          : 128
    icy-description : Hot Dance Music
    icy-genre       : Misc
    icy-name        : Radio Record
    icy-pub         : 1
    icy-url         : http://www.radiorecord.ru
    StreamTitle     : FEDER/ALEX AIONO - Lordly (Record Mix)
    encoder         : Lavf57.41.100
    Stream #0:0: Audio: pcm_s16le, 192000 Hz, stereo, s16, 6144 kb/s
    Metadata:
      encoder         : Lavc57.48.101 pcm_s16le
Stream mapping:
  Stream #0:0 -> #0:0 (mp3 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help

comment:10 in reply to: ↑ 9 Changed 22 months ago by cehoyos

Replying to leonnortje:

Can confirm that this behaviour is present in ffmpeg 3.1.2

Please test current FFmpeg git head.

Do I understand correctly that the issue is only reproducible with network input?

comment:11 Changed 21 months ago by cus

  • Cc cus@passwd.hu added

This should be fixed in 005d058f4230f3207ebcf1131df7426d4f57392f, but please check.

comment:12 Changed 21 months ago by cus

  • Resolution set to fixed
  • Status changed from new to closed

Reopen if this is still an issue.

comment:13 Changed 21 months ago by cehoyos

  • Component changed from undetermined to avfilter
  • Reproduced by developer set
  • Version changed from unspecified to git-master

I can confirm the issue and the fix for the following command line:

$ ffmpeg -i http://amber.hydrosila.com:8000/record.mp3 -af loudnorm -f null -

The change reduced the (initial) cpu load by 75% here.

Note: See TracTickets for help on using tickets.