Memory consumption

[Mark]

Dear,

First of all please excuse my bad english.

i'm working with ffmpeg for a time and it seems that i 've found a memory leak bug.

• I'm trying to capture a mpeg2ts stream and output only raw audio to a pipe or wav file.
• The main problem I encounter is that ffmpeg each 5 seconds consumer a range of 116-128 K of ram memory more. It never stops of taking more and more memory.
• The command line I'm using:

  ffmpeg -y -v 9 -loglevel 99 -i http://10.42.43.200:1245 -vn -ar 32000 -ac 2 -acodec pcm_s16le -f s16le test/test.raw
  

or

ffmpeg -y -v 9 -loglevel 99 -i http://10.42.43.200:1245 -vn -ar 32000 -ac 2 -acodec pcm_s16le -f s16le -

• Full output:

Well I have some outputs to attach.
First I attach full ffmpeg output:

mark@******:~$ ffmpeg -y -v 9 -loglevel 99 -i http://10.42.43.200:1245 -vn -ar 32000 -ac 2 -acodec pcm_s16le -f s16le test/test.raw
ffmpeg version N-39365-gc44417e Copyright (c) 2000-2012 the FFmpeg developers
  built on Mar 28 2012 16:34:14 with gcc 4.4.5
  configuration: --enable-libdc1394 --prefix=/usr --extra-cflags='-Wall -g ' --enable-shared --enable-libmp3lame --enable-gpl --enable-libvorbis --enable-pthreads --enable-libfaac --enable-libxvid --enable-postproc --enable-x11grab --enable-libgsm --enable-libtheora --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libx264 --enable-libspeex --enable-nonfree --disable-stripping --enable-avfilter --enable-libdirac --disable-decoder=libdirac --enable-libschroedinger --disable-encoder=libschroedinger --enable-version3 --enable-libopenjpeg --enable-libvpx --enable-librtmp --extra-libs=-lgcrypt --disable-altivec --disable-armv5te --disable-armv6 --disable-vis
  libavutil      51. 44.100 / 51. 44.100
  libavcodec     54. 12.100 / 54. 12.100
  libavformat    54.  3.100 / 54.  3.100
  libavdevice    53.  4.100 / 53.  4.100
  libavfilter     2. 66.100 /  2. 66.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 10.100 /  0. 10.100
  libpostproc    52.  0.100 / 52.  0.100
[ogg @ 0x106f380] Format ogg probed with size=2048 and score=100
[ogg @ 0x106f380] Codec not found
st:4 removing common factor 400000 from timebase
[mpeg2video @ 0x1078c20] mpeg_decode_postinit() failure
    Last message repeated 4 times
[ogg @ 0x106f380] max_analyze_duration 5000000 reached at 5016000
[ogg @ 0x106f380] Could not find codec parameters (Unknown: none)
rfps: 24.916667 0.015788
rfps: 25.000000 0.000000
rfps: 25.083333 0.015788
rfps: 49.916667 0.015788
    Last message repeated 1 times
rfps: 50.000000 0.000000
    Last message repeated 1 times
rfps: 50.083333 0.015788
    Last message repeated 1 times
[ogg @ 0x106f380] Estimating duration from bitrate, this may be inaccurate
Input #0, ogg, from 'http://10.42.43.200:1245':
  Duration: N/A, start: 0.000000, bitrate: 15352 kb/s
    Stream #0:0, 0, 1/1000000: Unknown: none
    Stream #0:1, 211, 1/48000: Audio: mp2, 48000 Hz, 1 channels, s16, 96 kb/s
    Stream #0:2, 210, 1/48000: Audio: mp2, 48000 Hz, 1 channels, s16, 96 kb/s
    Stream #0:3, 207, 1/48000: Audio: mp2, 48000 Hz, 2 channels, s16, 160 kb/s
    Stream #0:4, 125, 1/25: Video: mpeg2video (Main) (mpgv / 0x7667706D), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 1/50, 15000 kb/s, 25 fps, 25 tbr, 25 tbn, 50 tbc
Output #0, s16le, to 'test/test.raw':
  Metadata:
    encoder         : Lavf54.3.100
    Stream #0:0, 0, 1/90000: Audio: pcm_s16le, 32000 Hz, 2 channels, s16, 1024 kb/s
Stream mapping:
  Stream #0:3 -> #0:0 (mp2 -> pcm_s16le)
Press [q] to stop, [?] for help
size=   53841kB time=07:47:12.21 bitrate=  15.7kbits/s
video:0kB audio:53841kB global headers:0kB muxing overhead 0.000000%
Received signal 2: terminating.

I also took time to take a output from the memory is consuming every 5 seconds and the difference between last read:

mark@******:~$ j=0 ; while true ; do pid=`ps aux | grep ffmpeg | grep -v grep | awk '{print $2}'` ; i=`ps aux | grep /home/mark/ffmpeg | grep -v grep | awk '{print $6}'` ; let diff=$i-$j ; echo -e "Pid $pid uses memory $i , diff: $diff" ; j=$i ; sleep 5 ; done
Pid 7836 uses memory 10684 , diff: 10684
Pid 7836 uses memory 10684 , diff: 0
Pid 7836 uses memory 10684 , diff: 0
Pid 7836 uses memory 10684 , diff: 0
Pid 7836 uses memory 11740 , diff: 1056
Pid 7836 uses memory 11856 , diff: 116
Pid 7836 uses memory 11976 , diff: 120
Pid 7836 uses memory 12092 , diff: 116
Pid 7836 uses memory 12208 , diff: 116
Pid 7836 uses memory 12328 , diff: 120
Pid 7836 uses memory 12448 , diff: 120
Pid 7836 uses memory 12564 , diff: 116
Pid 7836 uses memory 12684 , diff: 120
Pid 7836 uses memory 12800 , diff: 116
Pid 7836 uses memory 12916 , diff: 116
Pid 7836 uses memory 13032 , diff: 116
Pid 7836 uses memory 13156 , diff: 124
Pid 7836 uses memory 13272 , diff: 116
Pid 7836 uses memory 13388 , diff: 116
Pid 7836 uses memory 13508 , diff: 120
Pid 7836 uses memory 13624 , diff: 116
Pid 7836 uses memory 13740 , diff: 116
Pid 7836 uses memory 13860 , diff: 120
Pid 7836 uses memory 13980 , diff: 120
Pid 7836 uses memory 14096 , diff: 116
Pid 7836 uses memory 14216 , diff: 120
Pid 7836 uses memory 14332 , diff: 116
Pid 7836 uses memory 14448 , diff: 116
Pid 7836 uses memory 14572 , diff: 124
Pid 7836 uses memory 14688 , diff: 116
Pid 7836 uses memory 14804 , diff: 116
Pid 7836 uses memory 14920 , diff: 116
Pid 7836 uses memory 15040 , diff: 120
Pid 7836 uses memory 15156 , diff: 116
Pid 7836 uses memory 15276 , diff: 120
Pid 7836 uses memory 15396 , diff: 120
Pid 7836 uses memory 15512 , diff: 116
Pid 7836 uses memory 15628 , diff: 116
Pid 7836 uses memory 15748 , diff: 120
Pid 7836 uses memory 15868 , diff: 120
Pid 7836 uses memory 15984 , diff: 116
Pid 7836 uses memory 16104 , diff: 120
Pid 7836 uses memory 16220 , diff: 116
Pid 7836 uses memory 16336 , diff: 116
Pid 7836 uses memory 16452 , diff: 116
Pid 7836 uses memory 16576 , diff: 124
Pid 7836 uses memory 16692 , diff: 116
Pid 7836 uses memory 16808 , diff: 116
Pid 7836 uses memory 16928 , diff: 120
Pid 7836 uses memory 17044 , diff: 116
Pid 7836 uses memory 17160 , diff: 116

And here I have a similar output after running for 10 hours (memory is taking 657292K):

Pid 762 uses memory 651164 , diff: 120
Pid 762 uses memory 651284 , diff: 120
Pid 762 uses memory 651400 , diff: 116
Pid 762 uses memory 651516 , diff: 116
Pid 762 uses memory 651636 , diff: 120
Pid 762 uses memory 651752 , diff: 116
Pid 762 uses memory 651872 , diff: 120
Pid 762 uses memory 651992 , diff: 120
Pid 762 uses memory 652108 , diff: 116
Pid 762 uses memory 652224 , diff: 116
Pid 762 uses memory 652340 , diff: 116
Pid 762 uses memory 652460 , diff: 120
Pid 762 uses memory 652580 , diff: 120
Pid 762 uses memory 652696 , diff: 116
Pid 762 uses memory 652816 , diff: 120
Pid 762 uses memory 652932 , diff: 116
Pid 762 uses memory 653048 , diff: 116
Pid 762 uses memory 653168 , diff: 120
Pid 762 uses memory 653288 , diff: 120
Pid 762 uses memory 653404 , diff: 116
Pid 762 uses memory 653524 , diff: 120
Pid 762 uses memory 653640 , diff: 116
Pid 762 uses memory 653756 , diff: 116
Pid 762 uses memory 653880 , diff: 124
Pid 762 uses memory 653996 , diff: 116
Pid 762 uses memory 654112 , diff: 116
Pid 762 uses memory 654228 , diff: 116
Pid 762 uses memory 654348 , diff: 120
Pid 762 uses memory 654464 , diff: 116
Pid 762 uses memory 654584 , diff: 120
Pid 762 uses memory 654704 , diff: 120
Pid 762 uses memory 654820 , diff: 116
Pid 762 uses memory 654936 , diff: 116
Pid 762 uses memory 655056 , diff: 120
Pid 762 uses memory 655176 , diff: 120
Pid 762 uses memory 655292 , diff: 116
Pid 762 uses memory 655412 , diff: 120
Pid 762 uses memory 655528 , diff: 116
Pid 762 uses memory 655644 , diff: 116
Pid 762 uses memory 655760 , diff: 116
Pid 762 uses memory 655884 , diff: 124
Pid 762 uses memory 656000 , diff: 116
Pid 762 uses memory 656116 , diff: 116
Pid 762 uses memory 656236 , diff: 120
Pid 762 uses memory 656352 , diff: 116
Pid 762 uses memory 656468 , diff: 116
Pid 762 uses memory 656592 , diff: 124
Pid 762 uses memory 656708 , diff: 116
Pid 762 uses memory 656824 , diff: 116
Pid 762 uses memory 656944 , diff: 120
Pid 762 uses memory 657060 , diff: 116
Pid 762 uses memory 657176 , diff: 116
Pid 762 uses memory 657292 , diff: 116

[Carl Eugen Hoyos]

For a memory leak, valgrind output is absolutely essential.

Please always test current git head.

Please understand that "memory leak" is very well defined and describes an (often) important bug, while memory consumption is sometimes inevitable.

[Mark]

Dear,

I did a valgrind output. I attach.

As this is a huge problem for me, i was doing some tests and I think is interesting what I've found.

This stream I'm trying to record is a DVB-T stream that I'm launching with a vlc instance. I tryed to replace the default muxer from OGG to TS and the memory leak isn't there anymore.

My conclusions are that the problem is with OGG demuxer itself.

I wantet to take oportunity to explain that I tryed this with many versions. version from debian multimedia repo (0.7.1), and also I compiled yesterday with this git as source : ​git://source.ffmpeg.org/ffmpeg.git

Here you have a output of memory consumption of same channel, same stream with TS as muxer:

mark@******:~$ j=0 ; while true ; do pid=`ps aux | grep /home/mark/ffmpeg | grep -v grep | awk '{print $2}'` ; i=`ps aux | grep /home/mark/ffmpeg | grep -v grep | awk '{print $6}'` ; let diff=$i-$j ; echo -e "Pid $pid uses memory $i , diff: $diff" ; j=$i ; sleep 5 ; done
Pid 590 uses memory 10140 , diff: 10140
Pid 590 uses memory 10868 , diff: 728
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0
Pid 590 uses memory 10868 , diff: 0

[Carl Eugen Hoyos]

Any reason you did not use memcheck which shows a very user-friendly output?

[Mark]

Because when you kill program it frees al blocks and tells there is no memory leak.

Now I run it with memcheck with this command line:

valgrind -v --tool=memcheck --leak-check=full /home/mark/ffmpeg -y -v 9 -loglevel 99 -i http://10.42.43.200:1260 -vn -ar 32000 -ac 2 -acodec pcm_s16le -f s16le test/test.raw

And what I see is that every time takes more and more memory:

--14609-- REDIR: 0xffffffffff600000 (???) redirected to 0x38040893 (vgPlain_amd64_linux_REDIR_FOR_vgettimeofday)
--14609-- memcheck GC: 1024 nodes, 1024 survivors (100.0%)
--14609-- memcheck GC: increase table size to 2048
--14609-- memcheck GC: 2048 nodes, 2048 survivors (100.0%)
--14609-- memcheck GC: increase table size to 4096
--14609-- memcheck GC: 4096 nodes, 3072 survivors ( 75.0%)
--14609-- memcheck GC: increase table size to 8192
--14609-- memcheck GC: 8192 nodes, 7168 survivors ( 87.5%)
--14609-- memcheck GC: increase table size to 16384
size=   29640kB time=00:08:23.04 bitrate= 482.7kbits/s    

But when I close it, the report says:

--14609-- REDIR: 0xffffffffff600000 (???) redirected to 0x38040893 (vgPlain_amd64_linux_REDIR_FOR_vgettimeofday)
--14609-- memcheck GC: 1024 nodes, 1024 survivors (100.0%)
--14609-- memcheck GC: increase table size to 2048
--14609-- memcheck GC: 2048 nodes, 2048 survivors (100.0%)
--14609-- memcheck GC: increase table size to 4096
--14609-- memcheck GC: 4096 nodes, 3072 survivors ( 75.0%)
--14609-- memcheck GC: increase table size to 8192
--14609-- memcheck GC: 8192 nodes, 7168 survivors ( 87.5%)
--14609-- memcheck GC: increase table size to 16384
size=   30123kB time=00:08:26.90 bitrate= 486.8kbits/s    
video:0kB audio:30123kB global headers:0kB muxing overhead 0.000000%
Received signal 2: terminating.
==14609== 
==14609== HEAP SUMMARY:
==14609==     in use at exit: 0 bytes in 0 blocks
==14609==   total heap usage: 47,940 allocs, 47,940 frees, 447,989,754 bytes allocated
==14609== 
==14609== All heap blocks were freed -- no leaks are possible
==14609== 
==14609== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)
--14609-- 
--14609-- used_suppression:      2 dl-hack3-cond-1
--14609-- used_suppression:      2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a
==14609== 
==14609== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)

There is anything else I can do to help with this bug?

Thanks anyways for your time.

[Carl Eugen Hoyos]

Replying to kriptha:

Because when you kill program it frees al blocks and tells there is no memory leak.

I hope you understand now why I did not believe there is a memleak.

Is the memory consumption you see also reproducible with a file or only with http input?

[Mark]

I dumped a file with OGG mux into the system, but the bug isn't there when I work with local files.

Seems to be only with http access.

[Michael Niedermayer]

Is there some way by which we can reproduce this issue? some public http with which this is reproduceable ?

[gineta]

Yes you can reproduce this problem. There a GPL software Clipbucket. If you mass upload a collection of files and start processing 2 at the time or 1 big size the memory never clean up after end the process. I already test video conversion in one VPS with 15 GB ram and after start processing I got 12GB block . If you start again any other bath at the end you run out of memory. I already monitor http, mysql and rest and only ffmpeg reproduce that problem.

[Roger Pack]

I think what he wants is some scripted way to reproduce it...or some easy way...like a loop that does some processing or what not (a simplified test case--i.e. in the heart of Clipbucket what is doing it)?

[Mark]

If you want some way to reproduce this locally and you have a TV tuner or something else, I can show you configurations to setup easy testing environment. You can do it very easy for example with VLC and later ffmpeg.

Also I can script some test so you have most of work done.

Let me know if you need my help.

[Michael Niedermayer]

Replying to kriptha:

If you want some way to reproduce this locally and you have a TV tuner or something else, I can show you configurations to setup easy testing environment. You can do it very easy for example with VLC and later ffmpeg.

Also I can script some test so you have most of work done.

yes this would help alot

[Michael Niedermayer]

If someone can provide a testcase with which the large memory consumption can be reproduced then please reopen this bug