Opened 11 years ago

Closed 11 years ago

#2298 closed defect (fixed)

High memory allocation with broken pmp file

Reported by: Carl Eugen Hoyos Owned by:
Priority: normal Component: avformat
Version: git-master Keywords: pmp videolan
Cc: reimar, donmoir@comcast.net Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by Carl Eugen Hoyos)

(vlc ticket 8226)
A sample was uploaded to videolan trac that allocates several Gigabyte at probing, I will attach the sample and a patch that improves the situation, but valgrind still reports 1,021,771,097 allocs with 1,902,250,215,888,401,222 bytes with the patch.

$ ffmpeg -i dos.pmp
ffmpeg version N-50197-g23c9180 Copyright (c) 2000-2013 the FFmpeg developers
  built on Feb 23 2013 13:10:17 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack
  libavutil      52. 17.103 / 52. 17.103
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.100 / 54. 63.100
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 39.101 /  3. 39.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[pmp @ 0x32a8520] Unsupported audio format
[NULL @ 0x32ae8e0] [IMGUTILS @ 0x7fffe55dd7e0] Picture size 1094795585x1094795585 is invalid
[NULL @ 0x32ae8e0] Ignoring invalid width/height values
[pmp @ 0x32a8520] Could not find codec parameters for stream 0 (Video: mpeg4, yuv420p): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 1 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 2 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 3 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 4 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 5 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 6 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 7 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 8 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 9 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 10 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options

...

[pmp @ 0x32a8520] Could not find codec parameters for stream 16691 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16692 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16693 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16694 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16695 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16696 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16697 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16698 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16699 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16700 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16701 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16702 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16703 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16704 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16705 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
dos.pmp: could not find codec parameters

Attachments (2)

dos.pmp (55 bytes ) - added by Carl Eugen Hoyos 11 years ago.
patchpmp.diff (529 bytes ) - added by Carl Eugen Hoyos 11 years ago.

Download all attachments as: .zip

Change History (13)

by Carl Eugen Hoyos, 11 years ago

Attachment: dos.pmp added

by Carl Eugen Hoyos, 11 years ago

Attachment: patchpmp.diff added

comment:1 by Carl Eugen Hoyos, 11 years ago

Description: modified (diff)

comment:2 by Elon Musk, 11 years ago

It should just stop when unsupported audio format is found. (It does not set codec id and because of that nothing is lost)

comment:3 by Elon Musk, 11 years ago

If format really allow so many number of streams (16bit), it could be also sensible to demux only some of them (because of brain dead format), but considering there is no such real sample there is no need for such feature. And you could just safely limit max number of streams to 16.

comment:4 by Hendrik, 11 years ago

Even if the file has 16k streams, why does that require gigabytes of memory?

comment:5 by Elon Musk, 11 years ago

By calling avformat_new_stream() 16k times.

comment:6 by Hendrik, 11 years ago

Is the stream structure so big that this adds up?

Last edited 11 years ago by Hendrik (previous) (diff)

comment:7 by Elon Musk, 11 years ago

Not just stream structure it allocates other stuff too.

comment:8 by reimar, 11 years ago

You're looking at the completely wrong place, 16k streams is not a problem, and 16 streams is more than ridulously low, there are more than enough real-world files using more than 30 stream.
I'm quite sure the issue is in a missing EOF check.
However in addition the valgrind numbers you quote give a completely wrong impression, it is not possible to allocate than much data, this is just a huge number of reallocs.
I do not think the allocation ever becomes unreasonably large, however we do have a av_max_alloc function and I have a suspicion this might be a user error, if a user does not want large allocations they should forbid them (though I admit there might be a good argument to be made to default to a lower value on 32 bit systems).
The CPU usage is a different issue, and that is due to the second loop, which in the worst case iterates 2 billion times.

Last edited 11 years ago by reimar (previous) (diff)

in reply to:  8 comment:9 by Carl Eugen Hoyos, 11 years ago

Cc: reimar added

Replying to reimar:

I do not think the allocation ever becomes unreasonably large, however we do have a av_max_alloc function and I have a suspicion this might be a user error

Allocation is >2G here.

comment:10 by DonMoir, 11 years ago

Cc: donmoir@comcast.net added

It uses approximately 400MB opening 16706 streams most likely all for nothing. I have to assume that number is wrong and index_cnt is definitely out of the park at 1094778880.

The remaining allocations come from the second loop or at least more of it. I did not wait 1 billion times to find out :)

Does anyone have a sample pmp that works correctly?

I had to reboot after my first attempt at opening dos.pmp.

Last edited 11 years ago by DonMoir (previous) (diff)

comment:11 by Carl Eugen Hoyos, 11 years ago

Resolution: fixed
Status: newclosed

Fixed by Reimar and Michael.

Note: See TracTickets for help on using tickets.