Opened 4 years ago

Closed 4 years ago

#2298 closed defect (fixed)

High memory allocation with broken pmp file

Reported by: cehoyos Owned by:
Priority: normal Component: avformat
Version: git-master Keywords: pmp videolan
Cc: reimar, donmoir@comcast.net Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by cehoyos)

(vlc ticket 8226)
A sample was uploaded to videolan trac that allocates several Gigabyte at probing, I will attach the sample and a patch that improves the situation, but valgrind still reports 1,021,771,097 allocs with 1,902,250,215,888,401,222 bytes with the patch.

$ ffmpeg -i dos.pmp
ffmpeg version N-50197-g23c9180 Copyright (c) 2000-2013 the FFmpeg developers
  built on Feb 23 2013 13:10:17 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack
  libavutil      52. 17.103 / 52. 17.103
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.100 / 54. 63.100
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 39.101 /  3. 39.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[pmp @ 0x32a8520] Unsupported audio format
[NULL @ 0x32ae8e0] [IMGUTILS @ 0x7fffe55dd7e0] Picture size 1094795585x1094795585 is invalid
[NULL @ 0x32ae8e0] Ignoring invalid width/height values
[pmp @ 0x32a8520] Could not find codec parameters for stream 0 (Video: mpeg4, yuv420p): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 1 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 2 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 3 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 4 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 5 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 6 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 7 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 8 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 9 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 10 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options

...

[pmp @ 0x32a8520] Could not find codec parameters for stream 16691 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16692 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16693 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16694 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16695 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16696 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16697 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16698 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16699 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16700 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16701 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16702 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16703 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16704 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16705 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
dos.pmp: could not find codec parameters

Attachments (2)

dos.pmp (55 bytes) - added by cehoyos 4 years ago.
patchpmp.diff (529 bytes) - added by cehoyos 4 years ago.

Download all attachments as: .zip

Change History (13)

Changed 4 years ago by cehoyos

Changed 4 years ago by cehoyos

comment:1 Changed 4 years ago by cehoyos

  • Description modified (diff)

comment:2 Changed 4 years ago by richardpl

It should just stop when unsupported audio format is found. (It does not set codec id and because of that nothing is lost)

comment:3 Changed 4 years ago by richardpl

If format really allow so many number of streams (16bit), it could be also sensible to demux only some of them (because of brain dead format), but considering there is no such real sample there is no need for such feature. And you could just safely limit max number of streams to 16.

comment:4 Changed 4 years ago by heleppkes

Even if the file has 16k streams, why does that require gigabytes of memory?

comment:5 Changed 4 years ago by richardpl

By calling avformat_new_stream() 16k times.

comment:6 Changed 4 years ago by heleppkes

Is the stream structure so big that this adds up?

Last edited 4 years ago by heleppkes (previous) (diff)

comment:7 Changed 4 years ago by richardpl

Not just stream structure it allocates other stuff too.

comment:8 follow-up: Changed 4 years ago by reimar

You're looking at the completely wrong place, 16k streams is not a problem, and 16 streams is more than ridulously low, there are more than enough real-world files using more than 30 stream.
I'm quite sure the issue is in a missing EOF check.
However in addition the valgrind numbers you quote give a completely wrong impression, it is not possible to allocate than much data, this is just a huge number of reallocs.
I do not think the allocation ever becomes unreasonably large, however we do have a av_max_alloc function and I have a suspicion this might be a user error, if a user does not want large allocations they should forbid them (though I admit there might be a good argument to be made to default to a lower value on 32 bit systems).
The CPU usage is a different issue, and that is due to the second loop, which in the worst case iterates 2 billion times.

Last edited 4 years ago by reimar (previous) (diff)

comment:9 in reply to: ↑ 8 Changed 4 years ago by cehoyos

  • Cc reimar added

Replying to reimar:

I do not think the allocation ever becomes unreasonably large, however we do have a av_max_alloc function and I have a suspicion this might be a user error

Allocation is >2G here.

comment:10 Changed 4 years ago by DonMoir

  • Cc donmoir@comcast.net added

It uses approximately 400MB opening 16706 streams most likely all for nothing. I have to assume that number is wrong and index_cnt is definitely out of the park at 1094778880.

The remaining allocations come from the second loop.

Does anyone have a sample pmp that works correctly?

I had to reboot after my first attempt at opening dos.pmp.

Version 2, edited 4 years ago by DonMoir (previous) (next) (diff)

comment:11 Changed 4 years ago by cehoyos

  • Resolution set to fixed
  • Status changed from new to closed

Fixed by Reimar and Michael.

Note: See TracTickets for help on using tickets.