Opened 5 years ago
Closed 20 months ago
#8020 closed defect (wontfix)
Decoding short broken transport stream has huge memory requirement
| Reported by: | vindicator | Owned by: | |
|---|---|---|---|
| Priority: | normal | Component: | avcodec |
| Version: | git-master | Keywords: | |
| Cc: | nroycea+ffmpeg@gmail.com | Blocked By: | |
| Blocking: | Reproduced by developer: | no | |
| Analyzed by developer: | no |
Description
Summary of the bug:
The short of it is tvheadend captured the raw OTA stream of a tv show from a channel that has a bad signal, producing a corrupt video.
There was one part of the stream, when played, caused the memory to jump significantly (1.5GB+).
Jumping past that brings the memory back to normal.
I just noticed an interesting effect...
I thought I would PAUSE the player somewhere in the time frame where the memory takes off.
The interesting bit is the console output still continually outputted data (while paused) like:
[mpeg2video @ 0x610f440cdec0] concealing 648774 DC, 648774 AC, 648774 MV errors in P frame
[mpeg2video @ 0x610f440cdec0] invalid frame_pred_frame_dct
[mpeg2video @ 0x610f440cdec0] MT_DMV in progressive_sequence
Last message repeated 3 times
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 132
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 133
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 134
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 135
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 136
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 137
[mpeg2video @ 0x610f440cdec0] MT_DMV in progressive_sequence
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 139
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 4 140
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 141
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 142
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 143
[mpeg2video @ 0x610f440cdec0] MT_DMV in progressive_sequence
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 145
[mpeg2video @ 0x610f440cdec0] MT_DMV in progressive_sequence
Last message repeated 1 times
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 148
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 149
[mpeg2video @ 0x610f440cdec0] MT_DMV in progressive_sequence
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 4 151
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 152
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 3 153
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 154
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 155
[mpeg2video @ 0x610f440cdec0] 00 motion_type at 2 156
[mpeg2video @ 0x610f440cdec0] MT_DMV in progressive_sequence
[mpeg2video @ 0x610f440cdec0] Warning MVs not available
[mpeg2video @ 0x610f440cdec0] concealing 648774 DC, 648774 AC, 648774 MV errors in B frame
90622.37 A-V: 3.979 fd= 66 aq= 27KB vq= 749KB sq= 0B f=3/10
Things really started to turn south around here:
[mpeg2video @ 0x65ba8c0cdec0] concealing 45 DC, 45 AC, 45 MV errors in P frame
[mpeg2video @ 0x65ba8c0cdec0] 00 motion_type at 33 15= 0B f=1/3
[mpeg2video @ 0x65ba8c0cdec0] invalid cbp -1 at 7 16
[mpeg2video @ 0x65ba8c0cdec0] 00 motion_type at 24 17
[mpeg2video @ 0x65ba8c0cdec0] Warning MVs not available
[mpeg2video @ 0x65ba8c0cdec0] concealing 360 DC, 360 AC, 360 MV errors in B frame
[mpeg2video @ 0x65ba8c0cdec0] 00 motion_type at 23 12= 0B f=1/3
[mpeg2video @ 0x65ba8c0cdec0] slice mismatch
[mpeg2video @ 0x65ba8c0cdec0] Warning MVs not available
[mpeg2video @ 0x65ba8c0cdec0] concealing 225 DC, 225 AC, 225 MV errors in B frame
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
[mpegts @ 0x65ba8c000b80] Invalid timestamps stream=0, pts=8155253088, dts=8589934595, size=1830
[mpegts @ 0x65ba8c000b80] Invalid timestamps stream=0, pts=8155394229, dts=8435092750, size=21917
[mpegts @ 0x65ba8c000b80] PES packet size mismatch
Last message repeated 1 times
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
Last message repeated 2 times
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
Last message repeated 5 times
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
Last message repeated 1 times
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
[mpegts @ 0x65ba8c000b80] Invalid timestamps stream=0, pts=8155739569, dts=8157827717, size=123
[mpegts @ 0x65ba8c000b80] PES packet size mismatch
Last message repeated 1 times
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
Last message repeated 1 times
[ac3 @ 0x65ba8c007cc0] expacc 125 is out-of-rangeB sq= 0B f=1/3
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] error in bit allocation
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] exponent 25 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] expacc 126 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] exponent -1 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] expacc 125 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] exponent 25 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] new coupling strategy must be present in block 0
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] exponent 25 is out-of-range sq= 0B f=1/3
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] exponent -1 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] Warning: new rematrixing strategy not present in block 0
[ac3 @ 0x65ba8c007cc0] expacc 125 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] Warning: new rematrixing strategy not present in block 0
[ac3 @ 0x65ba8c007cc0] expacc 125 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] expacc 125 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] expacc 126 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] expacc 125 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[mpegts @ 0x65ba8c000b80] PES packet size mismatch
[ac3 @ 0x65ba8c007cc0] expacc 126 is out-of-rangeB sq= 0B f=1/3
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] exponent 25 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[ac3 @ 0x65ba8c007cc0] expacc 127 is out-of-range
[ac3 @ 0x65ba8c007cc0] error decoding the audio block
[mpegts @ 0x65ba8c000b80] PES packet size mismatch sq= 0B f=1/3
Last message repeated 1 times
[mpeg2video @ 0x65ba8c0cdec0] Invalid mb type in P-frame at 10 273
[mpeg2video @ 0x65ba8c0cdec0] ac-tex damaged at 8 28
[mpeg2video @ 0x65ba8c0cdec0] 00 motion_type at 9 1
Needless to say, vlc doesn't like the corrupt stream either:
main warning: playback way too early (-886053): playing silence main warning: clock gap, unexpected stream discontinuity main warning: feeding synchro with a new reference point trying to recover from clock gap main warning: buffer too late (-59652228560 us): dropped main warning: buffer too late (-59652196737 us): dropped main warning: clock gap, unexpected stream discontinuity main warning: feeding synchro with a new reference point trying to recover from clock gap main warning: buffer too late (-71582265729 us): dropped ... avcodec warning: More than 11 late frames, dropping frame avcodec warning: More than 11 late frames, dropping frame ... avcodec error: Invalid frame size 13008x12768 vsz 13008x12768 avcodec error: Invalid frame size 13008x12768 vsz 13008x12768 ...
At which point, the stream just dies and jumps to the end, although I can always just skip over the bad parts.
Did I mention the corrupt video was captured OTA which had a bad signal? :)
Some valgrind stuff:
... ==26883== 1,003,513,704 bytes in 24 blocks are still reachable in loss record 1,583 of 1,583 ==26883== at 0x483AEC3: memalign (vg_replace_malloc.c:898) ==26883== by 0x483AFF0: posix_memalign (vg_replace_malloc.c:1062) ==26883== by 0x64352CB: av_malloc (in /usr/lib/libavutil.so.56.22.100) ==26883== by 0x641CD69: av_buffer_alloc (in /usr/lib/libavutil.so.56.22.100) ==26883== by 0x641CDED: av_buffer_allocz (in /usr/lib/libavutil.so.56.22.100) ==26883== by 0x641D44C: av_buffer_pool_get (in /usr/lib/libavutil.so.56.22.100) ==26883== by 0x5065E0D: avcodec_default_get_buffer2 (in /usr/lib/libavcodec.so.58.35.100) ==26883== by 0x5066682: ??? (in /usr/lib/libavcodec.so.58.35.100) ==26883== by 0x53F3708: ??? (in /usr/lib/libavcodec.so.58.35.100) ==26883== by 0x53749C5: ??? (in /usr/lib/libavcodec.so.58.35.100) ==26883== by 0x537B392: ??? (in /usr/lib/libavcodec.so.58.35.100) ==26883== by 0x5340B8F: ??? (in /usr/lib/libavcodec.so.58.35.100) ==26883== ==26883== LEAK SUMMARY: ==26883== definitely lost: 544 bytes in 8 blocks ==26883== indirectly lost: 0 bytes in 0 blocks ==26883== possibly lost: 23,767,813 bytes in 5,405 blocks ==26883== still reachable: 1,573,732,063 bytes in 4,375 blocks ==26883== suppressed: 0 bytes in 0 blocks ==26883== ==26883== For counts of detected and suppressed errors, rerun with: -v ==26883== ERROR SUMMARY: 673 errors from 673 contexts (suppressed: 0 from 0)
I'll try to remember to hold onto the stream for a little while.
I hope the information I've provided helps clue someone to the location of the (possible) looked-over leak.
Based on valgrind, I'm going to guess and attribute the component to avcodec.
Feel free to adjust what you need to (component) and let me know if you have any other questions or things you want me to try.
How to reproduce:
% ffplay -i {OTA_TV_show}.ts
ffmpeg version n4.1.3 (don't know why a 'n' is prepended)
built on(by) Arch Linux gcc 8.2.1
Change History (7)
comment:1 by , 5 years ago
comment:3 by , 5 years ago
I trimmed the 1 hour (1.1GB) stream down to the part that was affected, making it 1 minute (20MB) using:
$ ffmpeg -i <path>/<input>.ts -c:v copy -c:a copy -ss 17:15.0 -t 1:00.0 <output>.ts
For some reason, the audio doesn't play with ffplay when trimmed, but if you wait like maybe 10 seconds, your RAM usage will spike.
Mine just went from 2.9GB to 4.6GB before I forcefully closed it.
Also, for some reason I didn't get any email notification of mkver's reply.
Could it be due to special character's in my address?
I noticed the upload page is for videolan, but my issue was with ffplay.
The upload has completed now.
EDIT0: I see I have to CC my address.
comment:4 by , 5 years ago
| Cc: | added |
|---|
comment:5 by , 4 years ago
| Keywords: | leak removed |
|---|---|
| Summary: | avcodec_default_get_buffer2 Leak → Decoding short broken transport stream has huge memory requirement |
| Version: | 4.1 → git-master |
The reason for the memory spike is that the stream dimensions change to 13008x12768, not sure if this can be improved, I cannot reproduce a leak.
comment:6 by , 23 months ago
| Status: | new → open |
|---|
The sample is here. https://streams.videolan.org/ffmpeg/incoming/8020/
comment:7 by , 20 months ago
| Resolution: | → wontfix |
|---|---|
| Status: | open → closed |
You can use -max_pixels to limit the resolution (and memory allocation) when reading an input file, there will be no other solution.
OH, also an unrelated thing...
When I created my account, I had to go through a couple of captchas, but one was related to spamcannibal, I guess because of my IP address.
The thing is, my connection is through a mobile provider and is routed through a double NAT, so that public facing IP isn't directly associated with me.
You could liken it to a VPN/proxy.
In addition, it looks like spamcannibal is obsolete/inactive (https://mxtoolbox.com/problem/blacklist/spamcannibal).
It's not a big deal. The captchas were no problem, just a minor one-time inconvenience during registration. It's just a heads-up.