Opened 10 years ago

Closed 10 years ago

#1980 closed defect (needs_more_info)

ffmpeg process hangs with 100% CPU when getting a live rtsp stream (on redhat linux)

Reported by: muthu Owned by:
Priority: normal Component: undetermined
Version: 1.0 Keywords: rtsp
Cc: person79@rediffmail.com Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
ffmpeg process hangs with 100% CPU when getting a live rtsp stream (on redhat linux). This happens when 25 or more such processes are concurrently invoked.
How to reproduce:
On redhat linux (2.6+) launch 25 or more parallel live rtsp stream reads.

Command line is as follows,
/home/lsadm/StressTest/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/5EErJWzqY7hSRkrUL030_A_QVGA+email.share@yahoo.com-5EErJWzqY7hSRkrUL030_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.mov

The generated console (stats) data is,
ffmpeg version 1.0 Copyright (c) 2000-2012 the FFmpeg developers

built on Nov 12 2012 13:10:29 with gcc 4.1.2 (GCC) 20080704 (Red Hat 4.1.2-44)
configuration: --disable-yasm --enable-nonfree --prefix=/home/lsadm/StressTest/FFmpeg/
libavutil 51. 73.101 / 51. 73.101
libavcodec 54. 59.100 / 54. 59.100
libavformat 54. 29.104 / 54. 29.104
libavdevice 54. 2.101 / 54. 2.101
libavfilter 3. 17.100 / 3. 17.100
libswscale 2. 1.101 / 2. 1.101
libswresample 0. 15.100 / 0. 15.100

[h264 @ 0x17d6f540] Missing reference picture
[h264 @ 0x17d6f540] decode_slice_header error
[h264 @ 0x17d6f540] concealing 300 DC, 300 AC, 300 MV errors in P frame
[h264 @ 0x17d6f540] Missing reference picture
[h264 @ 0x17d6f540] RTP: missed 15429 packets
[h264 @ 0x17d6f540] RTP: missed 15430 packets
[h264 @ 0x17d6f540] RTP: missed 15431 packets
[h264 @ 0x17d6f540] RTP: missed 15432 packets
[h264 @ 0x17d6f540] RTP: missed 15433 packets

Below is a snapshot of the CPU captured with top command,
Tasks: 404 total, 2 running, 402 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.3%us, 0.1%sy, 0.0%ni, 93.4%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6096740k total, 5990044k used, 106696k free, 10404k buffers
Swap: 1020116k total, 336084k used, 684032k free, 3099316k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

22478 lsadm 25 0 27364 4732 1920 R 99.9 0.1 18:35.66 ffmpeg
16383 lsadm 15 0 2695m 2.3g 2624 S 0.7 40.4 32:52.16 perl
26991 lsadm 15 0 29340 6272 2776 S 0.3 0.1 0:00.73 ffmpeg
26994 lsadm 15 0 28192 4904 2416 S 0.3 0.1 0:00.78 ffmpeg
27319 lsadm 15 0 10988 1280 772 R 0.3 0.0 0:00.07 top

1 root 15 0 10344 80 48 S 0.0 0.0 3:56.92 init

[lsadm@grinder025 Scripts]$ ps -aef | grep 22478
lsadm 22478 22476 99 17:30 pts/1 00:28:54 /home/lsadm/StressTest/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/5EErJWzqY7hSRkrUL030_A_QVGA+email.share@yahoo.com-5EErJWzqY7hSRkrUL030_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.mov
lsadm 29402 2890 0 17:59 pts/3 00:00:00 grep 22478
[lsadm@grinder025 Scripts]$ ps -aef | grep 22476
lsadm 22476 16383 0 17:30 pts/1 00:00:00 sh -c /home/lsadm/StressTest/FFmpeg/bin/ffmpeg -i 'rtsp://xstream-bcast-grinder003:50554/5EErJWzqY7hSRkrUL030_A_QVGA+email.share@yahoo.com-5EErJWzqY7hSRkrUL030_A_QVGA.sdp' -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.mov 2> /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.log

Change History (6)

comment:1 by Carl Eugen Hoyos, 10 years ago

Component: FFmpegundetermined
Keywords: 100% cpu removed

Please test current git head and provide a backtrace.

in reply to:  1 comment:2 by muthu, 10 years ago

Replying to cehoyos:

Please test current git head and provide a backtrace.

Thanks for your quick response.

I downloaded the current version from (http://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2) and still see the issue after some time (15 minutes or so).
Below are the details,

Command line is,
/home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i 'rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp' -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov 2> /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.log

The generated console (stats) data is,
[lsadm@grinder025 Scripts]$ cat /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.log
ffmpeg version 1.0.git Copyright (c) 2000-2012 the FFmpeg developers

built on Nov 30 2012 12:44:30 with gcc 4.1.2 (GCC) 20080704 (Red Hat 4.1.2-44)
configuration: --prefix=/home/lsadm/test-muthu/newerffmpeg/FFmpeg/ --enable-gpl --enable-version3 --enable-nonfree --disable-yasm
libavutil 52. 9.102 / 52. 9.102
libavcodec 54. 77.100 / 54. 77.100
libavformat 54. 39.100 / 54. 39.100
libavdevice 54. 3.100 / 54. 3.100
libavfilter 3. 23.104 / 3. 23.104
libswscale 2. 1.103 / 2. 1.103
libswresample 0. 17.101 / 0. 17.101
libpostproc 52. 2.100 / 52. 2.100

[h264 @ 0x1b50e520] Missing reference picture, default is 0
[h264 @ 0x1b50e520] decode_slice_header error
[h264 @ 0x1b50e520] concealing 300 DC, 300 AC, 300 MV errors in P frame
[h264 @ 0x1b50e520] Missing reference picture, default is 14
[h264 @ 0x1b50e520] RTP: missed 7594 packets
[h264 @ 0x1b50e520] RTP: missed 7595 packets
[h264 @ 0x1b50e520] RTP: missed 7596 packets
[h264 @ 0x1b50e520] RTP: missed 7597 packets
[h264 @ 0x1b50e520] RTP: missed 7598 packets
[h264 @ 0x1b50e520] RTP: missed 7599 packets
[h264 @ 0x1b50e520] RTP: missed 7600 packets
[h264 @ 0x1b50e520] RTP: missed 7601 packets
[h264 @ 0x1b50e520] RTP: missed 7602 packets
[lsadm@grinder025 Scripts]$

Below is a snapshot of the CPU captured with top command,
top - 13:37:40 up 52 days, 2:06, 10 users, load average: 3.51, 5.50, 4.97
Tasks: 473 total, 3 running, 470 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 0.3%sy, 0.0%ni, 96.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6096740k total, 6073196k used, 23544k free, 8528k buffers
Swap: 1020116k total, 361720k used, 658396k free, 2466924k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

18206 lsadm 25 0 27792 4808 1972 R 99.9 0.1 18:29.11 ffmpeg
22322 lsadm 25 0 27968 4992 1976 R 99.9 0.1 6:42.93 ffmpeg
13406 lsadm 15 0 3933m 2.9g 2628 S 3.9 49.3 29:05.40 perl
23117 lsadm 15 0 28564 4976 2484 S 2.0 0.1 0:00.61 ffmpeg
23311 lsadm 15 0 28556 4964 2484 S 2.0 0.1 0:00.49 ffmpeg
23768 lsadm 15 0 11116 1232 684 R 2.0 0.0 0:00.01 top

1 root 15 0 10344 80 48 S 0.0 0.0 3:56.94 init

[lsadm@grinder025 Scripts]$ ps -aef | grep 18206
lsadm 18206 18205 99 13:19 pts/1 00:19:31 /home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov
lsadm 23831 13531 0 13:38 pts/7 00:00:00 grep 18206

[lsadm@grinder025 Scripts]$ ps -aef | grep 18205
lsadm 18205 13406 0 13:19 pts/1 00:00:00 sh -c /home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i 'rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp' -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov 2> /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.log
lsadm 18206 18205 99 13:19 pts/1 00:19:29 /home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov
lsadm 23799 13531 0 13:38 pts/7 00:00:00 grep 18205
[lsadm@grinder025 Scripts]$

comment:3 by Carl Eugen Hoyos, 10 years ago

If you report that FFmpeg hangs, I believe you, no need to post ps output.

Since no developer is able to reproduce the problem (because of your rtsp address), a backtrace is necessary.
Please run "$ gdb ffmpeg_g", then "r -i rtsp... " and when ffmpeg hangs, you interrupt it with Ctrl-C and type "bt" to get a backtrace, please post it here.

in reply to:  3 ; comment:4 by muthu, 10 years ago

Replying to cehoyos:

If you report that FFmpeg hangs, I believe you, no need to post ps output.

Since no developer is able to reproduce the problem (because of your rtsp address), a backtrace is necessary.
Please run "$ gdb ffmpeg_g", then "r -i rtsp... " and when ffmpeg hangs, you interrupt it with Ctrl-C and type "bt" to get a backtrace, please post it here.

Thank you.

I actually run/launch these from a perl script (launches 25 instances every ~6 minutes) and they fail at random (after 15-20 minutes or so). I don't think launching each invocation in gdb is feasible. I realize however that i can attach to an ffmpeg process & take a backtrace (cmd-line attach). I hope this is something that will work. Please correct me or point me in the right direction if this understanding is incorrect. I must admit i have been working in Java & am not too familiar with using gdb. So please forgive my amateurism.

Also please let me know how to build ffmpeg with debug enabled.

Thanks
Muthu

Last edited 10 years ago by muthu (previous) (diff)

in reply to:  4 comment:5 by Carl Eugen Hoyos, 10 years ago

Replying to muthuk:

Replying to cehoyos:

Since no developer is able to reproduce the problem (because of your rtsp address), a backtrace is necessary.
Please run "$ gdb ffmpeg_g", then "r -i rtsp... " and when ffmpeg hangs, you interrupt it with Ctrl-C and type "bt" to get a backtrace, please post it here.

I actually run/launch these from a perl script (launches 25 instances every ~6 minutes) and they fail at random (after 15-20 minutes or so). I don't think launching each invocation in gdb is feasible. I realize however that i can attach to an ffmpeg process & take a backtrace (cmd-line attach).

I have never done it myself but this sounds like a good idea.

Also please let me know how to build ffmpeg with debug enabled.

$ ./configure && make ;-)
Your configure line produces two executables, ffmpeg_g contains debug symbols, I suspect --disable-stripping leaves the symbols also in ffmpeg.
(I only realize now you are using --disable-yasm. This is actually a good idea for debugging, but don't forget to remove it in the future, it hurts performance.)

comment:6 by Carl Eugen Hoyos, 10 years ago

Resolution: needs_more_info
Status: newclosed

As said, this sounds like an important issue, please reopen if you can provide a rtsp link that allows to reproduce the problem or a backtrace etc. as explained on http://ffmpeg.org/bugreports.html

Note: See TracTickets for help on using tickets.