Opened 13 years ago
Closed 13 years ago
#596 closed defect (invalid)
ffmpeg killed while encoding
| Reported by: | Benoit Friry | Owned by: | |
|---|---|---|---|
| Priority: | normal | Component: | undetermined |
| Version: | 0.8.5 | Keywords: | |
| Cc: | Blocked By: | ||
| Blocking: | Reproduced by developer: | no | |
| Analyzed by developer: | no |
Description
Hello,
I try to convert a big mjpeg video into h264. ffmpeg crashes while encoding.
The command is:
$ ./ffmpeg -i ~/tmp/test.avi -an -vcodec libx264 -preset slow -profile high -crf 25 -f mp4 ~/tmp/test.mp4
$ ./ffmpeg -v 9 -loglevel 99 -i ~/tmp/test.avi
ffmpeg version 0.8.5, Copyright (c) 2000-2011 the FFmpeg developers
built on Oct 26 2011 22:09:58 with gcc 4.6.1
configuration: --prefix=/opt/bf_ffmpeg --enable-gpl --enable-libfaac --enable-libx264 --enable-nonfree --enable-postproc
libavutil 51. 9. 1 / 51. 9. 1
libavcodec 53. 7. 0 / 53. 7. 0
libavformat 53. 4. 0 / 53. 4. 0
libavdevice 53. 1. 1 / 53. 1. 1
libavfilter 2. 23. 0 / 2. 23. 0
libswscale 2. 0. 0 / 2. 0. 0
libpostproc 51. 2. 0 / 51. 2. 0
[avi @ 0x257d400] Format avi probed with size=2048 and score=100
[AVI demuxer @ 0x257da00] use odml:1
st:1 removing common factor 4 from timebase
[mjpeg @ 0x257f480] buffer too small, expanding to 108576 bytes
[mjpeg @ 0x257f480] marker=d8 avail_size_in_buf=108576
[mjpeg @ 0x257f480] marker parser used 0 bytes (0 bits)
[mjpeg @ 0x257f480] marker=fe avail_size_in_buf=108574
[mjpeg @ 0x257f480] marker parser used 14 bytes (112 bits)
[mjpeg @ 0x257f480] marker=db avail_size_in_buf=108558
[mjpeg @ 0x257f480] index=0
[mjpeg @ 0x257f480] qscale[0]: 2
[mjpeg @ 0x257f480] marker parser used 67 bytes (536 bits)
[mjpeg @ 0x257f480] marker=c4 avail_size_in_buf=108489
[mjpeg @ 0x257f480] class=0 index=0 nb_codes=12
[mjpeg @ 0x257f480] class=0 index=1 nb_codes=12
[mjpeg @ 0x257f480] class=1 index=0 nb_codes=251
[mjpeg @ 0x257f480] class=1 index=1 nb_codes=251
[mjpeg @ 0x257f480] marker parser used 418 bytes (3344 bits)
[mjpeg @ 0x257f480] marker=c0 avail_size_in_buf=108069
[mjpeg @ 0x257f480] sof0: picture: 720x576
[mjpeg @ 0x257f480] component 0 2:2 id: 0 quant:0
[mjpeg @ 0x257f480] component 1 1:1 id: 1 quant:0
[mjpeg @ 0x257f480] component 2 1:1 id: 2 quant:0
[mjpeg @ 0x257f480] pix fmt id 22111100
[mjpeg @ 0x257f480] marker parser used 17 bytes (136 bits)
[mjpeg @ 0x257f480] escaping removed 1257 bytes
[mjpeg @ 0x257f480] marker=da avail_size_in_buf=108050
[mjpeg @ 0x257f480] component: 0
[mjpeg @ 0x257f480] component: 1
[mjpeg @ 0x257f480] component: 2
[mjpeg @ 0x257f480] marker parser used 106792 bytes (854331 bits)
[mjpeg @ 0x257f480] marker=d9 avail_size_in_buf=0
[mjpeg @ 0x257f480] mjpeg decode frame unused 0 bytes
[avi @ 0x257d400] All info found
Input #0, avi, from '/home/benoit/tmp/test.avi':
Duration: 00:47:29.56, start: 0.000000, bitrate: 21670 kb/s
Stream #0.0, 1, 1/25: Video: mjpeg, yuvj420p, 720x576, 1/25, 25 tbr, 25 tbn, 25 tbc
Stream #0.1, 1, 1/44100: Audio: pcm_s16le, 44100 Hz, 2 channels, s16, 1411 kb/s
At least one output file must be specified
I didn't succeed to produce a bt.
$ gdb ./ffmpeg_g
GNU gdb (GDB) 7.3-debian
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/benoit/tmp/bf_ffmpeg/ffmpeg/ffmpeg_g...done.
(gdb) r -i ~/tmp/test.avi -an -vcodec libx264 -preset slow -profile high -crf 25 -f mp4 ~/tmp/test.mp4
Starting program: /home/benoit/tmp/bf_ffmpeg/ffmpeg/ffmpeg_g -i ~/tmp/test.avi -an -vcodec libx264 -preset slow -profile high -crf 25 -f mp4 ~/tmp/test.mp4
[Thread debugging using libthread_db enabled]
ffmpeg version 0.8.5, Copyright (c) 2000-2011 the FFmpeg developers
built on Oct 26 2011 22:09:58 with gcc 4.6.1
configuration: --prefix=/opt/bf_ffmpeg --enable-gpl --enable-libfaac --enable-libx264 --enable-nonfree --enable-postproc
libavutil 51. 9. 1 / 51. 9. 1
libavcodec 53. 7. 0 / 53. 7. 0
libavformat 53. 4. 0 / 53. 4. 0
libavdevice 53. 1. 1 / 53. 1. 1
libavfilter 2. 23. 0 / 2. 23. 0
libswscale 2. 0. 0 / 2. 0. 0
libpostproc 51. 2. 0 / 51. 2. 0
Input #0, avi, from '/home/benoit/tmp/test.avi':
Duration: 00:47:29.56, start: 0.000000, bitrate: 21670 kb/s
Stream #0.0: Video: mjpeg, yuvj420p, 720x576, 25 tbr, 25 tbn, 25 tbc
Stream #0.1: Audio: pcm_s16le, 44100 Hz, 2 channels, s16, 1411 kb/s
[buffer @ 0x13ee3c0] w:720 h:576 pixfmt:yuvj420p tb:1/1000000 sar:0/1 sws_param:
[libx264 @ 0x13e5e60] using cpu capabilities: MMX2 SSE2Slow SlowCTZ
[libx264 @ 0x13e5e60] profile High, level 3.0
[libx264 @ 0x13e5e60] 264 - core 119 r2106 07efeb4 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=1 ref=5 deblock=1:0:0 analyse=0x3:0x113 me=umh subme=8 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=2 b_bias=0 direct=3 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=50 rc=crf mbtree=1 crf=25.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/home/benoit/tmp/test.mp4':
Metadata:
encoder : Lavf53.4.0
Stream #0.0: Video: libx264, yuvj420p, 720x576, q=2-31, 200 kb/s, 25 tbn, 25 tbc
Stream mapping:
Press [q] to stop, ? for help
frame= 5934 fps= 1 q=30.0 size= 62734kB time=00:03:55.24 bitrate=2184.7kbits/s dup=537 drop=0
Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) bt
No stack.
(gdb) disass $pc-32,$pc+32
No registers.
(gdb) info all-registers
The program has no registers now.
(gdb) quit
Attachments (1)
Change History (14)
comment:1 by , 13 years ago
comment:2 by , 13 years ago
I did not reproduce the error with -vcodec mpeg4.
Same error with git head.
It do not reproduce the error with all input file. The one I use to do tests is 7.2GB! Not easy to upload.
I don't understand why bt do not work.
comment:3 by , 13 years ago
Since I suspect it goes OOM, can you try with valgrind (assuming it reports something useful in that case)?
follow-up: 5 comment:4 by , 13 years ago
I've never used valgrind. http://valgrind.org/docs/manual/quick-start.html#quick-start.intro states the program will run 20 to 30 times slower. As my test is 1 to 2 hours long (without valgrind), I'd prefer check twice what you need... ;)
valgrind --leak-check=yes ffmpeg_g -i ~/tmp/test.avi -an -vcodec libx264 -preset slow -profile high -crf 25 -f mp4 ~/tmp/test.mp4
Or without "--leak-check=yes"?
comment:5 by , 13 years ago
Replying to benoit_friry:
As my test is 1 to 2 hours long
Then I suggest you either watch memory consumption while running ffmpeg or significantly (temporarily) increase the size of your swap partition.
comment:6 by , 13 years ago
Run command:
$ ./ffmpeg_g -i ~/tmp/test.avi -an -vcodec libx264 -preset slow -profile high -crf 25 -f mp4 ~/tmp/test.h264.mp4
which ended by:
[libx264 @ 0x34b5860] frame=11886 QP=29.57 NAL=2 Slice:B Poc:96 I:29 P:1243 SKIP:272 size=8734 bytes [mjpeg @ 0x34a3bc0] marker=d8 avail_size_in_buf=105714 [mjpeg @ 0x34a3bc0] marker parser used 0 bytes (0 bits) [mjpeg @ 0x34a3bc0] marker=fe avail_size_in_buf=105712 [mjpeg @ 0x34a3bc0] mjpeg comment: 'Lavc52.72.2' [mjpeg @ 0x34a3bc0] marker parser used 14 bytes (112 bits) [mjpeg @ 0x34a3bc0] marker=db avail_size_in_buf=105696 [mjpeg @ 0x34a3bc0] index=0 [mjpeg @ 0x34a3bc0] qscale[0]: 2 [mjpeg @ 0x34a3bc0] marker parser used 67 bytes (536 bits) [mjpeg @ 0x34a3bc0] marker=c4 avail_size_in_buf=105627 [mjpeg @ 0x34a3bc0] class=0 index=0 nb_codes=12 [mjpeg @ 0x34a3bc0] class=0 index=1 nb_codes=12 [mjpeg @ 0x34a3bc0] class=1 index=0 nb_codes=251 [mjpeg @ 0x34a3bc0] class=1 index=1 nb_codes=251 [mjpeg @ 0x34a3bc0] marker parser used 418 bytes (3344 bits) [mjpeg @ 0x34a3bc0] marker=c0 avail_size_in_buf=105207 [mjpeg @ 0x34a3bc0] sof0: picture: 720x576 [mjpeg @ 0x34a3bc0] component 0 2:2 id: 0 quant:0 [mjpeg @ 0x34a3bc0] component 1 1:1 id: 1 quant:0 [mjpeg @ 0x34a3bc0] component 2 1:1 id: 2 quant:0 [mjpeg @ 0x34a3bc0] pix fmt id 22111100 [mjpeg @ 0x34a3bc0] marker parser used 17 bytes (136 bits) [mjpeg @ 0x34a3bc0] escaping removed 2132 bytes [mjpeg @ 0x34a3bc0] marker=da avail_size_in_buf=105188 [mjpeg @ 0x34a3bc0] component: 0 [mjpeg @ 0x34a3bc0] component: 1 [mjpeg @ 0x34a3bc0] component: 2 [mjpeg @ 0x34a3bc0] sequential DCT p:0 >>:0 ilv:63 bits:8 skip:0 comp:3 [mjpeg @ 0x34a3bc0] marker parser used 103055 bytes (824439 bits) [mjpeg @ 0x34a3bc0] marker=d9 avail_size_in_buf=0 [mjpeg @ 0x34a3bc0] mjpeg decode frame unused 0 bytes zsh: killed ./ffmpeg_g -i ~/tmp/test.avi -an -vcodec libx264 -preset slow -profile high
Meanwhile, i watched for memory consumption:
$ while true; do ps -o cp,cputime,%mem,rss,sz,vsz -p 6983 >> ~/tmp/ffmpeg_g_ps.log; sleep 1 ; done
The log is
CP TIME %MEM RSS SZ VSZ 3 00:00:00 4.3 89904 31373 125492 7 00:00:01 4.3 90172 31373 125492 11 00:00:02 4.8 99080 33216 132864 15 00:00:03 5.0 102988 34450 137800 18 00:00:04 5.1 105512 34450 137800 23 00:00:05 5.2 107976 35052 140208 26 00:00:05 5.2 108064 35052 140208 29 00:00:06 5.2 108064 35052 140208 33 00:00:07 5.2 108064 35052 140208 37 00:00:08 5.2 108084 35052 140208 [...] 850 00:56:27 5.3 110128 35151 140604 850 00:56:28 5.3 110128 35151 140604 850 00:56:29 5.3 110128 35151 140604 850 00:56:30 5.3 110128 35151 140604 850 00:56:31 5.3 110128 35151 140604 850 00:56:32 5.3 110128 35151 140604 851 00:56:33 5.3 110128 35151 140604 851 00:56:33 5.3 110128 35151 140604 851 00:56:34 5.3 110128 35151 140604 851 00:56:35 5.3 110128 35151 140604 851 00:56:36 5.3 110132 35151 140604 851 00:56:37 5.3 110132 35151 140604 851 00:56:38 5.3 110132 35151 140604 851 00:56:39 5.3 110132 35151 140604 851 00:56:40 5.3 110132 35151 140604 850 00:56:41 5.3 110132 35151 140604 851 00:56:42 5.3 110132 35151 140604 851 00:56:43 5.3 110132 35151 140604
After 00:56:43 process was died.
Full log attached, but nothing particular in it.
Should I try to run valgrind?
by , 13 years ago
| Attachment: | ffmpeg_g_ps.log added |
|---|
Output of memory watch during execution of ffmpeg
comment:7 by , 13 years ago
Valgrind run:
$ nice -20 valgrind ./ffmpeg -i ~/tmp/test.avi -an -vcodec libx264 -preset slow -profile high -crf 25 -f mp4 ~/tmp/test.mp4
==26549== Memcheck, a memory error detector
==26549== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==26549== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==26549== Command: ./ffmpeg -i /home/benoit/tmp/test.avi -an -vcodec libx264 -preset slow -profile high -crf 25 -f mp4 /home/benoit/tmp/test.mp4
==26549==
ffmpeg version git-2011-10-27-3e7db0a, Copyright (c) 2000-2011 the FFmpeg developers
built on Oct 27 2011 19:06:23 with gcc 4.6.1
configuration: --prefix=/opt/bf_ffmpeg --enable-gpl --enable-libfaac --enable-libx264 --enable-nonfree --enable-postproc
libavutil 51. 22. 0 / 51. 22. 0
libavcodec 53. 23. 0 / 53. 23. 0
libavformat 53. 17. 0 / 53. 17. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 45. 0 / 2. 45. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[avi @ 0x5866820] parser not found for codec pcm_s16le, packets or times may be invalid.
Input #0, avi, from '/home/benoit/tmp/test.avi':
Duration: 00:47:29.56, start: 0.000000, bitrate: 21670 kb/s
Stream #0:0: Video: mjpeg (MJPG / 0x47504A4D), yuvj420p, 720x576, 25 tbr, 25 tbn, 25 tbc
Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, 2 channels, s16, 1411 kb/s
[buffer @ 0x6460940] w:720 h:576 pixfmt:yuvj420p tb:1/1000000 sar:0/1 sws_param:
[libx264 @ 0x645b880] using cpu capabilities: MMX2 SSE2Slow SlowCTZ
[libx264 @ 0x645b880] profile High, level 3.0
[libx264 @ 0x645b880] 264 - core 119 r2106 07efeb4 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=1 ref=5 deblock=1:0:0 analyse=0x3:0x113 me=umh subme=8 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=2 b_bias=0 direct=3 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=50 rc=crf mbtree=1 crf=25.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/home/benoit/tmp/test.mp4':
Metadata:
encoder : Lavf53.17.0
Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuvj420p, 720x576, q=-1--1, 25 tbn, 25 tbc
Stream mapping:
Stream #0.0 -> #0.0 (mjpeg -> libx264)
Press [q] to stop, [?] for help
zsh: killed nice -20 valgrind ./ffmpeg -i ~/tmp/test.avi -an -vcodec libx264 -preset slow0
comment:8 by , 13 years ago
Definitely OOM killer. If I am right that you run a 32 bit FFmpeg, please use a more reasonable swap size. With a 4 GB swap a 32 bit version will always allow to get a proper backtrace instead of invoking the OOM killer.
In case of a 64 bit version you could set the memory ulimit to achieve a similar effect, however x264 using up several GB of RAM is not necessarily a bug, and I do not know what usual memory usage numbers are for which profile.
comment:9 by , 13 years ago
$ file ffmpeg_g ffmpeg_g: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0x9f7abf9f378f342a2bc75c4cd8e02f35cff6a8c8, not stripped $ ulimit unlimited
I have 2GB RAM and 1GB swap.
Can you please explain what makes you think about a OOM? The program being killed without any bt is a good sign, but I don't see any memory saturation. VSZ does not exceed 150 MB.
comment:10 by , 13 years ago
Well, what else goes around and does a SIGKILL on processes? Unless you have a very special setup, OOM is the only thing that would not only kill ffmpeg but also a valgrind process running ffmpeg with SIGKILL (and thus without a chance to print anything at all).
The only alternative to it being an OOM killer I can see is something else on your system just kills FFmpeg, in which case you'd be wrong on this bug tracker...
comment:11 by , 13 years ago
And if you're real sure that VSZ does not go over 150MB, "ulimit -v 153600" should not change anything (though 150 MB is quite restrained for a slow H.264 encode).
comment:12 by , 13 years ago
There was no limit on memory... But on cpu time!
$ ulimit -a -t: cpu time (seconds) 3600 -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 8192 -c: core file size (blocks) 0 -m: resident set size (kbytes) 1024000 -u: processes 1024 -n: file descriptors 1024 -l: locked-in-memory size (kb) 64 -v: address space (kb) unlimited -x: file locks unlimited -i: pending signals 16083 -q: bytes in POSIX msg queues 819200 -e: max nice 0 -r: max rt priority 0 -N 15: unlimited
Setting the cpu time to "unlimited" makes it possible to encode correctly the full video.
The flaw was between the keyboard and the seat.
Unfortunately, the kill is not verbose when set hard.
$ ulimit -S -t 1 $ bash $ while true; do foo=$(( foo + 1 )); done zsh: cpu limit exceeded bash $ ulimit -H -t 1 $ bash $ while true; do foo=$(( foo + 1 )); done zsh: killed bash
Sorry for the noise, and many thanks for your help.
This ticket can be closed. (I cannot close it myself.)
comment:13 by , 13 years ago
| Resolution: | → invalid |
|---|---|
| Status: | new → closed |
I didn't know you could set a CPU time limit;-)
Thank you for the update!
Is this also reproducible with -vcodec mpeg4 (instead of libx264)?
Do you see the same problem with current git head?
Is a specific sample needed? If yes, please upload to http://www.datafilehost.com/ and post the link.