Opened 7 years ago
Closed 7 years ago
#6204 closed defect (fixed)
AAC-only hls stream slowly exceeds file descriptors
Reported by: | Nicola Concetti | Owned by: | Steven Liu |
---|---|---|---|
Priority: | important | Component: | avformat |
Version: | git-master | Keywords: | hls regression |
Cc: | Blocked By: | ||
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description
Summary of the bug:
The hls muxer does not release the segment's file descriptors if there is only an audio-only stream. This happens regardless of the delete_segments flag or -re flag being set or not. Once ffmpeg runs out of avalable FD's, it terminates gracefully with the following output:
Excerpt (full output included in file):
[hls muxer @ 0x3a3d5a0] deleting old segment segment_128_0000000501.ts [file @ 0x59c0100] Setting default whitelist 'file,crypto' [file @ 0x5a85ba0] Setting default whitelist 'file,crypto' [hls muxer @ 0x3a3d5a0] EXT-X-MEDIA-SEQUENCE:506 [AVIOContext @ 0x5b21a60] Statistics: 0 seeks, 1 writeouts [hls muxer @ 0x3aa6500] deleting old segment segment_96_0000000501.ts [file @ 0x5b7ce00] Setting default whitelist 'file,crypto' [file @ 0x5b7cf00] Setting default whitelist 'file,crypto' av_interleaved_write_frame(): Too many open files No more output streams to write to, finishing.
How to reproduce:
% ffmpeg -v 9 -loglevel 99 -re -i http://127.0.0.1:8000/test.ogg \ -c:a aac \ -vn \ -b:a 128000 \ -f hls \ -hls_time 6 \ -hls_list_size 3 \ -hls_flags delete_segments \ -hls_segment_filename segment_128_%10d.ts \ playlist_128.m3u8 \ -c:a aac \ -vn \ -b:a 96000 \ -f hls \ -hls_time 6 \ -hls_list_size 3 \ -hls_flags delete_segments \ -hls_segment_filename segment_96_%10d.ts \ playlist_96.m3u8 2>>ffmpeg-all.log ffmpeg version N-83663-g7e9ba78 built on Ubuntu 16.04.2 LTS (4.4.0-62-generic) built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609 configured with --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree
Please note that it takes some time to happen and depends on the limits on the system.
the machine's "max open files" was set to 1024.
The provided live source can be any kind of audio source. However, the tested source was an OGG source provided by an Icecast2 instance.
while the process is running you can see the open files (deleted segments included) stacking up until the termination of the ffmpeg process:
lsof -p <PID> | grep segment | grep deleted | wc -l 656
Attachments (1)
Change History (11)
by , 7 years ago
Attachment: | ffmpeg-all.log.gz added |
---|
follow-up: 2 comment:1 by , 7 years ago
Component: | ffmpeg → undetermined |
---|---|
Keywords: | aac segments removed |
Priority: | important → normal |
Is the issue reproducible if the input is -f lavfi -i sine
?
Does it only happen using the aac encoder?
Are two output streams necessary to reproduce the issue or does it also happen with one output stream?
Do you think this is a regression?
comment:2 by , 7 years ago
Replying to cehoyos:
Is the issue reproducible if the input is
-f lavfi -i sine
?
Yes:
[hls muxer @ 0x3f36660] deleting old segment segment_128_0000001011.ts [file @ 0x5e23c40] Setting default whitelist 'file,crypto' [file @ 0x5fc7300] Setting default whitelist 'file,crypto' av_interleaved_write_frame(): Too many open files No more output streams to write to, finishing. [AVIOContext @ 0x5e23b00] Statistics: 0 seeks, 0 writeouts [hls muxer @ 0x3f36660] deleting old segment segment_128_0000001013.ts [hls muxer @ 0x3f36660] deleting old segment segment_128_0000001012.ts [file @ 0x3f61200] Setting default whitelist 'file,crypto' [hls muxer @ 0x3f36660] EXT-X-MEDIA-SEQUENCE:1018 [AVIOContext @ 0x3f3d4a0] Statistics: 0 seeks, 1 writeouts size=N/A time=00:17:00.00 bitrate=N/A speed=46.6x video:0kB audio:15975kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Input file #0 (sine): Input stream #0:0 (audio): 43930 packets read (89968640 bytes); 43930 frames decoded (44984320 samples); Total: 43930 packets (89968640 bytes) demuxed Output file #0 (playlist_128.m3u8): Output stream #0:0 (audio): 43930 frames encoded (44984320 samples); 43929 packets muxed (16358478 bytes); Total: 43929 packets (16358478 bytes) muxed 43930 frames successfully decoded, 0 decoding errors [aac @ 0x3f384c0] Qavg: 2072.816 Conversion failed!
Does it only happen using the aac encoder?
No, it also happens with libfdk_aac and libmp3lame (which would make no sense on hls)
Are two output streams necessary to reproduce the issue or does it also happen with one output stream?
One output stream is sufficient (see above output)
Do you think this is a regression?
I don't know yet, i recently ran the hls muxer for hours with no issues, but this is not comparable as the output had a video stream. I will test it against some older versions
comment:3 by , 7 years ago
This is definitely a regression. Tested the same with the 3.2 release and the behaviour is very consistent. Works well
lsof tells that only one segment file is open.
lsof -p 16633 | grep segment ffmpeg 16633 nicola 3w REG 252,0 45684 88215194 /var/www/html/segment_128_0000000607.ts
ffmpeg version 3.2 Copyright (c) 2000-2016 the FFmpeg developers built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
comment:4 by , 7 years ago
Owner: | set to |
---|---|
Status: | new → open |
follow-up: 6 comment:5 by , 7 years ago
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1285 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1286 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1287 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1288 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1289 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1290 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1291x size=N/A time=00:43:10.83 bitrate=N/A speed=56.3x video:0kB audio:21915kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Input file #0 (sine): Input stream #0:0 (audio): 111578 packets read (228511744 bytes); 111578 frames decoded (114255872 samples); Total: 111578 packets (228511744 bytes) demuxed Output file #0 (output_test.m3u8): Output stream #0:0 (audio): 111578 frames encoded (114255872 samples); 111579 packets muxed (22440703 bytes); Total: 111579 packets (22440703 bytes) muxed [aac @ 0x7faf1f002800] Qavg: 316.103 MacBook:ffmpeg StevenLiu$ look at the lsof MacBook:book StevenLiu$ MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ MacBook:ffmpeg StevenLiu$ ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8 MacBook:book StevenLiu$ ps -ef|grep ffmpeg 503 26387 15735 0 10:55下午 ttys006 0:00.00 grep ffmpeg 503 26380 15744 0 10:55下午 ttys008 0:18.67 ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8
comment:6 by , 7 years ago
Replying to stevenliu:
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1285 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1286 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1287 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1288 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1289 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1290 [hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1291x size=N/A time=00:43:10.83 bitrate=N/A speed=56.3x video:0kB audio:21915kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Input file #0 (sine): Input stream #0:0 (audio): 111578 packets read (228511744 bytes); 111578 frames decoded (114255872 samples); Total: 111578 packets (228511744 bytes) demuxed Output file #0 (output_test.m3u8): Output stream #0:0 (audio): 111578 frames encoded (114255872 samples); 111579 packets muxed (22440703 bytes); Total: 111579 packets (22440703 bytes) muxed [aac @ 0x7faf1f002800] Qavg: 316.103 MacBook:ffmpeg StevenLiu$ look at the lsof MacBook:book StevenLiu$ MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l 0 MacBook:book StevenLiu$ MacBook:ffmpeg StevenLiu$ ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8 MacBook:book StevenLiu$ ps -ef|grep ffmpeg 503 26387 15735 0 10:55下午 ttys006 0:00.00 grep ffmpeg 503 26380 15744 0 10:55下午 ttys008 0:18.67 ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8
Well the lsof with the grep does not work in that case, as the segment files get named output_test*.ts. If the process terminates, the handles get released. For testing, i started the process as a daemon
follow-up: 8 comment:7 by , 7 years ago
This a little helper i made to simplify testing
ffmpeg-mux-to-hls.sh
#!/bin/bash on_die () { # kill all children pkill -SIGTERM -P $$ } trap 'on_die' TERM ffmpeg -v verbose -f lavfi -i sine \ -c:a libfdk_aac \ -vn \ -b:a 128000 \ -f hls \ -hls_time 3 \ -hls_list_size 3 \ -hls_flags delete_segments \ -hls_segment_filename segment_128_%10d.ts \ playlist_128.m3u8 2>>ffmpeg-all.log & wait
and then:
./ffmpeg-mux-to-hls.sh &
then check with
lsof -p <PID>
in the current master the files won't get released, in the 3.2 stable they get released
comment:8 by , 7 years ago
Replying to ncon:
This a little helper i made to simplify testing
ffmpeg-mux-to-hls.sh
#!/bin/bash on_die () { # kill all children pkill -SIGTERM -P $$ } trap 'on_die' TERM ffmpeg -v verbose -f lavfi -i sine \ -c:a libfdk_aac \ -vn \ -b:a 128000 \ -f hls \ -hls_time 3 \ -hls_list_size 3 \ -hls_flags delete_segments \ -hls_segment_filename segment_128_%10d.ts \ playlist_128.m3u8 2>>ffmpeg-all.log & waitand then:
./ffmpeg-mux-to-hls.sh &then check with
lsof -p <PID>in the current master the files won't get released, in the 3.2 stable they get released
https://patchwork.ffmpeg.org/patch/2706/
try it please.
comment:9 by , 7 years ago
Component: | undetermined → avformat |
---|---|
Keywords: | regression added |
Priority: | normal → important |
comment:10 by , 7 years ago
Resolution: | → fixed |
---|---|
Status: | open → closed |
fix commit id: 4507f29e4a6a4363e0179c02bdb78d55e4d9a12c
uncut console output (compressed due to upload limit)