Opened 4 months ago

Closed 4 months ago

#6204 closed defect (fixed)

AAC-only hls stream slowly exceeds file descriptors

Reported by: ncon Owned by: stevenliu
Priority: important Component: avformat
Version: git-master Keywords: hls regression
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:

The hls muxer does not release the segment's file descriptors if there is only an audio-only stream. This happens regardless of the delete_segments flag or -re flag being set or not. Once ffmpeg runs out of avalable FD's, it terminates gracefully with the following output:

Excerpt (full output included in file):

[hls muxer @ 0x3a3d5a0] deleting old segment segment_128_0000000501.ts
[file @ 0x59c0100] Setting default whitelist 'file,crypto'
[file @ 0x5a85ba0] Setting default whitelist 'file,crypto'
[hls muxer @ 0x3a3d5a0] EXT-X-MEDIA-SEQUENCE:506
[AVIOContext @ 0x5b21a60] Statistics: 0 seeks, 1 writeouts
[hls muxer @ 0x3aa6500] deleting old segment segment_96_0000000501.ts
[file @ 0x5b7ce00] Setting default whitelist 'file,crypto'
[file @ 0x5b7cf00] Setting default whitelist 'file,crypto'
av_interleaved_write_frame(): Too many open files
No more output streams to write to, finishing.

How to reproduce:

% ffmpeg -v 9 -loglevel 99 -re -i http://127.0.0.1:8000/test.ogg \
          -c:a aac \
          -vn \
          -b:a 128000 \
          -f hls \
          -hls_time 6 \
	  -hls_list_size 3 \
	  -hls_flags delete_segments \
          -hls_segment_filename segment_128_%10d.ts \
          playlist_128.m3u8 \
          -c:a aac \
          -vn \
          -b:a 96000 \
          -f hls \
          -hls_time 6 \
	  -hls_list_size 3 \
	  -hls_flags delete_segments \
          -hls_segment_filename segment_96_%10d.ts \
          playlist_96.m3u8 2>>ffmpeg-all.log

ffmpeg version N-83663-g7e9ba78
built on Ubuntu 16.04.2 LTS (4.4.0-62-generic)
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
configured with --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree

Please note that it takes some time to happen and depends on the limits on the system.

the machine's "max open files" was set to 1024.

The provided live source can be any kind of audio source. However, the tested source was an OGG source provided by an Icecast2 instance.

while the process is running you can see the open files (deleted segments included) stacking up until the termination of the ffmpeg process:

lsof -p <PID> | grep segment | grep deleted | wc -l
656

Attachments (1)

ffmpeg-all.log.gz (1.5 MB) - added by ncon 4 months ago.
uncut console output (compressed due to upload limit)

Download all attachments as: .zip

Change History (11)

Changed 4 months ago by ncon

uncut console output (compressed due to upload limit)

comment:1 follow-up: Changed 4 months ago by cehoyos

  • Component changed from ffmpeg to undetermined
  • Keywords aac segments removed
  • Priority changed from important to normal

Is the issue reproducible if the input is -f lavfi -i sine?
Does it only happen using the aac encoder?
Are two output streams necessary to reproduce the issue or does it also happen with one output stream?
Do you think this is a regression?

comment:2 in reply to: ↑ 1 Changed 4 months ago by ncon

Replying to cehoyos:

Is the issue reproducible if the input is -f lavfi -i sine?

Yes:

[hls muxer @ 0x3f36660] deleting old segment segment_128_0000001011.ts
[file @ 0x5e23c40] Setting default whitelist 'file,crypto'
[file @ 0x5fc7300] Setting default whitelist 'file,crypto'
av_interleaved_write_frame(): Too many open files
No more output streams to write to, finishing.
[AVIOContext @ 0x5e23b00] Statistics: 0 seeks, 0 writeouts
[hls muxer @ 0x3f36660] deleting old segment segment_128_0000001013.ts
[hls muxer @ 0x3f36660] deleting old segment segment_128_0000001012.ts
[file @ 0x3f61200] Setting default whitelist 'file,crypto'
[hls muxer @ 0x3f36660] EXT-X-MEDIA-SEQUENCE:1018
[AVIOContext @ 0x3f3d4a0] Statistics: 0 seeks, 1 writeouts
size=N/A time=00:17:00.00 bitrate=N/A speed=46.6x    
video:0kB audio:15975kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (sine):
  Input stream #0:0 (audio): 43930 packets read (89968640 bytes); 43930 frames decoded (44984320 samples); 
  Total: 43930 packets (89968640 bytes) demuxed
Output file #0 (playlist_128.m3u8):
  Output stream #0:0 (audio): 43930 frames encoded (44984320 samples); 43929 packets muxed (16358478 bytes); 
  Total: 43929 packets (16358478 bytes) muxed
43930 frames successfully decoded, 0 decoding errors
[aac @ 0x3f384c0] Qavg: 2072.816
Conversion failed!

Does it only happen using the aac encoder?

No, it also happens with libfdk_aac and libmp3lame (which would make no sense on hls)

Are two output streams necessary to reproduce the issue or does it also happen with one output stream?

One output stream is sufficient (see above output)

Do you think this is a regression?

I don't know yet, i recently ran the hls muxer for hours with no issues, but this is not comparable as the output had a video stream. I will test it against some older versions

comment:3 Changed 4 months ago by ncon

This is definitely a regression. Tested the same with the 3.2 release and the behaviour is very consistent. Works well

lsof tells that only one segment file is open.

lsof -p 16633 | grep segment
ffmpeg  16633 nicola    3w   REG  252,0    45684  88215194 /var/www/html/segment_128_0000000607.ts
ffmpeg version 3.2 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609

Last edited 4 months ago by ncon (previous) (diff)

comment:4 Changed 4 months ago by stevenliu

  • Owner set to stevenliu
  • Status changed from new to open

comment:5 follow-up: Changed 4 months ago by stevenliu

[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1285
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1286
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1287
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1288
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1289
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1290
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1291x    
size=N/A time=00:43:10.83 bitrate=N/A speed=56.3x    
video:0kB audio:21915kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (sine):
  Input stream #0:0 (audio): 111578 packets read (228511744 bytes); 111578 frames decoded (114255872 samples); 
  Total: 111578 packets (228511744 bytes) demuxed
Output file #0 (output_test.m3u8):
  Output stream #0:0 (audio): 111578 frames encoded (114255872 samples); 111579 packets muxed (22440703 bytes); 
  Total: 111579 packets (22440703 bytes) muxed
[aac @ 0x7faf1f002800] Qavg: 316.103
MacBook:ffmpeg StevenLiu$ 





look at the lsof


MacBook:book StevenLiu$ 
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ 
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ 




MacBook:ffmpeg StevenLiu$ ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8






MacBook:book StevenLiu$ ps -ef|grep ffmpeg
  503 26387 15735   0 10:55下午 ttys006    0:00.00 grep ffmpeg
  503 26380 15744   0 10:55下午 ttys008    0:18.67 ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8





comment:6 in reply to: ↑ 5 Changed 4 months ago by ncon

Replying to stevenliu:

[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1285
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1286
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1287
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1288
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1289
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1290
[hls @ 0x7faf1f812c00] EXT-X-MEDIA-SEQUENCE:1291x    
size=N/A time=00:43:10.83 bitrate=N/A speed=56.3x    
video:0kB audio:21915kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (sine):
  Input stream #0:0 (audio): 111578 packets read (228511744 bytes); 111578 frames decoded (114255872 samples); 
  Total: 111578 packets (228511744 bytes) demuxed
Output file #0 (output_test.m3u8):
  Output stream #0:0 (audio): 111578 frames encoded (114255872 samples); 111579 packets muxed (22440703 bytes); 
  Total: 111579 packets (22440703 bytes) muxed
[aac @ 0x7faf1f002800] Qavg: 316.103
MacBook:ffmpeg StevenLiu$ 





look at the lsof


MacBook:book StevenLiu$ 
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ 
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ lsof -p 26380 | grep segment | grep deleted | wc -l
       0
MacBook:book StevenLiu$ 




MacBook:ffmpeg StevenLiu$ ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8






MacBook:book StevenLiu$ ps -ef|grep ffmpeg
  503 26387 15735   0 10:55下午 ttys006    0:00.00 grep ffmpeg
  503 26380 15744   0 10:55下午 ttys008    0:18.67 ffmpeg -v verbose -f lavfi -i sine -acodec aac -f hls output_test.m3u8





Well the lsof with the grep does not work in that case, as the segment files get named output_test*.ts. If the process terminates, the handles get released. For testing, i started the process as a daemon

comment:7 follow-up: Changed 4 months ago by ncon

This a little helper i made to simplify testing

ffmpeg-mux-to-hls.sh

#!/bin/bash

on_die ()
{
    # kill all children
    pkill -SIGTERM -P $$
}

trap 'on_die' TERM

ffmpeg -v verbose -f lavfi -i sine \
          -c:a libfdk_aac \
          -vn \
          -b:a 128000 \
          -f hls \
          -hls_time 3 \
          -hls_list_size 3 \
          -hls_flags delete_segments \
          -hls_segment_filename segment_128_%10d.ts \
          playlist_128.m3u8 2>>ffmpeg-all.log &
wait

and then:

./ffmpeg-mux-to-hls.sh &

then check with

lsof -p <PID>

in the current master the files won't get released, in the 3.2 stable they get released

comment:8 in reply to: ↑ 7 Changed 4 months ago by stevenliu

Replying to ncon:

This a little helper i made to simplify testing

ffmpeg-mux-to-hls.sh

#!/bin/bash

on_die ()
{
    # kill all children
    pkill -SIGTERM -P $$
}

trap 'on_die' TERM

ffmpeg -v verbose -f lavfi -i sine \
          -c:a libfdk_aac \
          -vn \
          -b:a 128000 \
          -f hls \
          -hls_time 3 \
          -hls_list_size 3 \
          -hls_flags delete_segments \
          -hls_segment_filename segment_128_%10d.ts \
          playlist_128.m3u8 2>>ffmpeg-all.log &
wait

and then:

./ffmpeg-mux-to-hls.sh &

then check with

lsof -p <PID>

in the current master the files won't get released, in the 3.2 stable they get released

https://patchwork.ffmpeg.org/patch/2706/

try it please.

comment:9 Changed 4 months ago by cehoyos

  • Component changed from undetermined to avformat
  • Keywords regression added
  • Priority changed from normal to important

comment:10 Changed 4 months ago by stevenliu

  • Resolution set to fixed
  • Status changed from open to closed
Note: See TracTickets for help on using tickets.