hls_flags delete_segments – file desctriptors not freeing, ffmpeg segfaults when system limit is reached

[Rafał Wiosna]

Summary of the bug:
When delete_segments HLS flag is enabled, the FFmpeg binary segfaults when max open files limit is reached.

How to reproduce:

/usr/local/bin/ffmpeg -loglevel info -thread_queue_size 1024 -f decklink -i DeckLink 4K Extreme@2 -audio_input embedded -video_input sdi -threads 0 -fflags +genpts -flags +global_header -c:a libfdk_aac 
-map 0:v -s:v 1280x720 -b:v 3600k -minrate:v 3000k -maxrate:v 3700k -bufsize:v 3600k -pix_fmt yuv420p -vf yadif
-map 0:a -af aresample=48000 -b:a 128k
-c:v libx264 -preset slow -profile:v main -x264opts keyint=100:min-keyint=100:scenecut=-1 -hls_time 4 -hls_list_size 5 -hls_start_number_source epoch -hls_flags delete_segments /var/www/html/live/test1/live3600k.m3u8
-map 0:v -s:v 1024x576 -b:v 2400k -minrate:v 2000k -maxrate:v 2500k -bufsize:v 2400k -pix_fmt yuv420p -vf yadif
-map 0:a -af aresample=44100 -b:a 128k
-c:v libx264 -preset slow -profile:v main -x264opts keyint=100:min-keyint=100:scenecut=-1 -hls_time 4 -hls_list_size 5 -hls_start_number_source epoch -hls_flags delete_segments /var/www/html/live/test1/live2400k.m3u8
-map 0:v -s:v 720x404 -b:v 1700k -minrate:v 1000k -maxrate:v 1800k -bufsize:v 1700k -pix_fmt yuv420p -vf yadif
-map 0:a -af aresample=44100 -b:a 96k -c:v libx264 -preset slow -profile:v main -x264opts keyint=100:min-keyint=100:scenecut=-1 -hls_time 4 -hls_list_size 5 -hls_start_number_source epoch -hls_flags delete_segments /var/www/html/live/test1/live1700k.m3u8
-map 0:v -s:v 512x288 -b:v 900k -minrate:v 800k -maxrate:v 1000k -bufsize:v 900k -pix_fmt yuv420p -vf yadif
-map 0:a -af aresample=44100 -b:a 64k
-c:v libx264 -preset slow -profile:v main -x264opts keyint=100:min-keyint=100:scenecut=-1 -hls_time 4 -hls_list_size 5 -hls_start_number_source epoch -hls_flags delete_segments /var/www/html/live/test1/live900k.m3u8
-map 0:v -s:v 512x288 -b:v 450k -minrate:v 400k -maxrate:v 500k -bufsize:v 450k -pix_fmt yuv420p -vf yadif
-map 0:a -af aresample=44100 -b:a 64k
-c:v libx264 -preset slow -profile:v main -x264opts keyint=100:min-keyint=100:scenecut=-1 -hls_time 4 -hls_list_size 5 -hls_start_number_source epoch -hls_flags delete_segments /var/www/html/live/test1/live450k.m3u8
-map 0:v -s:v 512x288 -b:v 120k -minrate:v 100k -maxrate:v 150k -bufsize:v 120k -pix_fmt yuv420p -vf yadif
-map 0:a -af aresample=32000 -b:a 32k
-c:v libx264 -preset slow -profile:v main -x264opts keyint=100:min-keyint=100:scenecut=-1 -hls_time 4 -hls_list_size 5 -hls_start_number_source epoch -hls_flags delete_segments /var/www/html/live/test1/live120k.m3u8

Build:
ffmpeg version N-83663-g7e9ba78 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 4.9.2 (Debian 4.9.2-10)
  configuration: --prefix=/home/vagrant/ffmpeg-src/ffmpeg/ --bindir=/usr/local --pkg-config-flags=--static --extra-cflags='-I/home/vagrant/decklink-include -static' --extra-ldflags=-L/home/vagrant/decklink-include --enable-gpl --disable-shared --disable-doc --enable-static --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-decklink
  libavutil      55. 47.100 / 55. 47.100
  libavcodec     57. 81.100 / 57. 81.100
  libavformat    57. 66.102 / 57. 66.102
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 74.100 /  6. 74.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Hyper fast Audio and Video encoder

How to check:

... after a few minutes of encoding ...
lsof -p `pidof ffmpeg`|grep '(deleted)'|wc -l
1812

Temporary fix:

# grep NOFILE /etc/systemd/system/ffhls_decklink.service
LimitNOFILE=65500

Comment:
I've looked into the source and it seems that this is a problem in libavformat/hlsenc.c hls_delete_old_segments function:

proto = avio_find_protocol_name(s->filename);
        if (hls->method || (proto && !av_strcasecmp(proto, "http"))) {
            av_dict_set(&options, "method", "DELETE", 0);
            if ((ret = hls->avf->io_open(hls->avf, &out, path, AVIO_FLAG_WRITE, &options)) < 0)
                goto fail;
            ff_format_io_close(hls->avf, &out);
        } else if (unlink(path) < 0) {
            av_log(hls, AV_LOG_ERROR, "failed to delete old segment %s: %s\n",
                                     path, strerror(errno));
        }

I'm no expert, in fact I'm a rookie when it comes to debuging ffmpeg source, but unlink() without some sort of close() [is it ff_format_io_close()?] may be the cause of this problem. [And the same applies to unlink(sub_path) found later in that function.]

[Steven Liu]

use git pull to update to newest commit please,
This problem has been fix several days ago.

commit id: 4507f29e4a6a4363e0179c02bdb78d55e4d9a12c
refer to : ​https://trac.ffmpeg.org/ticket/6204

[Steven Liu]

duplicate: #6204

[Rafał Wiosna]

That's not it. I'm aware of 4507f29e4a6a4363e0179c02bdb78d55e4d9a12c patch. I've applied it to my source tree and it still segfaulting.

ff_format_io_close(s, &oc->pb) only happens under this condition:

    if (can_split && av_compare_ts(pkt->pts - hls->start_pts, st->time_base,
                                   end_pts, AV_TIME_BASE_Q) >= 0) {

My setup is rather strange, with 6 concurrent outputs. Please test my invocation, it only takes a minute or so to see if the process hugs those deleted segments.

[Rafał Wiosna]

I've found one cause for can_split being false.

    if (hls->has_video) {
        can_split = st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO &&
                    ((pkt->flags & AV_PKT_FLAG_KEY) || (hls->flags & HLS_SPLIT_BY_TIME));
        is_ref_pkt = st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO;
    }

The variable is positive when a keyframe happens or split is forced and it's a video stream. But just after this this check happens:

    if (pkt->pts == AV_NOPTS_VALUE)
        is_ref_pkt = can_split = 0;

Maybe my setup is somewhat brokend and AV_NOPTS happens? As you can see, I use Decklink for input.

[Steven Liu]

Replying to rafamiga:

That's not it. I'm aware of 4507f29e4a6a4363e0179c02bdb78d55e4d9a12c patch. I've applied it to my source tree and it still segfaulting.

ff_format_io_close(s, &oc->pb) only happens under this condition:

    if (can_split && av_compare_ts(pkt->pts - hls->start_pts, st->time_base,
                                   end_pts, AV_TIME_BASE_Q) >= 0) {

My setup is rather strange, with 6 concurrent outputs. Please test my invocation, it only takes a minute or so to see if the process hugs those deleted segments.

use the newest version and check it, i saw your version is same with the ticket 6204

[Rafał Wiosna]

Sure, the version string is the same but the source's patched. I may pull the newest source but I'm pretty sure the problem won't disappear. And since it's trivial to run the test, maybe it's easier to do so?

[Steven Liu]

And dump the input stream to a file ,upload the file here, then let's reproduce it

[Carl Eugen Hoyos]

Please reopen this ticket if the issue is reproducible with current FFmpeg git head, please do not test patched FFmpeg versions.