Opened 2 years ago

Closed 2 years ago

#8647 closed defect (invalid)

nginx rtmp via ffmpeg to hls output buffering issues

Reported by: lukea Owned by:
Priority: normal Component: undetermined
Version: unspecified Keywords: hls
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Summary of the bug:


We have a kubernetes cluster which we are using to setup a streaming service. The service uses the nginx-rtmp module behind a TCP load balancer. The nginx-rtmp module runs an exec command which calls ffmpeg (4.2.2 built from source with libfdk-aac enabled, full compile options are below). The ffmpeg command then sends the output via -f hls (through an nginx ingress which balances requests) to a second nginx instance (pod) which allows the hls files to be played. We have observed a couple of issues which we think are linked to ffmpeg (probably hls parts of it).

Firstly, we noticed that when the buffering happens although the hls segments arrive bang on time as expected, the index file sometimes does not - so the index is then unaware of the segment file, which appears to cause the buffering until the index is re-sent with the next segment data.

Further to this we disabled the sending of files over http to the second nginx instance and just started writing the segments locally - although the problem took longer to appear, we still observed the same behaviour.

On the nginx ingress (which balances between the ffmpeg command and the nginx pods that handle the segments) we also noticed a bunch of 499 error codes (see attached image)

We are not certain if this is a config issue with our ffmpeg command, or a bug with ffmpeg - most likely the hls component within, though we have cut down the ffmpeg command somewhat to be more minimal to try and debug this.

Please let me know if you need any more debug output?

How to reproduce:

    /opt/ffmpeg/bin/ffmpeg -i rtmp://${APP_NAME}/${INPUT_STREAM_NAME} \
        -map 0 \
        -framerate 30 -g 30 -keyint_min 30 -force_key_frames "expr:gte(t,n_forced*1)" -sc_threshold 0 -b_strategy 0 \
        -c:v:0 libx264 -b:v:0 2000k -maxrate:v:0 2200k -bufsize:v:0 3000k -s:v:0 1280x720 -aspect:v:0 16\:9 -profile:v:0 main -crf:v:0 25 \
        -c:a libfdk_aac \
        -b:a:0 320k -maxrate:a:0 480k -bufsize:a:0 320k \
        -hls_time 2 \
        -hls_list_size 30 \
        -hls_segment_type fmp4 \
        -hls_start_number_source datetime \
        -var_stream_map "v:0,a:0,name:720p" \
        -master_pl_name "master.m3u8" \
        -master_pl_publish_rate 1 \
        -hls_flags +program_date_time-temp_file \
        -f hls -method PUT ${UPSTREAM_TS_CACHE}/${PUBLIC_STREAM_NAME}/index-%v.m3u8

ffmpeg compiled with:

  ./configure \
  --disable-debug \
  --disable-doc \
  --disable-ffplay \
  --enable-avfilter \
  --enable-avresample \
  --enable-gpl \
  --enable-libaom \
  --enable-libass \
  --enable-libfdk-aac \
  --enable-libfreetype \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-librtmp \
  --enable-libtheora \
  --enable-libvorbis \
  --enable-libvpx \
  --enable-libwebp \
  --enable-libx264 \
  --enable-libx265 \
  --enable-nonfree \
  --enable-openssl \
  --enable-pic \
  --enable-postproc \
  --enable-pthreads \
  --enable-small \
  --enable-version3 \
  --extra-cflags="-I${PREFIX}/include" \
  --extra-ldflags="-L${PREFIX}/lib" \
  --extra-libs="-lpthread -lm" \
  --prefix="${PREFIX}" && \
  make && make install && make distclean

Attachments (1)

Screenshot 2020-05-01 at 08.46.41.png (1.8 MB ) - added by lukea 2 years ago.
Screenshot of nginx ingress 499s

Download all attachments as: .zip

Change History (3)

by lukea, 2 years ago

Screenshot of nginx ingress 499s

comment:1 by Carl Eugen Hoyos, 2 years ago

Component: ffmpegundetermined
Priority: importantnormal
Version: 4.2unspecified

Please test current FFmpeg git head, the only version supported on this bug tracker, and provide a simplified command line together with the complete, uncut console output to make this a valid ticket.

comment:2 by lukea, 2 years ago

Resolution: invalid
Status: newclosed


I did as instructed, compiled a version of ffmpeg from git head - however when we noticed that the 499 error code was coming from the nginx ingress that was proxying on to the pod that handled the hls segments - it made us think about temporarily bypassing it, we then noticed that the hls segments and index were getting through to the hls segment pods without issue.

So it looks like ffmpeg hls was not waiting for a return code and the ingress was closing the connection.

After we enabled proxy_ignore_client_abort on; in the ingress configuration, the 499s stopped occurring which resulted in the service working as expected.

I'll close this one, thanks for responding though!

Note: See TracTickets for help on using tickets.