Opened 3 months ago

Last modified 20 hours ago

#7547 new defect

When relaying RTMP streams ffmpeg quits with av_interleaved_write_frame message

Reported by: regstuff Owned by:
Priority: important Component: avformat
Version: git-master Keywords: regression librtmp
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug: I have an Nginx server (with rtmp-module) on a Ubuntu-16.04 VM. I'm using it to receive an input rtmp stream from my PC. FFmpeg is then used in the VM to take this input and relay to multiple destinations after transcoding, overlaying a lower third, and changing the volume.

However, FFmpeg quits after a random duration between 3-30 minutes with an "av_interleaved_write_frame(): End of file" message.

How to reproduce: Pull an rtmp stream as input. Command used was as below. In the below command, I have used aac and strict -2 as well. I am aware that strict -2 is no longer necessary. The error occurs even with other aac encoders such as libfdk_aac and without strict -2. The reason I have used this particular command is explained in more detail below.

/usr/local/bin/ffmpeg -v 9 -loglevel 99 -nostdin -thread_queue_size 512 -i rtmp://127.0.0.1:1935/input/stream1 -i /usr/local/nginx/scripts/images/lowerthird.png -af azmq,volume=2 -c:a aac -filter_complex 'zmq=bind_address=tcp\\\://127.0.0.1\\\:5559,overlay=0:H' -vcodec libx264 -pix_fmt yuv420p -preset veryfast -r 25 -g 50 -b:v 6000k -maxrate 6M -minrate 6M -bufsize 6M -f flv -strict -2 -f flv rtmp://127.0.0.1:1935/distribute/stream1 -y -report

This happens with the latest ffmpeg version (details below). However, I do not see this issue when I use the standard FFmpeg that comes with Ubuntu 16.04's apt-get install (ffmpeg version 2.8.15). To do an aac encode with the Ubuntu standard Ffmpeg, I have to use aac and strict -2. This is why I have used the command above with the latest Ffmpeg version as well, just to standardize the comparsion. Just to reiterate, this issue occurs even when I use a different encoder and without the -strict -2 flags with the latest ffmpeg.

ffmpeg version

ffmpeg version N-92427-gae43235 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --extra-libs='-lpthread -lm' --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzmq --enable-librtmp --enable-network --enable-nonfree
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 43.100 /  7. 43.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100

Have attached log output with this ticket. The issue occurs even if I remove all encoding and just do a codec copy as below.

/usr/local/bin/ffmpeg -nostdin -thread_queue_size 512 -i rtmp://127.0.0.1:1935/stream1/input -c copy -f flv rtmp://a.rtmp.youtube.com/live2/[my key] -y  

Issue occurs irrespective of destination, whether youtube, facebook or just another location on the same nginx server.

Issue also occurs whether thread_queue_size is 8 or 1024 or 512. It appears that this issue is more frequent in a 4-core system rather than a single core system I was using. Not totally sure about that though.

Attachments (2)

ffmpeg-20181114-092056.log (824.0 KB) - added by regstuff 3 months ago.
Ffmpeg Log File
issue7547.mp4 (2.2 MB) - added by ruddell 3 weeks ago.
Sample for reproduction

Change History (16)

Changed 3 months ago by regstuff

Ffmpeg Log File

comment:1 Changed 4 weeks ago by lbgaus

I was having the same problem as regstuff, and found that I was able to resolve my problem with 'av_interleaved_write_frame(): End of file' errors being thrown after about 2 to 3 hours of publishing to an RTMP server by compiling off the 3.4.5 branch, instead of the master. Something must be wrong with the latest version causing this issue?

Last edited 4 weeks ago by lbgaus (previous) (diff)

comment:2 Changed 4 weeks ago by cehoyos

  • Component changed from ffmpeg to undetermined
  • Keywords regression added
  • Version changed from unspecified to git-master

Please use git bisect to find the change introducing the regression.

comment:3 Changed 4 weeks ago by ikolesnikov

  • Priority changed from normal to important

I have the same problem while transcoding udp multicast to rtmp via the following command:

ffmpeg -y -hwaccel cuvid -c:v mpeg2_cuvid -resize 1280x720 -deint 2 -drop_second_field 1 -i "udp://235.1.10.1:1234?overrun_nonfatal=1" -aspect 16:9 -vcodec h264_nvenc -b:v 1500k -acodec libfdk_aac -f flv rtmp://127.0.0.1/stream/out

This issue occurs for the newest snapshot from Github. Also, I got the same issue for the 4.1 release version. The same command works well with the 3.4 release version. Unfortunately, we can't use the new features implemented in FFmpeg due to this bug. The problem with RTMP muxing is important for many users

comment:4 Changed 4 weeks ago by cehoyos

  • Priority changed from important to normal

If the ticket were important you would have used git bisect to point us to the offending commit.

Changed 3 weeks ago by ruddell

Sample for reproduction

comment:5 Changed 3 weeks ago by ruddell

I have this problem on Ubuntu 16.04.5 and Mac with v4.1 when using librtmp. It works with v3.4.5 on Ubuntu as described above.

I attached a file that shows the issue with the following command (about 4 seconds in):

ffmpeg -re -i issue7547.mp4 -vcodec copy -acodec copy -bsf:a aac_adtstoasc -f flv 'rtmp://a.rtmp.youtube.com/live2/stream-key-here'

The first commit where streaming hangs, git bisect shows the following:

commit 858db4b01fa2b55ee55056c033054ca54ac9b0fd
Author: Daniel Kucera <daniel.kucera@gmail.com>
Date:   Tue Oct 17 10:29:30 2017 +0200

    libavformat: not treat 0 as EOF
    
    transfer_func variable passed to retry_transfer_wrapper
    are h->prot->url_read and h->prot->url_write functions.
    These need to return EOF or other error properly.
    In case of returning >= 0, url_read/url_write is retried
    until error is returned.
    
    Signed-off-by: Daniel Kucera <daniel.kucera@gmail.com>

:040000 040000 03f37e2b4749f5f1d21a9a528c0a10eef8ff2cf1 6c2edb3a01134a03e79534f6cbb048abb59121d2 M	libavformat

The first commit where the error message is correctly displayed, git bisect shows the following:

commit ed647ab79f9a54d8d3a8e345f6a1643b60b849f4
Author: Timo Rothenpieler <timo@rothenpieler.org>
Date:   Thu Jul 26 00:37:35 2018 +0200

    avformat/librtmp: fix returning EOF from Read/Write
    
    Ticket #7052

:040000 040000 964b972c7b3df952e7b634e4a76cdefaa8da3622 200f7038acf21315bee42874f69ae41d6c31315d M	libavformat
Last edited 22 hours ago by ruddell (previous) (diff)

comment:6 Changed 3 weeks ago by ikolesnikov

  • Component changed from undetermined to avformat

comment:7 Changed 3 weeks ago by ikolesnikov

  • Priority changed from normal to important

comment:8 Changed 3 weeks ago by cehoyos

  • Keywords librtmp added

comment:9 Changed 3 weeks ago by pgalow

I can confirm we are running into the same issue on macOS (10.13.6 and 10.14.3) using 4.1 (installed via Homebrew). So far no issues after downgrading to 3.4.5.

comment:10 Changed 34 hours ago by danman

Guys, can someone of you make a tcpdump of the output RTMP stream?

comment:11 Changed 26 hours ago by oromit

Try reverting the first hunk of ed647ab79f9a54d8d3a8e345f6a1643b60b849f4.
The one that changes RTMP_Write. Only RTMP_Read seems to actually do EOF.

comment:12 Changed 26 hours ago by oromit

On another note: Try just not using librtmp. The ffmpeg native rtmp protocol works fine for me.

comment:13 Changed 22 hours ago by ruddell

@oromit Reverting the first hunk of that commit causes it to hang at the failing point rather than exit with an error message. The issue first appeared in the other commit in my previous message.

You are correct that using ffmpeg's rtmp instead of librtmp works fine, which explains why I couldn't reproduce on Mac earlier. I updated my comment to note that.

@danman I tried but I'm not familiar with tcpdump. I attached a file and command that reproduces it though.

Last edited 22 hours ago by ruddell (previous) (diff)

comment:14 Changed 20 hours ago by oromit

I'd say it's time to get rid of the librtmp support.
I see the same issues here, and the easy fix is to not use librtmp.

Note: See TracTickets for help on using tickets.