When relaying RTMP streams ffmpeg quits with av_interleaved_write_frame message

[regstuff]

Summary of the bug: I have an Nginx server (with rtmp-module) on a Ubuntu-16.04 VM. I'm using it to receive an input rtmp stream from my PC. FFmpeg is then used in the VM to take this input and relay to multiple destinations after transcoding, overlaying a lower third, and changing the volume.

However, FFmpeg quits after a random duration between 3-30 minutes with an "av_interleaved_write_frame(): End of file" message.

How to reproduce: Pull an rtmp stream as input. Command used was as below. In the below command, I have used aac and strict -2 as well. I am aware that strict -2 is no longer necessary. The error occurs even with other aac encoders such as libfdk_aac and without strict -2. The reason I have used this particular command is explained in more detail below.

/usr/local/bin/ffmpeg -v 9 -loglevel 99 -nostdin -thread_queue_size 512 -i rtmp://127.0.0.1:1935/input/stream1 -i /usr/local/nginx/scripts/images/lowerthird.png -af azmq,volume=2 -c:a aac -filter_complex 'zmq=bind_address=tcp\\\://127.0.0.1\\\:5559,overlay=0:H' -vcodec libx264 -pix_fmt yuv420p -preset veryfast -r 25 -g 50 -b:v 6000k -maxrate 6M -minrate 6M -bufsize 6M -f flv -strict -2 -f flv rtmp://127.0.0.1:1935/distribute/stream1 -y -report

This happens with the latest ffmpeg version (details below). However, I do not see this issue when I use the standard FFmpeg that comes with Ubuntu 16.04's apt-get install (ffmpeg version 2.8.15). To do an aac encode with the Ubuntu standard Ffmpeg, I have to use aac and strict -2. This is why I have used the command above with the latest Ffmpeg version as well, just to standardize the comparsion. Just to reiterate, this issue occurs even when I use a different encoder and without the -strict -2 flags with the latest ffmpeg.

ffmpeg version

ffmpeg version N-92427-gae43235 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --extra-libs='-lpthread -lm' --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzmq --enable-librtmp --enable-network --enable-nonfree
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 43.100 /  7. 43.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100

Have attached log output with this ticket. The issue occurs even if I remove all encoding and just do a codec copy as below.

/usr/local/bin/ffmpeg -nostdin -thread_queue_size 512 -i rtmp://127.0.0.1:1935/stream1/input -c copy -f flv rtmp://a.rtmp.youtube.com/live2/[my key] -y  

Issue occurs irrespective of destination, whether youtube, facebook or just another location on the same nginx server.

Issue also occurs whether thread_queue_size is 8 or 1024 or 512. It appears that this issue is more frequent in a 4-core system rather than a single core system I was using. Not totally sure about that though.

[regstuff]

Ffmpeg Log File

[LG]

I was having the same problem as regstuff, and found that I was able to resolve my problem with 'av_interleaved_write_frame(): End of file' errors being thrown after about 2 to 3 hours of publishing to an RTMP server by compiling off the 3.4.5 branch, instead of the master. Something must be wrong with the latest version causing this issue?

[Carl Eugen Hoyos]

Please use git bisect to find the change introducing the regression.

[Ivan Kolesnikov]

I have the same problem while transcoding udp multicast to rtmp via the following command:

ffmpeg -y -hwaccel cuvid -c:v mpeg2_cuvid -resize 1280x720 -deint 2 -drop_second_field 1 -i "udp://235.1.10.1:1234?overrun_nonfatal=1" -aspect 16:9 -vcodec h264_nvenc -b:v 1500k -acodec libfdk_aac -f flv rtmp://127.0.0.1/stream/out

This issue occurs for the newest snapshot from Github. Also, I got the same issue for the 4.1 release version. The same command works well with the 3.4 release version. Unfortunately, we can't use the new features implemented in FFmpeg due to this bug. The problem with RTMP muxing is important for many users

[Carl Eugen Hoyos]

If the ticket were important you would have used git bisect to point us to the offending commit.

[Jon Ruddell]

Sample for reproduction

[Jon Ruddell]

I have this problem on Ubuntu 16.04.5 and Mac with v4.1 when using librtmp. It works with v3.4.5 on Ubuntu as described above.

I attached a file that shows the issue with the following command (about 4 seconds in):

ffmpeg -re -i issue7547.mp4 -vcodec copy -acodec copy -bsf:a aac_adtstoasc -f flv 'rtmp://a.rtmp.youtube.com/live2/stream-key-here'

The first commit where streaming hangs, git bisect shows the following:

commit 858db4b01fa2b55ee55056c033054ca54ac9b0fd
Author: Daniel Kucera <daniel.kucera@gmail.com>
Date:   Tue Oct 17 10:29:30 2017 +0200

    libavformat: not treat 0 as EOF
    
    transfer_func variable passed to retry_transfer_wrapper
    are h->prot->url_read and h->prot->url_write functions.
    These need to return EOF or other error properly.
    In case of returning >= 0, url_read/url_write is retried
    until error is returned.
    
    Signed-off-by: Daniel Kucera <daniel.kucera@gmail.com>

:040000 040000 03f37e2b4749f5f1d21a9a528c0a10eef8ff2cf1 6c2edb3a01134a03e79534f6cbb048abb59121d2 M	libavformat

The first commit where the error message is correctly displayed, git bisect shows the following:

commit ed647ab79f9a54d8d3a8e345f6a1643b60b849f4
Author: Timo Rothenpieler <timo@rothenpieler.org>
Date:   Thu Jul 26 00:37:35 2018 +0200

    avformat/librtmp: fix returning EOF from Read/Write
    
    Ticket #7052

:040000 040000 964b972c7b3df952e7b634e4a76cdefaa8da3622 200f7038acf21315bee42874f69ae41d6c31315d M	libavformat

[pgalow]

I can confirm we are running into the same issue on macOS (10.13.6 and 10.14.3) using 4.1 (installed via Homebrew). So far no issues after downgrading to 3.4.5.

[danman]

Guys, can someone of you make a tcpdump of the output RTMP stream?

[Timo R.]

Try reverting the first hunk of ed647ab79f9a54d8d3a8e345f6a1643b60b849f4.
The one that changes RTMP_Write. Only RTMP_Read seems to actually do EOF.

[Timo R.]

On another note: Try just not using librtmp. The ffmpeg native rtmp protocol works fine for me.

[Jon Ruddell]

@oromit Reverting the first hunk of that commit causes it to hang at the failing point rather than exit with an error message. The issue first appeared in the other commit in my previous message.

You are correct that using ffmpeg's rtmp instead of librtmp works fine, which explains why I couldn't reproduce on Mac earlier. I updated my comment to note that.

@danman I tried but I'm not familiar with tcpdump. I attached a file and command that reproduces it though.

[Timo R.]

I'd say it's time to get rid of the librtmp support.
I see the same issues here, and the easy fix is to not use librtmp.

[danman]

@ruddell : I don't see the attachment but my random idea is to replace:

if (!ret)

with:

if (!ret && size)

or debug the value of "size" in ​http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=ed647ab79f9a54d8d3a8e345f6a1643b60b849f4

[Jon Ruddell]

@danman Please check the issue's attachments at the top. And the command (copied from my first message): ffmpeg -re -i issue7547.mp4 -vcodec copy -acodec copy -bsf:a aac_adtstoasc -f flv 'rtmp://a.rtmp.youtube.com/live2/stream-key-here'

I tested if (!ret && size) but it did not cause any difference. The value of size is 3 at the failing point.

...
Write Size: 361 
Write Size: 55 
Write Size: 32768 
Write Size: 3 
Write Size: 3 

I also tried if (!ret && size > 3) but it infinitely loops with Write Size: 3 (which is what happens before ed647ab79f9a54d8d3a8e345f6a1643b60b849f4)

Again, the issue occurs because of the other commit 858db4b01fa2b55ee55056c033054ca54ac9b0fd in my previous message. The ed647ab79f9a54d8d3a8e345f6a1643b60b849f4 commit simply makes FFMPEG print an error message instead of hanging on RTMP_Write with Size 3.

Thanks for looking into this.

[danman]

@ruddell the input file is not enough. I'd like to see the output stream produced. You should be able to capture it with:
tcpdump -i any port 1935 -w 7547.pcap

[Jon Ruddell]

output stream pcap

[Jon Ruddell]

Thank you for the help with the command. I have attached a pcap file to this issue.

[regstuff]

To add to the previous comments, I can confirm that the latest version of ffmpeg works fine if I dont use librtmp. Just did a 16 hour test with no issues.

[Ivan Kolesnikov]

@regstuff @orimit could you please explain how to use ffmpeg native rtmp protocol instead of librtmp?
--disable-librtmp is it enough?

[Tom Wilson]

Replying to ikolesnikov:

I can confirm that --disable-librtmp at the ./configure step resolved this issue

This worked for me when I had the same issue

ffmpeg 4.1 with librtmp + Ubuntu 16.04 streaming to nginx-rtmp = error described in the original post
ffmpeg 4.1 with librtmp disabled is fine

[llogan]

No need to use --disable-librtmp as librtmp support is not automatically enabled and is only enabled if you use --enable-librtmp. Refer to ./configure --help to see what is automatically enabled.

So just omit --enable-librtmp.

[terry]

hi all:

I have the same problem.

reproduce steps:

1. send the stream some seconds (about 10 seconds).
2. stop the stream 5~10 seconds
3. resend the stream, the ffmpeg exit with:

[flv @ 0x3e733c0] Failed to update header with correct duration.
[flv @ 0x3e733c0] Failed to update header with correct filesize.
Error writing trailer of rtmp://xxxxx
I have try:

--disable-librtmp but no effect, the problem cause again.

ffmpeg version:
================================================================================
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.12) 20160609
configuration: --prefix=/usr/sourcecode/ffmpegout --pkg-config-flags=--static -- extra-cflags=-I/usr/sourcecode/ffmpegout/include --extra-ldflags=-L/usr/sourceco de/ffmpegout/lib --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfr eetype --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-nonfre e --enable-shared --enable-postproc --enable-zlib --enable-libxvid --enable-libo pencore-amrnb --enable-libopencore-amrwb --enable-version3 --enable-libx264 --di sable-debug
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100

[terry]

while enable-librtmp:
WriteN, RTMP send error 32 (129 bytes)
RTMP_SendPacket: fd=5, size=47

0000: 43 00 00 00 00 00 2f 14 C...../.
0000: 02 00 0b 46 43 55 6e 70 75 62 6c 69 73 68 00 40 ...FCUnpublish.@
0010: 1c 00 00 00 00 00 00 05 02 00 14 31 33 35 35 35 ...........13555
0020: 30 30 30 30 30 33 5f 31 5f 30 5f 31 30 37 38 000003_1_0_1078

WriteN, RTMP send error 32 (55 bytes)
RTMP_SendPacket: fd=-1, size=34

0000: 43 00 00 00 00 00 22 14 C.....".
0000: 02 00 0c 64 65 6c 65 74 65 53 74 72 65 61 6d 00 ...deleteStream.
0010: 3f f0 00 00 00 00 00 00 05 00 3f f0 00 00 00 00 ?.........?.....
0020: 00 00 ..

WriteN, RTMP send error 9 (42 bytes)
[rtmp @ 0x128d740] rtmp_write:-1