Opened 5 months ago

Last modified 2 weeks ago

#7547 new defect

When relaying RTMP streams ffmpeg quits with av_interleaved_write_frame message

Reported by: regstuff Owned by:
Priority: important Component: avformat
Version: git-master Keywords: regression librtmp
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug: I have an Nginx server (with rtmp-module) on a Ubuntu-16.04 VM. I'm using it to receive an input rtmp stream from my PC. FFmpeg is then used in the VM to take this input and relay to multiple destinations after transcoding, overlaying a lower third, and changing the volume.

However, FFmpeg quits after a random duration between 3-30 minutes with an "av_interleaved_write_frame(): End of file" message.

How to reproduce: Pull an rtmp stream as input. Command used was as below. In the below command, I have used aac and strict -2 as well. I am aware that strict -2 is no longer necessary. The error occurs even with other aac encoders such as libfdk_aac and without strict -2. The reason I have used this particular command is explained in more detail below.

/usr/local/bin/ffmpeg -v 9 -loglevel 99 -nostdin -thread_queue_size 512 -i rtmp://127.0.0.1:1935/input/stream1 -i /usr/local/nginx/scripts/images/lowerthird.png -af azmq,volume=2 -c:a aac -filter_complex 'zmq=bind_address=tcp\\\://127.0.0.1\\\:5559,overlay=0:H' -vcodec libx264 -pix_fmt yuv420p -preset veryfast -r 25 -g 50 -b:v 6000k -maxrate 6M -minrate 6M -bufsize 6M -f flv -strict -2 -f flv rtmp://127.0.0.1:1935/distribute/stream1 -y -report

This happens with the latest ffmpeg version (details below). However, I do not see this issue when I use the standard FFmpeg that comes with Ubuntu 16.04's apt-get install (ffmpeg version 2.8.15). To do an aac encode with the Ubuntu standard Ffmpeg, I have to use aac and strict -2. This is why I have used the command above with the latest Ffmpeg version as well, just to standardize the comparsion. Just to reiterate, this issue occurs even when I use a different encoder and without the -strict -2 flags with the latest ffmpeg.

ffmpeg version

ffmpeg version N-92427-gae43235 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --extra-libs='-lpthread -lm' --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzmq --enable-librtmp --enable-network --enable-nonfree
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 43.100 /  7. 43.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100

Have attached log output with this ticket. The issue occurs even if I remove all encoding and just do a codec copy as below.

/usr/local/bin/ffmpeg -nostdin -thread_queue_size 512 -i rtmp://127.0.0.1:1935/stream1/input -c copy -f flv rtmp://a.rtmp.youtube.com/live2/[my key] -y  

Issue occurs irrespective of destination, whether youtube, facebook or just another location on the same nginx server.

Issue also occurs whether thread_queue_size is 8 or 1024 or 512. It appears that this issue is more frequent in a 4-core system rather than a single core system I was using. Not totally sure about that though.

Attachments (3)

ffmpeg-20181114-092056.log (824.0 KB) - added by regstuff 5 months ago.
Ffmpeg Log File
issue7547.mp4 (2.2 MB) - added by ruddell 3 months ago.
Sample for reproduction
7547.pcap (2.0 MB) - added by ruddell 7 weeks ago.
output stream pcap

Change History (24)

Changed 5 months ago by regstuff

Ffmpeg Log File

comment:1 Changed 3 months ago by lbgaus

I was having the same problem as regstuff, and found that I was able to resolve my problem with 'av_interleaved_write_frame(): End of file' errors being thrown after about 2 to 3 hours of publishing to an RTMP server by compiling off the 3.4.5 branch, instead of the master. Something must be wrong with the latest version causing this issue?

Last edited 3 months ago by lbgaus (previous) (diff)

comment:2 Changed 3 months ago by cehoyos

  • Component changed from ffmpeg to undetermined
  • Keywords regression added
  • Version changed from unspecified to git-master

Please use git bisect to find the change introducing the regression.

comment:3 Changed 3 months ago by ikolesnikov

  • Priority changed from normal to important

I have the same problem while transcoding udp multicast to rtmp via the following command:

ffmpeg -y -hwaccel cuvid -c:v mpeg2_cuvid -resize 1280x720 -deint 2 -drop_second_field 1 -i "udp://235.1.10.1:1234?overrun_nonfatal=1" -aspect 16:9 -vcodec h264_nvenc -b:v 1500k -acodec libfdk_aac -f flv rtmp://127.0.0.1/stream/out

This issue occurs for the newest snapshot from Github. Also, I got the same issue for the 4.1 release version. The same command works well with the 3.4 release version. Unfortunately, we can't use the new features implemented in FFmpeg due to this bug. The problem with RTMP muxing is important for many users

comment:4 Changed 3 months ago by cehoyos

  • Priority changed from important to normal

If the ticket were important you would have used git bisect to point us to the offending commit.

Changed 3 months ago by ruddell

Sample for reproduction

comment:5 Changed 3 months ago by ruddell

I have this problem on Ubuntu 16.04.5 and Mac with v4.1 when using librtmp. It works with v3.4.5 on Ubuntu as described above.

I attached a file that shows the issue with the following command (about 4 seconds in):

ffmpeg -re -i issue7547.mp4 -vcodec copy -acodec copy -bsf:a aac_adtstoasc -f flv 'rtmp://a.rtmp.youtube.com/live2/stream-key-here'

The first commit where streaming hangs, git bisect shows the following:

commit 858db4b01fa2b55ee55056c033054ca54ac9b0fd
Author: Daniel Kucera <daniel.kucera@gmail.com>
Date:   Tue Oct 17 10:29:30 2017 +0200

    libavformat: not treat 0 as EOF
    
    transfer_func variable passed to retry_transfer_wrapper
    are h->prot->url_read and h->prot->url_write functions.
    These need to return EOF or other error properly.
    In case of returning >= 0, url_read/url_write is retried
    until error is returned.
    
    Signed-off-by: Daniel Kucera <daniel.kucera@gmail.com>

:040000 040000 03f37e2b4749f5f1d21a9a528c0a10eef8ff2cf1 6c2edb3a01134a03e79534f6cbb048abb59121d2 M	libavformat

The first commit where the error message is correctly displayed, git bisect shows the following:

commit ed647ab79f9a54d8d3a8e345f6a1643b60b849f4
Author: Timo Rothenpieler <timo@rothenpieler.org>
Date:   Thu Jul 26 00:37:35 2018 +0200

    avformat/librtmp: fix returning EOF from Read/Write
    
    Ticket #7052

:040000 040000 964b972c7b3df952e7b634e4a76cdefaa8da3622 200f7038acf21315bee42874f69ae41d6c31315d M	libavformat
Last edited 2 months ago by ruddell (previous) (diff)

comment:6 Changed 3 months ago by ikolesnikov

  • Component changed from undetermined to avformat

comment:7 Changed 3 months ago by ikolesnikov

  • Priority changed from normal to important

comment:8 Changed 3 months ago by cehoyos

  • Keywords librtmp added

comment:9 Changed 3 months ago by pgalow

I can confirm we are running into the same issue on macOS (10.13.6 and 10.14.3) using 4.1 (installed via Homebrew). So far no issues after downgrading to 3.4.5.

comment:10 Changed 2 months ago by danman

Guys, can someone of you make a tcpdump of the output RTMP stream?

comment:11 Changed 2 months ago by oromit

Try reverting the first hunk of ed647ab79f9a54d8d3a8e345f6a1643b60b849f4.
The one that changes RTMP_Write. Only RTMP_Read seems to actually do EOF.

comment:12 Changed 2 months ago by oromit

On another note: Try just not using librtmp. The ffmpeg native rtmp protocol works fine for me.

comment:13 Changed 2 months ago by ruddell

@oromit Reverting the first hunk of that commit causes it to hang at the failing point rather than exit with an error message. The issue first appeared in the other commit in my previous message.

You are correct that using ffmpeg's rtmp instead of librtmp works fine, which explains why I couldn't reproduce on Mac earlier. I updated my comment to note that.

@danman I tried but I'm not familiar with tcpdump. I attached a file and command that reproduces it though.

Last edited 2 months ago by ruddell (previous) (diff)

comment:14 Changed 2 months ago by oromit

I'd say it's time to get rid of the librtmp support.
I see the same issues here, and the easy fix is to not use librtmp.

comment:15 Changed 2 months ago by danman

@ruddell : I don't see the attachment but my random idea is to replace:

if (!ret)

with:

if (!ret && size)

or debug the value of "size" in http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=ed647ab79f9a54d8d3a8e345f6a1643b60b849f4

comment:16 Changed 2 months ago by ruddell

@danman Please check the issue's attachments at the top. And the command (copied from my first message): ffmpeg -re -i issue7547.mp4 -vcodec copy -acodec copy -bsf:a aac_adtstoasc -f flv 'rtmp://a.rtmp.youtube.com/live2/stream-key-here'

I tested if (!ret && size) but it did not cause any difference. The value of size is 3 at the failing point.

...
Write Size: 361 
Write Size: 55 
Write Size: 32768 
Write Size: 3 
Write Size: 3 

I also tried if (!ret && size > 3) but it infinitely loops with Write Size: 3 (which is what happens before ed647ab79f9a54d8d3a8e345f6a1643b60b849f4)

Again, the issue occurs because of the other commit 858db4b01fa2b55ee55056c033054ca54ac9b0fd in my previous message. The ed647ab79f9a54d8d3a8e345f6a1643b60b849f4 commit simply makes FFMPEG print an error message instead of hanging on RTMP_Write with Size 3.

Thanks for looking into this.

Last edited 2 months ago by ruddell (previous) (diff)

comment:17 Changed 8 weeks ago by danman

@ruddell the input file is not enough. I'd like to see the output stream produced. You should be able to capture it with:
tcpdump -i any port 1935 -w 7547.pcap

Last edited 8 weeks ago by danman (previous) (diff)

Changed 7 weeks ago by ruddell

output stream pcap

comment:18 Changed 7 weeks ago by ruddell

Thank you for the help with the command. I have attached a pcap file to this issue.

comment:19 Changed 6 weeks ago by regstuff

To add to the previous comments, I can confirm that the latest version of ffmpeg works fine if I dont use librtmp. Just did a 16 hour test with no issues.

comment:20 follow-up: Changed 5 weeks ago by ikolesnikov

I can confirm that --disable-librtmp at the ./configure step resolved this issue

Last edited 4 weeks ago by ikolesnikov (previous) (diff)

comment:21 in reply to: ↑ 20 Changed 2 weeks ago by tomwilson

Replying to ikolesnikov:

I can confirm that --disable-librtmp at the ./configure step resolved this issue

This worked for me when I had the same issue

ffmpeg 4.1 with librtmp + Ubuntu 16.04 streaming to nginx-rtmp = error described in the original post
ffmpeg 4.1 with librtmp disabled is fine

Note: See TracTickets for help on using tickets.