Error submitting a packet to the muxer: Broken pipe, Error muxing a packet

[Rob]

Summary of the bug:

I am streaming a video (mp4) to a RTMP destination and hitting this error. The RTMP destination is receiving the data and creating HLS TS files from it - using nginx RTMP module.

Once this error it hit, FFMPEG keeps going, but the server stops creating TS files at that point.

[aost#0:1/copy @ 0x135ab40] Error submitting a packet to the muxer: Broken pipe/s speed=1.05x
[out#0/flv @ 0x1359700] Error muxing a packet
[flv @ 0x13c96c0] Failed to update header with correct duration.ate= 17.9kbits/s speed=1.04x
[flv @ 0x13c96c0] Failed to update header with correct filesize.
[out#0/flv @ 0x1359700] Error writing trailer: Broken pipe
[out#0/flv @ 0x1359700] Error closing file: Broken pipe

How to reproduce:

ffmpeg -re -i low.mp4 -map 0 -c copy -f flv rtmp://127.0.0.1:1935/<my stream key>

ffmpeg version N-112061-g654e4b00e2 Copyright (c) 2000-2023 the FFmpeg developers

I have attached a ZIP of the file I am using to reproduce it.

[Rob]

Hi,

I would like to attach the video file that I can reproduce this issue but the size limit is 2.5 MB and my video (lowest resolution) is 40MB. Is there any way I can attach a file bigger than 2.5MB?

[Rob]

Here is the file I use to reproduce please let me know if you have trouble downloading it:

​https://www.dropbox.com/scl/fi/5ybita2l41tr7tyhzgy25/low.mp4?rlkey=gzahw6wp6soytuyc2185zs4ef&dl=0

[Rob]

It looks like this is a similar issue when this command is run:

ffmpeg -stream_loop -1 -re -i low2.mp4 -map 0 -c copy -f flv rtmp://127.0.0.1:1935/<stream key>

This file plays fine once, but when it repeats (start over) it gives the broken pipe and causes a problem.

av_interleaved_write_frame(): Broken pipesize= 71202kB time=00:05:10.50 bitrate=1878.6kbits/s speed= 1x
[out#0/flv @ 0x7fecbe7124c0] Error muxing a packet

Here is the file to test with:

​https://www.dropbox.com/scl/fi/0ark8tz3d9p0qjqgtuk8w/low2.mp4?rlkey=bvkirv59rtxut0933kkc0rueg&dl=0

[Steven Liu]

Please provide the command line you tested together with the complete, uncut console output to make this a valid question.

[Steven Liu]

(base) adeMacBook-Pro-2:ufbuild liuqi$ ./ffmpeg -i ~/kuaishou/Test/low.mp4 -c copy -t 10 out.flv
ffmpeg(82080,0x7ff85bdb6640) malloc: nano zone abandoned due to inability to reserve vm space.
ffmpeg version N-112061-g654e4b00e2 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 14.0.3 (clang-1403.0.22.14.1)
  configuration: --prefix=/usr/local/ --libdir=/usr/local/lib/ --cc=clang --enable-htmlpages --enable-libx264 --enable-gpl --extra-cflags='-I/usr/local/include/GraphicsMagick/ -I/usr/local/opt/openssl@3.1/include/ -O0 -g3 -fsanitize=address -Wno-error -fPIC -I/usr/local/include -fno-omit-frame-pointer' --extra-ldflags='-O0 -g3 -fsanitize=address -fno-omit-frame-pointer -Wno-error -fPIC -L/usr/local/lib -L/usr/local/opt/openssl@3.1/lib -lpng' --enable-libfreetype --enable-fontconfig --enable-libspeex --enable-libopus --enable-libzmq --enable-libx265 --enable-libass --enable-videotoolbox --disable-optimizations --enable-audiotoolbox --enable-opengl --disable-stripping --samples=../../fate-suite/ --enable-encoder=hevc_videotoolbox --enable-hwaccel=hevc_videotoolbox --enable-hwaccel=h264_videotoolbox --enable-openssl --enable-nonfree --disable-shared --enable-libfdk_aac --enable-libxml2 --enable-libmp3lame --enable-lcms2 --enable-libwebp --enable-libvpx --enable-libaom
  libavutil      58. 24.100 / 58. 24.100
  libavcodec     60. 26.100 / 60. 26.100
  libavformat    60. 12.100 / 60. 12.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9. 11.100 /  9. 11.100
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/liuqi/kuaishou/Test/low.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.27.100
  Duration: 00:08:44.33, start: 0.000000, bitrate: 677 kb/s
  Stream #0:0[0x1](und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 384x216 [SAR 1:1 DAR 16:9], 349 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.37.100 h264_nvenc
      timecode        : 01:00:00:00
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 319 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      handler_name    : VideoHandler
      timecode        : 01:00:00:00
Output #0, flv, to 'out.flv':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.12.100
  Stream #0:0(und): Video: h264 (Constrained Baseline) ([7][0][0][0] / 0x0007), yuv420p(tv, bt709, progressive), 384x216 [SAR 1:1 DAR 16:9], q=2-31, 349 kb/s, 30 fps, 30 tbr, 1k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.37.100 h264_nvenc
      timecode        : 01:00:00:00
  Stream #0:1(und): Audio: aac (LC) ([10][0][0][0] / 0x000A), 48000 Hz, stereo, fltp, 319 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[out#0/flv @ 0x611000003100] video:431kB audio:392kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.729315%
frame=  303 fps=0.0 q=-1.0 Lsize=     837kB time=00:00:09.98 bitrate= 687.2kbits/s speed= 991x

write flv ok.

(base) adeMacBook-Pro-2:ufbuild liuqi$ ./ffmpeg -re -i ~/kuaishou/Test/low.mp4 -c copy -f flv rtmp://127.0.0.1/stream/live
ffmpeg(82623,0x7ff85bdb6640) malloc: nano zone abandoned due to inability to reserve vm space.
ffmpeg version N-112061-g654e4b00e2 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 14.0.3 (clang-1403.0.22.14.1)
  configuration: --prefix=/usr/local/ --libdir=/usr/local/lib/ --cc=clang --enable-htmlpages --enable-libx264 --enable-gpl --extra-cflags='-I/usr/local/include/GraphicsMagick/ -I/usr/local/opt/openssl@3.1/include/ -O0 -g3 -fsanitize=address -Wno-error -fPIC -I/usr/local/include -fno-omit-frame-pointer' --extra-ldflags='-O0 -g3 -fsanitize=address -fno-omit-frame-pointer -Wno-error -fPIC -L/usr/local/lib -L/usr/local/opt/openssl@3.1/lib -lpng' --enable-libfreetype --enable-fontconfig --enable-libspeex --enable-libopus --enable-libzmq --enable-libx265 --enable-libass --enable-videotoolbox --disable-optimizations --enable-audiotoolbox --enable-opengl --disable-stripping --samples=../../fate-suite/ --enable-encoder=hevc_videotoolbox --enable-hwaccel=hevc_videotoolbox --enable-hwaccel=h264_videotoolbox --enable-openssl --enable-nonfree --disable-shared --enable-libfdk_aac --enable-libxml2 --enable-libmp3lame --enable-lcms2 --enable-libwebp --enable-libvpx --enable-libaom
  libavutil      58. 24.100 / 58. 24.100
  libavcodec     60. 26.100 / 60. 26.100
  libavformat    60. 12.100 / 60. 12.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9. 11.100 /  9. 11.100
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/liuqi/kuaishou/Test/low.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.27.100
  Duration: 00:08:44.33, start: 0.000000, bitrate: 677 kb/s
  Stream #0:0[0x1](und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 384x216 [SAR 1:1 DAR 16:9], 349 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.37.100 h264_nvenc
      timecode        : 01:00:00:00
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 319 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      handler_name    : VideoHandler
      timecode        : 01:00:00:00
Output #0, flv, to 'rtmp://127.0.0.1/stream/live':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.12.100
  Stream #0:0(und): Video: h264 (Constrained Baseline) ([7][0][0][0] / 0x0007), yuv420p(tv, bt709, progressive), 384x216 [SAR 1:1 DAR 16:9], q=2-31, 349 kb/s, 30 fps, 30 tbr, 1k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.37.100 h264_nvenc
      timecode        : 01:00:00:00
  Stream #0:1(und): Audio: aac (LC) ([10][0][0][0] / 0x000A), 48000 Hz, stereo, fltp, 319 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  738 fps= 31 q=-1.0 size=    2008kB time=00:00:24.50 bitrate= 671.3kbits/s speed=1.02x

push rtmp ok

(base) adeMacBook-Pro-2:ufbuild liuqi$ ./ffmpeg -i rtmp://127.0.0.1/stream/live
ffmpeg(82694,0x7ff85bdb6640) malloc: nano zone abandoned due to inability to reserve vm space.
ffmpeg version N-112061-g654e4b00e2 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 14.0.3 (clang-1403.0.22.14.1)
  configuration: --prefix=/usr/local/ --libdir=/usr/local/lib/ --cc=clang --enable-htmlpages --enable-libx264 --enable-gpl --extra-cflags='-I/usr/local/include/GraphicsMagick/ -I/usr/local/opt/openssl@3.1/include/ -O0 -g3 -fsanitize=address -Wno-error -fPIC -I/usr/local/include -fno-omit-frame-pointer' --extra-ldflags='-O0 -g3 -fsanitize=address -fno-omit-frame-pointer -Wno-error -fPIC -L/usr/local/lib -L/usr/local/opt/openssl@3.1/lib -lpng' --enable-libfreetype --enable-fontconfig --enable-libspeex --enable-libopus --enable-libzmq --enable-libx265 --enable-libass --enable-videotoolbox --disable-optimizations --enable-audiotoolbox --enable-opengl --disable-stripping --samples=../../fate-suite/ --enable-encoder=hevc_videotoolbox --enable-hwaccel=hevc_videotoolbox --enable-hwaccel=h264_videotoolbox --enable-openssl --enable-nonfree --disable-shared --enable-libfdk_aac --enable-libxml2 --enable-libmp3lame --enable-lcms2 --enable-libwebp --enable-libvpx --enable-libaom
  libavutil      58. 24.100 / 58. 24.100
  libavcodec     60. 26.100 / 60. 26.100
  libavformat    60. 12.100 / 60. 12.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9. 11.100 /  9. 11.100
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
Input #0, flv, from 'rtmp://127.0.0.1/stream/live':
  Metadata:
    |RtmpSampleAccess: true
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.12.100
    server          : SRS/6.0.66(Bee)
    server_version  : 6.0.66
  Duration: N/A, start: 0.000000, bitrate: 669 kb/s
  Stream #0:0: Data: none
  Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 319 kb/s
  Stream #0:2: Video: h264 (Constrained Baseline), yuv420p(tv, bt709, progressive), 384x216 [SAR 1:1 DAR 16:9], 349 kb/s, 30 fps, 30 tbr, 1k tbn
At least one output file must be specified
(base) adeMacBook-Pro-2:ufbuild liuqi$

get stream info from rtmp sever ok.

[Steven Liu]

[2023-09-18 10:09:14.122][INFO][82422][5ht538q7] Process: cpu=0.00%,0MB, threads=2
[2023-09-18 10:09:17.907][INFO][82422][w829w4s4] Hybrid cpu=0.00%,0MB, cid=1,0, timer=62,0,0, clock=0,48,1,0,0,0,0,0,0
[2023-09-18 10:09:19.124][INFO][82422][5ht538q7] Process: cpu=0.00%,0MB, threads=2
[2023-09-18 10:09:21.239][INFO][82422][l8s714l8] RTMP client ip=127.0.0.1:55821, fd=14
[2023-09-18 10:09:21.243][INFO][82422][l8s714l8] complex handshake success
[2023-09-18 10:09:21.243][INFO][82422][l8s714l8] connect app, tcUrl=rtmp://127.0.0.1:1935/stream, pageUrl=, swfUrl=, schema=rtmp, vhost=127.0.0.1, port=1935, app=stream, args=null
[2023-09-18 10:09:21.244][INFO][82422][l8s714l8] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
[2023-09-18 10:09:21.244][INFO][82422][l8s714l8] client identified, type=fmle-publish, vhost=127.0.0.1, app=stream, stream=live, param=, duration=0ms
[2023-09-18 10:09:21.245][INFO][82422][l8s714l8] connected stream, tcUrl=rtmp://127.0.0.1:1935/stream, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1935, app=stream, stream=live, param=, args=null
[2023-09-18 10:09:21.245][INFO][82422][l8s714l8] source url=/stream/live, ip=127.0.0.1, cache=1/2500, is_edge=0, source_id=/0519o9uu
[2023-09-18 10:09:21.246][INFO][82422][l8s714l8] ignore disabled exec for vhost=__defaultVhost__
[2023-09-18 10:09:21.247][INFO][82422][l8s714l8] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0
[2023-09-18 10:09:21.247][INFO][82422][l8s714l8] got metadata, width=384, height=216, vcodec=7, acodec=10
[2023-09-18 10:09:21.250][INFO][82422][l8s714l8] 55B video sh, codec(7, profile=Baseline, level=1.3, 384x216, 0kbps, 0.0fps, 0.0s)
[2023-09-18 10:09:21.250][INFO][82422][l8s714l8] 4B audio sh, codec(10, profile=LC, 2channels, 0kbps, 48000HZ), flv(16bits, 2channels, 44100HZ)
[2023-09-18 10:09:21.251][INFO][82422][l8s714l8] cleanup when unpublish
[2023-09-18 10:09:21.251][INFO][82422][l8s714l8] TCP: before dispose resource(RtmpConn)(0x61200005fbc0), conns=1, zombies=0, ign=0, inz=0, ind=0
[2023-09-18 10:09:21.253][ERROR][82422][l8s714l8][4] serve error code=2003(Amf0Decode)(Decode AMF0 message failed) : service cycle : rtmp: stream service : rtmp: receive thread : handle publish message : rtmp: consume message : rtmp: decode message : decode message : decode command name : String invalid marker=0
thread [82422][l8s714l8]: do_cycle() [./src/app/srs_app_rtmp_conn.cpp:263][errno=4]
thread [82422][l8s714l8]: service_cycle() [./src/app/srs_app_rtmp_conn.cpp:457][errno=4]
thread [82422][l8s714l8]: do_publishing() [./src/app/srs_app_rtmp_conn.cpp:1031][errno=35]
thread [82422][l8s714l8]: consume() [./src/app/srs_app_recv_thread.cpp:380][errno=35]
thread [82422][l8s714l8]: handle_publish_message() [./src/app/srs_app_rtmp_conn.cpp:1186][errno=35]
thread [82422][l8s714l8]: process_publish_message() [./src/app/srs_app_rtmp_conn.cpp:1231][errno=35]
thread [82422][l8s714l8]: decode_message() [./src/protocol/srs_protocol_rtmp_stack.cpp:384][errno=35]
thread [82422][l8s714l8]: do_decode_message() [./src/protocol/srs_protocol_rtmp_stack.cpp:607][errno=35]
thread [82422][l8s714l8]: srs_amf0_read_string() [./src/protocol/srs_protocol_amf0.cpp:1513][errno=35](Interrupted system call)
[2023-09-18 10:09:21.255][INFO][82422][70a0507r] TCP: clear zombies=1 resources, conns=1, removing=0, unsubs=0
[2023-09-18 10:09:21.255][INFO][82422][l8s714l8] TCP: disposing #0 resource(RtmpConn)(0x61200005fbc0), conns=1, disposing=1, zombies=0

I look at the rtmp server log about the error mesage as your command

./ffmpeg -re -i ~/kuaishou/Test/low.mp4 -map 0 -c copy -f flv rtmp://127.0.0.1/stream/live

.

Maybe the problem is happened in rtmp server side, or you can try command

./ffmpeg -re -i ~/kuaishou/Test/low.mp4 -map 0:a -map 0:v -c copy -f flv rtmp://127.0.0.1/stream/live

[Steven Liu]

Looks the problem is happened when it publish timecode stream:

  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      handler_name    : VideoHandler
      timecode        : 01:00:00:00

[Rob]

Hi Steven,

It's late here and I will follow up with your requested information tomorrow (if you need any from me). What I did find out earlier is when I send to the NGINX RTMP module it fails but when I sent to Wowza it does not seem to fail, as you suggest it might be on in the NGINX RTMP module.

This is the one we are using:

​https://github.com/winshining/nginx-http-flv-module

When you say above that the problem happened were you able to reproduce the issue or are you not able to reproduce it to your RTMP server?

[winlin]

This is caused by timecode of FFmpeg, SRS RTMP server fixed by ​https://github.com/ossrs/srs/issues/3803

FFmpeg sends AMF0 data, but it doesn't begin with the AMF0 string:

Packet #72:
04 00 00 64 00 00 04 12 01 00 00 00

Packet #74:
00 01 a5 e0

(lldb) p *msg
(SrsCommonMessage) $0 = {
  header = {
    timestamp_delta = 100
    payload_length = 4
    message_type = '\x12' 18 '\x12'
    stream_id = 1
    timestamp = 100
    perfer_cid = 4
  }
  size = 4
  payload = 0x0000602000003570 ""
}
(lldb) x/4xb msg->payload
0x602000003570: 0x00 0x01 0xa5 0xe0

RTMP server should be able to handle this message by simply ignoring it:

[2023-09-18 11:38:45.099][WARN][91322][148t53jz][35] Ignore FFmpeg timecode, data=[00 01 a5 e0]

Highly recommend to use SRS v5.0.179+ or v6.0.77+ to replace Nginx RTMP, or to forward stream to Nginx RTMP.

However, there is a workaround for this issue, to remove the -map 0 then it works well:

ffmpeg -stream_loop -1 -re -i low2.mp4 -c copy -f flv rtmp://127.0.0.1:1935/<stream key>

Or you can patch your NGINX RTMP server, refer to ​https://github.com/ossrs/srs/pull/3804

[Rob]

Hi,

Thank you for taking the time to look into this.

In my example, I gave you the most simplified command line to reproduce it and kept the -map 0 even though I did not need it as I had a single input. However, my actual command line (for our application) has N inputs with N outputs so I need to use the -map N command to properly tie the inputs to the output (so I will either need to patch RTMP NGINX or use SRS to forward).

Have a great day.

[Rob]

So this does appear to be a bug in ffmpeg, and the RMTP servers (ossrs and nginx rtmp) put in workarounds. From the developer of the nginx rtmp:

FFmpeg sends a buggy AMF0 structure when -map option is used: the AMF type is 0x00, it means it is a number and the following 8 bytes are used to represent a number, but there are only 3 bytes left.

[Rob]

Hi Steven,

I have follow up question for you and since you originally replied it appears that you have a great understanding of this (much better than I do).

I am experiencing another issue related to this and I am hoping you can shed some light on it.

Both nginx RTMP and SRS developers put in workarounds on the server to now blow up when this condition is hit:

(from the developer)

"FFmpeg sends a buggy AMF0 structure when -map option is used: the AMF type is 0x00, it means it is a number and the following 8 bytes are used to represent a number, but there are only 3 bytes left."

However, it seems like after this condition is hit ffmpeg stops sending some information that the server determines if the client is still alive.

Both nginx RTMP and SRS have a "drop idle publisher" option:

drop_idle_publisher 5s;

We set this so publishers that legitimately stop will be dropped.

However, in the case of FFMPEG is it still sending RTMP data but the server thinks it's dead and will drop it. This is what the nginx rtmp developer said:

"Debug logs in nginx's error.log showed that FFmpeg indeed didn't send any messages in 5s after malformed AMF was sent."

Do you have any insight as to why the "map" function stops sending the data that the server needs to determine that the client is still connected.

Even though the server developers put in workarounds for the bug in ffmpeg, it seems that there is an underlying issue that happens after that bug condition is met.

Thanks for your time.

-Rob

[Steven Liu]

No, it's not ScriptData, It just DATA if you using map option, it's only timecode data. There have Data channel for Data in flv muxer.

libavformat/flvenc.c

 685         case AVMEDIA_TYPE_DATA:
 686             if (par->codec_id != AV_CODEC_ID_TEXT && par->codec_id != AV_CODEC_ID_NONE)
 687                 return unsupported_codec(s, "Data", par->codec_id);
 688             flv->data_par = par;
 689             break;