Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#8426 closed enhancement (wontfix)

Streaming some MXF files off S3 takes very long to prepare

Reported by: Gavin Owned by:
Priority: normal Component: avformat
Version: git-master Keywords: mxf
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:

I'm using ffplay to stream some MXF files off S3 (via https) for simple preview. I have a sufficient broadband connection to stream these XDCAMHD 422 (50Mbps) files off S3.

I note that when I try to stream some of the larger MXF files (this one is ~57GiB), FFplay is taking well over a minute to commence rendering.

Interestingly, streaming the same file using GStreamer takes considerably less time - a few seconds to commence rendering.

How to reproduce:

% ffplay "https://s3.eu-west-1.amazonaws.com/test-media.dev.coralbay.tv/Testi60M/2951385_black_box.mxf?AWSAccessKeyId=AKIAJPFO54NSWS2T3CWA&Signature=xSN5XyUULme9xOy%2BOv%2BzjGDE8g0%3D&Expires=1581328167"
ffplay started on 2019-12-12 at 09:50:31
Report written to "ffplay-20191212-095031.log"
Log level: 48
ffplay version git-2019-12-08-9f7b2b3 Copyright (c) 2003-2019 the FFmpeg developers
  built with gcc 9.2.1 (GCC) 20191125
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 36.101 / 56. 36.101
  libavcodec     58. 64.101 / 58. 64.101
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.101 / 58.  9.101
  libavfilter     7. 68.100 /  7. 68.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100

Additional Information:

Having generated a report (-report), I notice that the demuxer/IO looks to seek all the way to the end and start reading backwards. Here is a small snippet from the enclosed report/log file showing the seek:

Range: bytes=0-
Range: bytes=61964236619-
Range: bytes=61964225536-
Range: bytes=0-
Range: bytes=61964220928-
Range: bytes=61899003904-
Range: bytes=61831648768-
Range: bytes=61764293632-
Range: bytes=61696938496-
Range: bytes=61629583360-
Range: bytes=61562228224-
Range: bytes=61494873088-
Range: bytes=61427517952-
Range: bytes=61360162816-
Range: bytes=61292807680-
Range: bytes=61225452544-
Range: bytes=61158097408-

It looks like the same behaviour is seen when reading from local filesystem too, but it is far less noticeable when reading off my SSD.

If run the following to disable seeking...

ffplay -seekable 0 "https://s3.eu-west-1.amazonaws.com/test-media.dev.coralbay.tv/Testi60M/2951385_black_box.mxf?AWSAccessKeyId=AKIAJPFO54NSWS2T3CWA&Signature=xSN5XyUULme9xOy%2BOv%2BzjGDE8g0%3D&Expires=1581328167"

the file commences rendering as fast as GStreamer

I am very aware that devs like sample files to be kept short, but I have not been able to reproduce this with many other files.

Attachments (2)

ffplay-20191212-095108.log (1.5 MB ) - added by Gavin 4 years ago.
FFmpeg report
ffmpeg_console_log.txt (268.9 KB ) - added by Gavin 4 years ago.
FFmpeg console log

Download all attachments as: .zip

Change History (7)

by Gavin, 4 years ago

Attachment: ffplay-20191212-095108.log added

FFmpeg report

by Gavin, 4 years ago

Attachment: ffmpeg_console_log.txt added

FFmpeg console log

comment:1 by Tomas Härdin, 4 years ago

This is because mxfdec.c parses all partition headers in the file before demuxing. This is done backwards since there is no "NextPartition" value in the partition headers. This is necessary to figure out the BodySID of each partition, which is necessary for seeking to work. See mxf_read_seek(), mxf_edit_unit_absolute_offset() and mxf_absolute_bodysid_offset().

If you do not need seeking you can mark the file as not seekable and it'll work just fine, as you have noticed. You could configure your web server to refuse seeking on the files to make this work automagically for everyone else.

Potential solutions:

The file has a RandomIndexPack (RIP) of size 11087, with 921 entries. This means 921+ seeks to parse all partitions, and at least as many allocations. It should be possible to do this lazily, especially when a file has a RIP. You still run into an almost impossible problem if a user wants to seek into the middle of a file though, and I would rather not make the code even more complicated to try and handle that.

comment:2 by Tomas Härdin, 4 years ago

Another solution would be to use just one large BodyPartition, and write the index to a FooterPartition, and a RIP after that.

comment:3 by Gavin, 4 years ago

Resolution: wontfix
Status: newclosed
Type: defectenhancement

Thanks for the thorough explanation Tjoppen. I'm not too familiar with the MXF spec, so your analysis was helpful.

Closing and marking as 'enhancement' but 'wontfix'. Please adjust as seen fit.

in reply to:  3 comment:4 by Tomas Härdin, 4 years ago

Replying to zerodefect:

Thanks for the thorough explanation Tjoppen. I'm not too familiar with the MXF spec, so your analysis was helpful.

Closing and marking as 'enhancement' but 'wontfix'. Please adjust as seen fit.

I'll just add that I'm not opposed to fixing this if it can be done in some not-tearing-ones-hair-out way. Usually with MXF one writes a demuxer for one's specific use case, using say bmxlib.

comment:5 by Carl Eugen Hoyos, 4 years ago

Keywords: seekable removed
Note: See TracTickets for help on using tickets.