Streaming some MXF files off S3 takes very long to prepare

[Gavin]

Summary of the bug:

I'm using ffplay to stream some MXF files off S3 (via https) for simple preview. I have a sufficient broadband connection to stream these XDCAMHD 422 (50Mbps) files off S3.

I note that when I try to stream some of the larger MXF files (this one is ~57GiB), FFplay is taking well over a minute to commence rendering.

Interestingly, streaming the same file using GStreamer takes considerably less time - a few seconds to commence rendering.

How to reproduce:

% ffplay "https://s3.eu-west-1.amazonaws.com/test-media.dev.coralbay.tv/Testi60M/2951385_black_box.mxf?AWSAccessKeyId=AKIAJPFO54NSWS2T3CWA&Signature=xSN5XyUULme9xOy%2BOv%2BzjGDE8g0%3D&Expires=1581328167"
ffplay started on 2019-12-12 at 09:50:31
Report written to "ffplay-20191212-095031.log"
Log level: 48
ffplay version git-2019-12-08-9f7b2b3 Copyright (c) 2003-2019 the FFmpeg developers
  built with gcc 9.2.1 (GCC) 20191125
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 36.101 / 56. 36.101
  libavcodec     58. 64.101 / 58. 64.101
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.101 / 58.  9.101
  libavfilter     7. 68.100 /  7. 68.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100

Additional Information:

Having generated a report (-report), I notice that the demuxer/IO looks to seek all the way to the end and start reading backwards. Here is a small snippet from the enclosed report/log file showing the seek:

Range: bytes=0-
Range: bytes=61964236619-
Range: bytes=61964225536-
Range: bytes=0-
Range: bytes=61964220928-
Range: bytes=61899003904-
Range: bytes=61831648768-
Range: bytes=61764293632-
Range: bytes=61696938496-
Range: bytes=61629583360-
Range: bytes=61562228224-
Range: bytes=61494873088-
Range: bytes=61427517952-
Range: bytes=61360162816-
Range: bytes=61292807680-
Range: bytes=61225452544-
Range: bytes=61158097408-

It looks like the same behaviour is seen when reading from local filesystem too, but it is far less noticeable when reading off my SSD.

If run the following to disable seeking...

ffplay -seekable 0 "https://s3.eu-west-1.amazonaws.com/test-media.dev.coralbay.tv/Testi60M/2951385_black_box.mxf?AWSAccessKeyId=AKIAJPFO54NSWS2T3CWA&Signature=xSN5XyUULme9xOy%2BOv%2BzjGDE8g0%3D&Expires=1581328167"

the file commences rendering as fast as GStreamer

I am very aware that devs like sample files to be kept short, but I have not been able to reproduce this with many other files.

[Gavin]

FFmpeg report

[Gavin]

FFmpeg console log

[Tomas Härdin]

This is because mxfdec.c parses all partition headers in the file before demuxing. This is done backwards since there is no "NextPartition" value in the partition headers. This is necessary to figure out the BodySID of each partition, which is necessary for seeking to work. See mxf_read_seek(), mxf_edit_unit_absolute_offset() and mxf_absolute_bodysid_offset().

If you do not need seeking you can mark the file as not seekable and it'll work just fine, as you have noticed. You could configure your web server to refuse seeking on the files to make this work automagically for everyone else.

Potential solutions:

The file has a RandomIndexPack (RIP) of size 11087, with 921 entries. This means 921+ seeks to parse all partitions, and at least as many allocations. It should be possible to do this lazily, especially when a file has a RIP. You still run into an almost impossible problem if a user wants to seek into the middle of a file though, and I would rather not make the code even more complicated to try and handle that.

[Tomas Härdin]

Another solution would be to use just one large BodyPartition, and write the index to a FooterPartition, and a RIP after that.

[Gavin]

Thanks for the thorough explanation Tjoppen. I'm not too familiar with the MXF spec, so your analysis was helpful.

Closing and marking as 'enhancement' but 'wontfix'. Please adjust as seen fit.

[Tomas Härdin]

Replying to zerodefect:

Thanks for the thorough explanation Tjoppen. I'm not too familiar with the MXF spec, so your analysis was helpful.

Closing and marking as 'enhancement' but 'wontfix'. Please adjust as seen fit.

I'll just add that I'm not opposed to fixing this if it can be done in some not-tearing-ones-hair-out way. Usually with MXF one writes a demuxer for one's specific use case, using say bmxlib.