FFmpeg Libraries Seek Wildly On Opening W64-Format Files Via VIO

[John Fitzgerald]

I am testing the next patchset for the MOC player which adds the ability to feed FFmpeg's libav* libraries input via the callbacks registered using avio_alloc_context(). Some of those tests involve a variety of formats and codecs.

When attempting to play a W64-format file, the application becomes unresponsive and GDB shows seeking patterns well outside the bounds of the files through repeated calls to the seek and read callbacks.

As this is a library problem, there is no command line invocation of an FFmpeg utility, and those utilities do not appear to use callbacks via avio_alloc_context().

There is no console output (obviously) and no messages are issued by the libraries (via the logging callback).

The sequence of library calls (omitting error checking) is:

AVFormatContext *ic = avformat_alloc_context();
ic->pb = avio_alloc_context(...);
avformat_open_input(ic, ...);

avformat_open_input() never returns.

The problem occurs on a W64-format file of any size, but a suitable test file can be generated with:

sox -b16 -c2 -r48000 -n -L test.wav synth 10 sine 440 vol 0.5

The problem was introduced in commit 14d50c19 in FFmpeg 1.1 and still exists in FFmpeg 3.0.2.

It is possible that the error return values of the seek and read callbacks are incorrect, but they appear undocumented and do work in all other formats tested and also with the W64 format prior to commit 14d50c19 and with LibAV 11.6. The return values used by similar code found vary.

Full FFmpeg configure originally used is:

CFLAGS="-O2 -fPIC" ./configure --prefix=/usr/local/stow/$(basename $PWD) --libdir='${prefix}/lib64' --shlibdir='${prefix}/lib64' --enable-shared --disable-static --disable-yasm --enable-libspeex --enable-libtheora --enable-libopus

'$PWD' was '/tmp/ffmpeg-14d50c19'.

MOC is available using:

svn co -r2872 ​svn://svn.daper.net/moc/trunk

Aggregated patch for the patchset being tested is available using:

wget ​ftp://ftp.daper.net/pub/soft/moc/unstable/moc-r2872+ffmpeg_memleak-1.patch.gz

Apply to the SVN trunk revision 2872 using:

zcat moc-r2872+ffmpeg_memleak-1.patch.gz | patch -p1

The MOC FFmpeg decoder will be found in:

decoder_plugins/ffmpeg

Build as per instructions in README. After installing, delete all installed decoder plugins in 'lib{,64}/moc/decoder_plugins' *except* 'libffmpeg_decoder.*'.

Then launch MOC as 'mocp' in the directory containing W64 files.

[Carl Eugen Hoyos]

Is this issue reproducible with current FFmpeg git head?

[John Fitzgerald]

Replying to cehoyos:

Is this issue reproducible with current FFmpeg git head?

Yes - e8a236ad.

[Michael Niedermayer]

Which *seek calls in the ffmpeg source seek wildly?

[John Fitzgerald]

If you use GDB to set a breakpoint on ffmpeg_io_seek_cb() in MOC's FFmpeg decoder you will be able to backtrace into the FFmpeg libraries to identify the seeks when 'offset' goes into the petabyte range.

[Michael Niedermayer]

i get "FATAL_ERROR: Can't send() int to the server!"

without the callback being called
no doubt thats a mistake on my side, but i dont have the time to figure out atm what i did wrong

[John Fitzgerald]

Replying to michael:

"FATAL_ERROR: Can't send() int to the server!"

That indicates that the server side of MOC has failed for some reason. The reason may appear in your system's log files, or you can run the server separately from the client using:

mocp -D -SF .

in one VT or xterm to start the server in non-daemon mode and then just:

mocp

in a second VT or xterm to start the client.

When that message appears you can check the server's output for more information.

Note that you will have to have configured MOC without '--disable-debug' (and '--enable-debug=gdb' is best when using GDB) for this to work.

[John Fitzgerald]

I have created a test rig for you which is available at:

​ftp://ftp.daper.net/pub/soft/moc/unstable/seek-test.c.gz

This test rig emulates the essential behaviour of MOC and triggers the problem on *.w64 files (but not on others).

Please read the comments at the start of the file on compiling and running, and especially on the definition of ATEOF.

In creating this test rig, I uncovered a potential bug in MOC which causes it to return a non-zero value at EOF, but this does not trigger the problem on non-W64 files although forcing the return value to zero causes FFmpeg to handle W64 files correctly (or at least circumvents the seeking problem).

[John Fitzgerald]

I have now circumvented this problem within MOC, although it and the documentation issue still exist within FFmpeg. However, for my purposes you could now lower the priority of this problem if you wish.

[Michael Niedermayer]

I dont see any huge seek offsets:

sox -b16 -c2 -r48000 -n -L test.wav synth 10 sine 440 vol 0.5
md5sum test.wav
23429ea98c131c67ba8b160fa5fea002 test.wav
./seek-test test.wav
Calling ffmpeg_open():

Calling avformat_alloc_context()
Calling avio_alloc_context()
Calling avformat_open_input()

read_cb: 2048

read(): 2048

seek_cb called: 0, 65536

file_size(): 1920044

seek_cb called: 0, 65536

file_size(): 1920044

seek_cb called: 0, 65536

file_size(): 1920044

seek_cb called: 0, 65536

file_size(): 1920044

seek_cb called: 0, 65536

file_size(): 1920044

read_cb: 63532

read(): 63532

seek_cb called: 44, 0

lseek(): 44

Calling ffmpeg_close():

Calling avformat_close_input()

[Michael Niedermayer]

I see this:
==17514== Conditional jump or move depends on uninitialised value(s)
==17514== at 0x478278: main (seek-test.c:134)
but thats not related

[John Fitzgerald]

I have now commited the latest MOC patchset which includes a mitigation to this problem which limits FFmpeg to a single out-of-bounds seek (which MOC can tolerate). I will leave it in your hands to pursue it or close it as you wish.

[Elon Musk]

How is this related to w64 if you are creating wav files with sox?

[Elon Musk]

w64 demuxer parses complete file if file is seekable to find metadata.
Looks like your program implementation of avio callbacks is fragile.
It should not keep seeking or reading if eof is reached.