ffmpeg crashes on win64 when decoding a dirac stream

[jamal]

Summary: ffmpeg (mingw64 x86_64 GCC 4.7.0, yasm 1.2.0) crashes when trying to decode a dirac stream.
This however doesn't happen if i compile ffmpeg with the --disable-yasm option.
ffprobe and even ffmpeg with no output parameters run fine when reading the file and wont crash.

The command line i used was:

./ffmpeg -i bbb-tr1000.ts -vcodec zlib -an asd.mkv

And this is the sample i used: ​http://dirac.kw.bbc.co.uk/download/video/maybefinal/bbb-tr1000.ts

I'm attaching both the ffmpeg log file and gdb's output.
gdb mentions ff_add_rect_clamped_sse2 as the source of the problem, which might explain why disabling yasm got rid of the crash.

[jamal]

gdb output

[jamal]

ffmpeg log

[Carl Eugen Hoyos]

Does it also crash with --disable-sse ?

[jamal]

No, it doesn't crash when configuring with --disable-sse.

Also, in case it's of any use, decoding the same file with a mingw32 x86 build (gcc 4.6.2) works fine, with or without yasm or sse.

[jamal]

Ok, just tried this on Gentoo x86_64 GCC 4.5.3 and it crashed as well, so i suppose this is an issue with x86_64 in general.

I'm attaching the gdb output since it's not the same as the one from the mingw64 build. It points to the general decode function instead of that sse2 one.

[jamal]

gdb output, gentoo x64

[Carl Eugen Hoyos]

Is the crash on gentoo also reproducible with --disable-asm --disable-optimizations --disable-yasm?
Does the backtrace look different?

[jamal]

It doesn't crash with either --disable-asm --disable-optimizations --disable-yasm or --disable-sse.

The backtrace is almost the same. While the one from mingw64 starts with the ff_add_rect_clamped_sse2 function followed by dirac_decode_frame_internal, the one from linux gentoo starts with the later.

[Michael Niedermayer]

might be fixed, please re-test

[jamal]

Ok, mixed results.

I tested the fix and it ran without issues on Linux x64. However, when i reverted the changes of the proposed fix it still ran without issues.
To make sure i went back and compiled the commit i originally tested two weeks ago which again crashed as expected. Then, applying the fix to that old commit fixed the crash for it.
I checked what commit in between the fix and the one i tested two weeks ago could have generated this, and found it was d5a7229b, "Add a float DSP framework to libavutil".

On Win64 however it didn't work. The backtrace is the same as the old one.