checkasm float_dsp.vector_dmul sometimes fails on i386

[James Cowgill]

Sometimes (about 1 in 10 times), the float_dsp.vector_dmul checkasm test fails on i386.

Example:

$ tests/checkasm/checkasm --test=float_dsp 2642491962
checkasm: using random seed 2642491962
SSE:
 - float_dsp.vector_fmul         [OK]
 - float_dsp.vector_fmac         [OK]
 - float_dsp.butterflies_float   [OK]
 - float_dsp.scalarproduct_float [OK]
93: -53.395181798898 - -53.395181798898 = -7.1054273576e-15
SSE2:
   vector_dmul_scalar_sse2 (float_dsp.c:171)
 - float_dsp.vector_dmul         [FAILED]
 - float_dsp.vector_dmac         [OK]

The failures are reprodicible given the same random seed. Note you probably need a toolchain which uses the x87 FPU to reproduce this (eg Fedora's gcc will not work).

I think this happens because the SSE routines compute the multiplication to double precision, but the reference C routines compute the multiplcation to 80-bit precision. Sometimes the results are different.

I noticed this patch which I think is an attempt to solve this. While it does reduce the probability of failure, I can still get the test to fail when run > 1000 times.
​https://patchwork.ffmpeg.org/patch/5743/

Downstream bug: ​https://bugs.debian.org/882075

[Carl Eugen Hoyos]

Fixed by Michael in 38f966b2222db4bfeeaca4642a63049253536c46