swscale/yuv2yuvX yasm port broke non-yasm builds

[Ben Boeckel]

VTK's AVI writer started having a strange behavior when writing out AVI files where the result would have stripes which change colors "scan in" over and over. Attached is the resulting file with this behavior. The relevant FFmpeg code in VTK can be see here:

​https://gitlab.kitware.com/vtk/vtk/-/blob/master/IO/FFMPEG/vtkFFMPEGWriter.cxx

Reverting the following commits resolves the problem when applied to 4.4, but not master (more debugging is needed there):

c00567647e9002094255df755e18c719e75b3333 "swscale/x86/swscale: fix mix of inline and external function definitions"
c2bf1dcace8d331e672b955f9cf5b59211749f00 "swscale/x86/swscale: fix compilation with old yasm"
1371647fc381774d40b8beecf69604331262172d "checkasm/sw_scale: use av_free() instead of free()"
d52ceed9fd3655113657d9b572fc2c83b7f913cc "tests/checkasm/sw_scale: use memset() to fill dither"
ee18edb13a9ae3041df961dd5003c2055b5cab35 "checkasm/sw_scale: properly initialize src_pixer and filter_coeff buffers"
6bc2058d00b119d265c9970eac213d2922d15129 "tests/checkasm/sw_scale: adds additional tests sizes for yux2yuvX"
c23a5523b5c5bc63d206cd18a669875034e62ae5 "swscale/x86/swscale: Remove unused ASM constants"
95aacf30e3803c57d91ff62975b375e394f61d49 "libswscale/x86/swscale: Only call ff_yuv2yuvX functions if the input size is > 0"
d512ebbaedefc977d7628da4dc3ecfe4e52a066a "swscale/x86/yuv2yuvX: use the SPLATW helper macro"
ebb48d85a07551ca82965bb66b88e50606683b4d "swscale/x86/yuv2yuvX: use movq to load 8 bytes in all non-AVX2 functions"
1a555d3c604804dcedacd230d410cfc822da3f4c "swscale/x86/yuv2yuvX: use the movsxdifnidn helper macro"
4aeedf4c2a8f35be667d5dd40c84bd27730ef1db "libswscale/x86/yuv2yuvX: Removes unrolling for mmx and mmxext"
554c2bc7086f49ef5a6a989ad6bc4bc11807eb6f "swscale: move yuv2yuvX_sse3 to yasm, unrolls main loop"

FFmpeg is configured with:

--disable-x86asm --prefix=… --enable-shared --disable-static

to test this behavior.

[Ben Boeckel]

Video written out with the striping

[Ben Boeckel]

Video written out as intended

[Ben Boeckel]

This problem still persists with 6.0 and master commit 9f66286f0bdd42fe388a7366909c11954fd330a6.

[Ben Boeckel]

Additionally reverts needed:

51a34e8525fea2bbc29b42831d7a17f34e8518d3 sws: Replace call to yuv2yuvX_mmx by yuv2yuvX_mmxext
a05f22eaf393177b94432431c145cbc5ba10390a swscale/x86/swscale: Remove obsolete and harmful MMX(EXT) functions
bbe95f7353a972f28a48be8da883549f02c59e4b x86: replace explicit REP_RETs with RETs

There are more conflicts now, so some manual fixups were needed, but I'm able to get a patchset which works on 6.0 now.