mingw-w64 clang i686 "inline assembly requires more registers than available"

[Christopher Degawa]

Summary of the bug:

On windows with a mingw-w64 clang targeting i686 obtained through msys2's pacman by installing the "mingw32/mingw-w64-i686-clang" version 10.0.1-1 package, trying to compile "libavcodec/hevc_cabac.o" will result in

CC      libavcodec/hevc_cabac.o
D:/mabs/build/test-ffmpeg/libavcodec/hevc_cabac.c:37:21: warning: variable 'num_bins_in_se' is not needed and will not be emitted [-Wunneeded-internal-declaration]
static const int8_t num_bins_in_se[] = {
                    ^
In file included from D:/mabs/build/test-ffmpeg/libavcodec/hevc_cabac.c:27:
In file included from D:/mabs/build/test-ffmpeg/libavcodec/cabac_functions.h:46:
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
        BRANCHLESS_GET_CABAC("%0", "%q0", "(%4)", "%1", "%w1",
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:193:9: error: inline assembly requires more registers than available
D:/mabs/build/test-ffmpeg/libavcodec/x86/cabac.h:143:9: note: expanded from macro 'BRANCHLESS_GET_CABAC'
        "movzbl "statep"    , "ret"                                     \n\t"\
        ^
1 warning and 11 errors generated.
make: *** [/build/test-ffmpeg/ffbuild/common.mak:67: libavcodec/hevc_cabac.o] Error 1

the error message itself looks pretty much exactly like ​https://trac.ffmpeg.org/ticket/7796, but since that one seems to specifically be tracking android x86 and I haven't gotten a response on if I should append to that or create a new issue, I decided to make new issue.

The commit I have tested this on is ​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/e71d73b09652f4fc96e512a7d6d4c2ab41860f27

head part of configure output

install prefix            /usr/local
source path               /home/user/test-ffmpeg
C compiler                clang
C library                 mingw64
ARCH                      x86 (generic)
big-endian                no
runtime cpu detection     yes
standalone assembly       yes
x86 assembler             nasm
MMX enabled               yes
MMXEXT enabled            yes
3DNow! enabled            yes
3DNow! extended enabled   yes
SSE enabled               yes
SSSE3 enabled             yes
AESNI enabled             yes
AVX enabled               yes
AVX2 enabled              yes
AVX-512 enabled           yes
XOP enabled               yes
FMA3 enabled              yes
FMA4 enabled              yes
i686 features enabled     yes
CMOV is fast              no
EBX available             yes
EBP available             yes
debug symbols             yes
strip symbols             yes
optimize for size         no
optimizations             yes
static                    yes
shared                    no
postprocessing support    no
network support           yes
threading support         w32threads
safe bitstream reader     yes
texi2html enabled         no
perl enabled              yes
pod2man enabled           yes
makeinfo enabled          yes
makeinfo supports HTML    yes

I can attach the ffbuild/config.log if requested

How to reproduce:

1. on a Windows machine or VM, download msys2, then run inside msys2.exe

   pacman -Syu --noconfirm
   pacman -Suu --noconfirm
   # clang and its deps are quite huge already, 2.5 GiB installed size as reported by pacman
   pacman -S --needed --noconfirm mingw-w64-i686-clang git base-devel
   

2. Then, run the following inside mingw32.exe

   git clone --branch master https://github.com/FFmpeg/FFmpeg.git test-ffmpeg
   mkdir test-ffmpeg/clang-build && cd test-ffmpeg/clang-build
   ../configure --cc="clang" --cxx="clang++"
   make libavcodec/hevc_cabac.o
   

the clang line from make V=1

clang -I. -I/home/cddeg/test-ffmpeg/ -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -U__STRICT_ANSI__ -D__USE_MINGW_ANSI_STDIO=1 -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DZLIB_CONST -DHAVE_AV_CONFIG_H -DBUILDING_avcodec -std=c11 -fomit-frame-pointer -I/mingw32/include -g -Wdeclaration-after-statement -Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wwrite-strings -Wtype-limits -Wundef -Wmissing-prototypes -Wno-pointer-to-int-cast -Wstrict-prototypes -Wempty-body -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign -Wno-unused-const-variable -Wno-bool-operation -Wno-char-subscripts -O3 -fno-math-errno -fno-signed-zeros -Qunused-arguments -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type   -MMD -MF libavcodec/hevc_cabac.d -MT libavcodec/hevc_cabac.o -c -o libavcodec/hevc_cabac.o /home/cddeg/test-ffmpeg/libavcodec/hevc_cabac.c

I am not sure how to try to reproduce this on a Linux machine with a clang cross compiler as I do not have any experience doing cross-compilation and trying to use mstorsjo/llvm-mingw resulted in a failure on my part to understand how to use it, and it doesn't seem like there's an easy to follow guide for cross-compiling ffmpeg with clang.

Attempting to use the post-processed source file using -E -P with both 32-bit gcc and clang results in gcc compiling successfully, but clang failing.
I've tried extracting the "get_cabac_inline_x86" function and the necessary components needed to compile it, along with making it non-static and non-inline and compiling just that function with clang and gcc, but in the isolated function case, both clang and gcc compile the source file without error and I see the assembly in the disassembly of the object file, which makes me guess it might be related to inlining, etc, but I am not a compiler expert.

[Christopher Degawa]

Applying ​https://gist.github.com/1480c1/86830546ed555b50f91a023100e526a9 seems to allow it to compile

[Christopher Degawa]

demoting to minor since I doubt clang + 32-bit + windows is a common enough configuration