Endianness detection breaks when LTO is enabled through means other than the ffmpeg --enable-lto option

[RedDwarf82]

ffmpeg's configure wrongly thinks my toolchain uses little-endian because of LTO.

$ cat test.c
unsigned int endian = 'B' << 24 | 'I' << 16 | 'G' << 8 | 'E';
$ gcc -c -o test.o test.c 
$ od -t x1 test.o | grep '42 *49 *47 *45'
$ gcc -c -o test.o test.c -fno-lto
$ od -t x1 test.o | grep '42 *49 *47 *45'
0000160 42 49 47 45 00 47 43 43 3a 20 28 42 75 69 6c 64

[RedDwarf82]

To be clear, I show that code because that's what configure does -> ​https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/c2a221c5ae5017e11654b9688ac97e9f5d3570b2:/configure#l5594

[Hendrik]

If you intend to build ffmpeg with LTO enabled, you should do so by using the --enable-lto switch, which will only enable LTO after such tests are being performed, and not enable it behind FFmpegs back.

[Carl Eugen Hoyos]

I cannot reproduce: --enable-lto does not break big-endian detection on different systems with different compilers I tested.
Please test current FFmpeg git head (nothing else is supported here) and tell us about your configure line, compiler and operating system (gcc -v, uname -a).

[RedDwarf82]

It's a buildroot gcc build. I can see "-flto" was added to BR2_TARGET_OPTIMIZATION, which is why it's the default.

I can workaround my specific case. And I don't expect lots of people (but probably more than 1) to have gcc with LTO enabled by default. So feel free to close the issue if you want to.

But at the same time it seems to me that depending on people not enabling it "behind FFmpegs back" is error-prone.
I would say at the very least the check in ​https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/c2a221c5ae5017e11654b9688ac97e9f5d3570b2:/configure#l5596 should also check if it can find "45 *47 *49 *42". If it can't find either configure should fail.

[RedDwarf82]

Reopening since I really don't think this is a "needs_more_info" case. heleppkes has clearly described the issue. Also because doesn't even require the compiler to use LTO by default, so this may be a more common issue than I suggested before. Probably it's not more reported only because the more common desktop architecture happens to be little endian.

But here you have the extra information requested:

$ git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
Cloning into 'ffmpeg'...
remote: Counting objects: 563150, done.
remote: Compressing objects: 100% (118600/118600), done.
remote: Total 563150 (delta 450571), reused 554797 (delta 443403)
Receiving objects: 100% (563150/563150), 133.44 MiB | 6.03 MiB/s, done.
Resolving deltas: 100% (450571/450571), done.
$ cd ffmpeg/
$ git rev-parse HEAD
0321370601833f4ae47e8e11c44570ea4bd382a4
$ mkdir build_normal build_lto
$ cd build_normal/
$ ../configure --enable-cross-compile --cross-prefix=mips-linux- --target-os=linux --arch=mips --cpu=74kc --disable-all
install prefix            /usr/local
source path               /opt/ffmpeg
C compiler                mips-linux-gcc
C library                 uclibc
host C compiler           gcc
host C library            glibc
ARCH                      mips (74kc)
big-endian                yes
runtime cpu detection     yes
MIPS FPU enabled          no
MIPS DSP R1 enabled       yes
MIPS DSP R2 enabled       yes
MIPS MSA enabled          no
LOONGSON MMI enabled      no
debug symbols             yes
strip symbols             yes
optimize for size         no
optimizations             yes
static                    yes
shared                    no
postprocessing support    no
network support           no
threading support         pthreads
safe bitstream reader     yes
texi2html enabled         no
perl enabled              yes
pod2man enabled           yes
makeinfo enabled          no
makeinfo supports HTML    no

External libraries:
alsa			  bzlib			    zlib

External libraries providing hardware acceleration:

Libraries:
avutil

Programs:

Enabled decoders:

Enabled encoders:

Enabled hwaccels:

Enabled parsers:

Enabled demuxers:

Enabled muxers:

Enabled protocols:

Enabled filters:

Enabled bsfs:

Enabled indevs:

Enabled outdevs:

License: LGPL version 2.1 or later

WARNING: mips-linux-pkg-config not found, library detection may fail.
$ cd ../build_lto
$ CFLAGS='-flto' ../configure --enable-cross-compile --cross-prefix=mips-linux- --target-os=linux --arch=mips --cpu=74kc --disable-all --enable-lto
install prefix            /usr/local
source path               /opt/ffmpeg
C compiler                mips-linux-gcc
C library                 uclibc
host C compiler           gcc
host C library            glibc
ARCH                      mips (74kc)
big-endian                no
runtime cpu detection     yes
MIPS FPU enabled          no
MIPS DSP R1 enabled       yes
MIPS DSP R2 enabled       yes
MIPS MSA enabled          no
LOONGSON MMI enabled      yes
debug symbols             yes
strip symbols             yes
optimize for size         no
optimizations             yes
static                    yes
shared                    no
postprocessing support    no
network support           no
threading support         pthreads
safe bitstream reader     yes
texi2html enabled         no
perl enabled              yes
pod2man enabled           yes
makeinfo enabled          no
makeinfo supports HTML    no

External libraries:
alsa			  bzlib			    zlib

External libraries providing hardware acceleration:

Libraries:
avutil

Programs:

Enabled decoders:

Enabled encoders:

Enabled hwaccels:

Enabled parsers:

Enabled demuxers:

Enabled muxers:

Enabled protocols:

Enabled filters:

Enabled bsfs:

Enabled indevs:

Enabled outdevs:

License: LGPL version 2.1 or later

WARNING: mips-linux-pkg-config not found, library detection may fail.
$ uname -a
Linux 7d04a811e585 4.20.13-200.fc29.x86_64 #1 SMP Wed Feb 27 19:42:55 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ mips-linux-gcc -v
Using built-in specs.
COLLECT_GCC=/opt/buildroot_wdr3600/output/host/bin/mips-linux-gcc.br_real
COLLECT_LTO_WRAPPER=/opt/buildroot_wdr3600/output/host/libexec/gcc/mips-buildroot-linux-uclibc/7.4.0/lto-wrapper
Target: mips-buildroot-linux-uclibc
Configured with: ./configure --prefix=/opt/buildroot_wdr3600/output/host --sysconfdir=/opt/buildroot_wdr3600/output/host/etc --enable-static --disable-libstdcxx-dual-abi --disable-symvers --target=mips-buildroot-linux-uclibc --with-sysroot=/opt/buildroot_wdr3600/output/host/mips-buildroot-linux-uclibc/sysroot --enable-__cxa_atexit --with-gnu-ld --disable-libssp --disable-multilib --disable-decimal-float --with-gmp=/opt/buildroot_wdr3600/output/host --with-mpc=/opt/buildroot_wdr3600/output/host --with-mpfr=/opt/buildroot_wdr3600/output/host --with-pkgversion='Buildroot 2019.02-00010-g07502cdd25-dirty' --with-bugurl=http://bugs.buildroot.net/ --disable-libquadmath --disable-libsanitizer --enable-tls --enable-plugins --enable-lto --disable-libmudflap --enable-threads --without-isl --without-cloog --with-float=soft --with-arch=mips32r2 --with-abi=32 --with-nan=legacy --enable-languages=c,c++ --with-build-time-tools=/opt/buildroot_wdr3600/output/host/mips-buildroot-linux-uclibc/bin --enable-shared --disable-libgomp
Thread model: posix
gcc version 7.4.0 (Buildroot 2019.02-00010-g07502cdd25-dirty) 
$

I really don't think it's OK for ffmpeg to just say "it's the user's fault for using CFLAGS='-flto'" ("enable LTO behind FFmpegs back"). At the very least the configure scripts needs to notice the situation and report failure.

Notice the "LOONGSON MMI" detection is also broken when using -flto.

[Carl Eugen Hoyos]

Replying to RedDwarf82:

$ CFLAGS='-flto' ../configure --enable-cross-compile --cross-prefix=mips-linux- --target-os=linux --arch=mips --cpu=74kc --disable-all --enable-lto

Thank you for the explanation!
CFLAGS (and --extra-cflags) can make the configure line invalid, in general this cannot be detected so it makes little sense to detect it in some exceptional cases.

[RedDwarf]

FWIW nowadays LTO can have different options. Just from gcc/clang:

-flto
-flto=auto
-flto=thin
-flto=4

And by not letting the user specify LTO in the flags he is losing access to them.