Altivec compilation broken by --enable-small

[ernsteiswuerfel]

Summary of the bug:

Building ffmpeg 3.4.2 with gcc 7.3.0 + altivec .... FAILS TO BUILD
Building ffmpeg 3.4.2 with gcc 7.3.0 - altivec .... BUILDS OK
Building ffmpeg 3.4.2 with gcc 6.4.0 + altivec .... BUILDS OK
Building ffmpeg 3.4.2 with gcc 6.4.0 - altivec .... BUILDS OK

I tried building ffmpeg 3.4.2 on my PowerMac G5 with gcc 7.3.0 and altivec support enabled. The build stops giving me this error:

powerpc-unknown-linux-gnu-gcc -I. -Isrc/ -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DPIC -DZLIB_CONST -DHAVE_AV_CONFIG_H -Os -pipe -mcpu=G5 -mtune=G5 -mcpu=970 -std=c11 -fomit-frame-pointer -fPIC -maltivec -mabi=altivec -pthread -I/usr/include/freetype2 -I/usr/include/opus -I/usr/include/opus -D_REENTRANT -pthread -I/usr/include/librsvg-2.0 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libdrm -I/usr/include/libpng16 -Wdeclaration-after-statement -Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wwrite-strings -Wtype-limits -Wundef -Wmissing-prototypes -Wno-pointer-to-int-cast -Wstrict-prototypes -Wempty-body -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign -Wno-unused-const-variable -Wno-bool-operation -Os -pipe -mcpu=G5 -mtune=G5 -fno-math-errno -fno-signed-zeros -fno-tree-vectorize -Werror=format-security -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type -Werror=vla -Wformat -fdiagnostics-color=auto -Wno-maybe-uninitialized -MMD -MF libavcodec/ppc/hevcdsp.d -MT libavcodec/ppc/hevcdsp.o -c -o libavcodec/ppc/hevcdsp.o src/libavcodec/ppc/hevcdsp.c
In file included from src/libavutil/ppc/util_altivec.h:32:0,

from src/libavcodec/ppc/hevcdsp.c:27:

src/libavcodec/ppc/hevcdsp.c: In function ‘scale’:
src/libavcodec/ppc/hevcdsp.c:73:23: error: argument 1 must be a 5-bit signed literal

vec_u32 v_shift = vec_splat_u32(shift);

src/libavcodec/ppc/hevcdsp.c: In function ‘transform4x4.isra.0’:
src/libavcodec/ppc/hevcdsp.c:60:9: error: argument 1 must be a 5-bit signed literal

add = vec_sl(vec_splat_s32(1), vec_splat_u32(shift - 1));
^~

make: libavcodec/ppc/hevcdsp.o Error 1

Building with altivec support disabled succeeds (gcc 7.3.0).

How to reproduce:

Build ffmpeg 3.4.2 on ppc hardware with gcc 7.3.0 and altivec support enabled.

[ernsteiswuerfel]

build.log

[ernsteiswuerfel]

build environment

[Carl Eugen Hoyos]

Please test current FFmpeg git head.

[Carl Eugen Hoyos]

And please test a sane configure line like ./configure --enable-gpl.

[ernsteiswuerfel]

Tried as you suggested. So far I was not able to reproduce the bug from git master with ./configure --enable-gpl, nor with ./configure --enable-gpl --disable-runtime-cpudetect --cpu=G5 (similar options like the Gentoo build).

[Carl Eugen Hoyos]

Unrelated: --disable-runtime-cpudetect does not do what you/Gentoo expect, you cannot disable the call to the cpu detection function.

I assume this is the the same issue as Gentoo bugs 578802 and 645778: Perhaps you can find out if completely removing --optflags from the build has a disadvantage (after confirming this is also your issue)?

I wonder if it would be an advantage for you if configure would detect the broken custom compilation flags and disable AltiVec...

[ernsteiswuerfel]

I did expect --disable-runtime-cpudetec to do what the config options tell, which is to create a smaller binary. Probably that's what Gentoo thinks too. ;)

I don't think it's necessary to change configure to deal with custom compilations flags, as altivec is not among the default flags in Gentoo ppc/ppc64. If you build ffmpeg-3.4.2 without altivec it does just fine (even with gcc-7.3.0).

I will look into the Gentoo bugs as you suggested and report back in a few days.

[Carl Eugen Hoyos]

Please test if removing --opflags from your configure line fixes the issue for you.

The next question will be if --enable-small works with your toolchain.

[Carl Eugen Hoyos]

I can confirm the issue with the following simplified configure line and gcc 4.8.5, gcc 6.3.1, gcc 7.2.0 and gcc 7.2.1, I was unable to find a gcc version that does not allow to reproduce:

$ ./configure --enable-small && make libavcodec/ppc/hevcdsp.o

Regression since f6e8d54f

[ernsteiswuerfel]

Thanks for your work! Sadly I had no time yet to do the builds on the G5s Gentoo (other issues)...

[A. Wilcox]

This patch has been tested by me in the following environments:

(All environments running Adélie Linux, kernel 4.14.48-mc8-easy, musl libc 1.1.19, GCC 6.4.0)

• 32-bit PowerPC 7447 "G4" (Apple PowerBook G4 1.25 GHz)
• 64-bit PowerPC 970fx "G5" (Apple Power Mac G5 1.8 GHz DP)
• 64-bit PowerPC POWER9 "Sforza" (Raptor Talos II 3.8 GHz)

All built and passed fate. I additionally ran the example clip of Big Buck Bunny from x265.org in ffplay to great success.

[Carl Eugen Hoyos]

Should this be av_always_inline?

[A. Wilcox]

Indeed, I am sorry, I did not know about this. I'm uploading a new patch using that now.

[A. Wilcox]

Fix HEVC AltiVec routines

[Carl Eugen Hoyos]

Applied as 153fcd6de6ba558a3720c64589a7e4e9e4d62420, thank you!

For future patches: Please remember that tabs cannot be committed to the FFmpeg repository.