Opened 4 years ago

Last modified 22 months ago

#7191 open enhancement

Allow to set number of cpu cores for --enable-lto

Reported by: Artem S. Tashkinov Owned by:
Priority: wish Component: build system
Version: git-master Keywords: lto
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Currently all ffmpeg libraries are linked sequentially which makes the whole process extremely slow.

Change History (10)

comment:1 by Carl Eugen Hoyos, 4 years ago

Keywords: lto added
Priority: importantwish
Type: defectenhancement

Parallel linking of libraries works fine here, please elaborate.

comment:2 by Artem S. Tashkinov, 4 years ago

Are you absolutely sure about that?

Because when I see:

make -j4
...
STRIP	libswresample/x86/audio_convert.o
GEN	libavdevice/libavdevice.ver
GEN	libavfilter/libavfilter.ver
GEN	libswscale/libswscale.ver
GEN	libavutil/libavutil.ver
GEN	libpostproc/libpostproc.ver
GEN	libavformat/libavformat.ver
GEN	libswresample/libswresample.ver
LD	libavutil/libavutil.so.56
LD	libswscale/libswscale.so.5
LD	libpostproc/libpostproc.so.55
LD	libswresample/libswresample.so.3
STRIP	libavcodec/x86/vp9itxfm.o
GEN	libavcodec/libavcodec.ver
LD	libavcodec/libavcodec.so.58
LD	libavdevice/libavdevice.so.58
LD	ffmpeg_g
LD	ffplay_g
LD	ffprobe_g

All the libraries are linked sequentially, i.e. just one CPU core is active.

comment:3 by Timo R., 4 years ago

Most of the libraries depend on each other, so they can only be linked when their dependencies are already done linking.

comment:4 by Carl Eugen Hoyos, 4 years ago

I tested the following and see a speed impact if I increase the number of threads for make:

$ rm -f libpostproc/libpostproc.so* libswscale/libswscale.so* libswresample/libswresample.so* && time make -j1 libpostproc/libpostproc.so libswscale/libswscale.so libswresample/libswresample.so

As explained, some libraries depend on others.
Did you ever test the speed impact of lto on the final FFmpeg binaries?
If you really need faster build time, search for LDFLAGS in ffbuild/config.mak and change -flto to -flto=4. Not sure how easy this is to allow in configure.

comment:5 by Carl Eugen Hoyos, 4 years ago

Summary: Please enable parallel linking when --enable-lto is specifiedAllow to set number of cpu cores for --enable-lto

in reply to:  4 ; comment:6 by Artem S. Tashkinov, 4 years ago

Replying to cehoyos:

If you really need faster build time, search for LDFLAGS in ffbuild/config.mak and change -flto to -flto=4. Not sure how easy this is to allow in configure.

This sounds like a great idea - thanks but I still hope that Makefile could be altered, so that 'make -jN' also made linking with -flto faster by default.

in reply to:  6 comment:7 by Carl Eugen Hoyos, 4 years ago

Replying to birdie:

Replying to cehoyos:

If you really need faster build time, search for LDFLAGS in ffbuild/config.mak and change -flto to -flto=4. Not sure how easy this is to allow in configure.

This sounds like a great idea - thanks but I still hope that Makefile could be altered, so that 'make -jN' also made linking with -flto faster by default.

Please understand that this is simply not possible, sorry for not closing this ticket as invalid, I have been flamed too often;-(

comment:8 by Carl Eugen Hoyos, 4 years ago

Btw, did you ignore the most important part of my post?

Did you ever test the speed impact of lto on the final FFmpeg binaries?

comment:9 by Artem S. Tashkinov, 4 years ago

Nope, never.

From what I've read and understood -flto is a safe [optimization] option even though it might give you zero gains in regard to performance which is why I prefer to enable it by default. Also I've indeed tested other pieces of software and usually -flto nets a small performance boost.

Of course, if you're a developer you'd want to disable it because it makes building significantly slower (and million times slower if you e.g. alter just one source file).

comment:10 by Balling, 22 months ago

Status: newopen

and million times slower if you e.g. alter just one source file

Excuse me, what??? Are you sure? How does it even connect to linking? I think this should be done though.

Last edited 22 months ago by Balling (previous) (diff)
Note: See TracTickets for help on using tickets.