Opened 6 months ago

Last modified 6 months ago

#7191 new enhancement

Allow to set number of cpu cores for --enable-lto

Reported by: birdie Owned by:
Priority: wish Component: build system
Version: git-master Keywords: lto
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Currently all ffmpeg libraries are linked sequentially which makes the whole process extremely slow.

Change History (9)

comment:1 Changed 6 months ago by cehoyos

  • Keywords lto added
  • Priority changed from important to wish
  • Type changed from defect to enhancement

Parallel linking of libraries works fine here, please elaborate.

comment:2 Changed 6 months ago by birdie

Are you absolutely sure about that?

Because when I see:

make -j4
...
STRIP	libswresample/x86/audio_convert.o
GEN	libavdevice/libavdevice.ver
GEN	libavfilter/libavfilter.ver
GEN	libswscale/libswscale.ver
GEN	libavutil/libavutil.ver
GEN	libpostproc/libpostproc.ver
GEN	libavformat/libavformat.ver
GEN	libswresample/libswresample.ver
LD	libavutil/libavutil.so.56
LD	libswscale/libswscale.so.5
LD	libpostproc/libpostproc.so.55
LD	libswresample/libswresample.so.3
STRIP	libavcodec/x86/vp9itxfm.o
GEN	libavcodec/libavcodec.ver
LD	libavcodec/libavcodec.so.58
LD	libavdevice/libavdevice.so.58
LD	ffmpeg_g
LD	ffplay_g
LD	ffprobe_g

All the libraries are linked sequentially, i.e. just one CPU core is active.

comment:3 Changed 6 months ago by oromit

Most of the libraries depend on each other, so they can only be linked when their dependencies are already done linking.

comment:4 follow-up: Changed 6 months ago by cehoyos

I tested the following and see a speed impact if I increase the number of threads for make:

$ rm -f libpostproc/libpostproc.so* libswscale/libswscale.so* libswresample/libswresample.so* && time make -j1 libpostproc/libpostproc.so libswscale/libswscale.so libswresample/libswresample.so

As explained, some libraries depend on others.
Did you ever test the speed impact of lto on the final FFmpeg binaries?
If you really need faster build time, search for LDFLAGS in ffbuild/config.mak and change -flto to -flto=4. Not sure how easy this is to allow in configure.

comment:5 Changed 6 months ago by cehoyos

  • Summary changed from Please enable parallel linking when --enable-lto is specified to Allow to set number of cpu cores for --enable-lto

comment:6 in reply to: ↑ 4 ; follow-up: Changed 6 months ago by birdie

Replying to cehoyos:

If you really need faster build time, search for LDFLAGS in ffbuild/config.mak and change -flto to -flto=4. Not sure how easy this is to allow in configure.

This sounds like a great idea - thanks but I still hope that Makefile could be altered, so that 'make -jN' also made linking with -flto faster by default.

comment:7 in reply to: ↑ 6 Changed 6 months ago by cehoyos

Replying to birdie:

Replying to cehoyos:

If you really need faster build time, search for LDFLAGS in ffbuild/config.mak and change -flto to -flto=4. Not sure how easy this is to allow in configure.

This sounds like a great idea - thanks but I still hope that Makefile could be altered, so that 'make -jN' also made linking with -flto faster by default.

Please understand that this is simply not possible, sorry for not closing this ticket as invalid, I have been flamed too often;-(

comment:8 Changed 6 months ago by cehoyos

Btw, did you ignore the most important part of my post?

Did you ever test the speed impact of lto on the final FFmpeg binaries?

comment:9 Changed 6 months ago by birdie

Nope, never.

From what I've read and understood -flto is a safe [optimization] option even though it might give you zero gains in regard to performance which is why I prefer to enable it by default. Also I've indeed tested other pieces of software and usually -flto nets a small performance boost.

Of course, if you're a developer you'd want to disable it because it makes building significantly slower (and million times slower if you e.g. alter just one source file).

Note: See TracTickets for help on using tickets.