distcc reduced compilation times to less than a half!
distcc is a program to distribute compilation of C, C++, Objective C or Objective C++ code across several machines on a network.
Just evaluated distcc for building a ~1,4 Mio lines of code C++ code base on Linux with GCC. My local machine is already quite powerful. I usually build on 8 cores simultaneously (ninja -j8) with some quad-core Intel i7 and a quite fast SSD. A full build takes ~26 minutes. Also partial rebuilds quite often take more than 10 minutes” - annoying and too long.
Now the cool thing:
A full build with distcc, 18 simultaneous build processes, distributed to 3 machines, just takes 12 minutes instead of 26!
And it’s totally easy to setup. This is what I did on Ubuntu:
sudo apt-get install distcc
edit /etc/default/distcc - this is mine:
STARTDISTCC="true"
ALLOWEDNETS="127.0.0.1 192.168.1.0/24"
LISTENER=""
NICE="10"
JOBS="8"
ZEROCONF="false"
Ensure you add 127.0.0.1 to ALLOWEDNETS to allow connections from localhost.
Edit /etc/distcc/hosts and add the hosts of your cluster (hostnames or IPs):
localhost hostName1 hostName2 hostIPx
(Re)start the daemon:
sudo service distcc restart
Set the C compiler to /usr/lib/distcc/gcc and the C++ compiler to /usr/lib/distcc/g++. I’m using CMake and I just do the following before calling CMake:
export CC=/usr/lib/distcc/gcc
export CXX=/usr/lib/distcc/g++
Now, when you build your project, the build process gets distributed to the hosts of your cluster. You can watch what’s going on with:
watch distccmon-text
There is one important thing to consider: ALL MACHINES IN THE CLUSTER MUST USE THE SAME TOOLCHAIN/COMPILER VERSION!
I’m not using the even faster “pump mode”. Pump seems to have some problems with our code that heavily uses boost everywhere, also see problem with distcc in pump mode.
Everything went smoothly (except “pump mode”) and a distcc cluster is really easy to set up. The gains are huge and it’s a pity I haven’t used it before. I can definitely recommend it!