Distcc

From Void Linux Wiki
Jump to: navigation, search

Distcc is a utility that allows the user to distribute their compiling jobs across a network. On Void Linux, this can be used with xbps-src. This guide explains how to setup a build server (the machine running compilations) and a client (the machine that the server will distribute jobs to).

Please keep in mind that there should be matching compiler toolchains on all of the distcc enabled systems. Any mismatches will likely lead to compiler errors and negate any of the potential speed benefits that distcc offers. The simplest way to achieve this is to ensure all hosts are using the same distro (Void Linux ), and are all up-to-date.

Installation

Install distcc on both the server and client machines.

xbps-install -Sy distcc

Configuring Distcc's Whitelist

Distcc requires that all compiler names are symlinked in it's whitelist directory. Without these symlinks, the clients won't accept incoming jobs, and the server will likely fail to distribute jobs.

The whitelist directory is located at /usr/lib/distcc for clients, and /usr/lib/distcc/bin for servers.

Luckily distcc comes with a Python 3 scipt called update-distcc-symlinks which automatically symlinks all available compilers for us. Install python3 to utilize this script.

xbps-install -Sy python3

The script expects there to be a /usr/lib/gcc-cross directory present. On Void Linux, there isn't a /usr/lib/gcc-cross directory present by default. An easy work-around is to simply create the directory and then we can run update-distcc-symlinks.

mkdir /usr/lib/gcc-cross
update-distcc-symlinks
rmdir /usr/lib/gcc-cross

/usr/lib/distcc and /usr/lib/distcc/bin should now look like the following.

/usr/lib/distcc
drwxr-xr-x  1 root root 15 Jun 9 09:40 bin/
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c++ -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c89 -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c99 -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 cc -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 g++ -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 gcc -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-g++ -> /usr/bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-gcc -> /usr/bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-gcc-8.3.0 -> /usr/bin/distcc

Client and Server Configuration

The following configurations are for compiling without xbps-src.

Server Configuration

The server configuration files are located in /etc/distcc/.

Clients.allow

This file simply needs the IP address of participating clients, or even a CIDR notation address.

/etc/distcc/clients.allow
127.0.0.1
10.0.0.3
192.168.0.0/24

Hosts

/etc/distcc/hosts specifies client that the server will attempt to distribute to and client behavior. For a simple configuration, set distcc to use localhost, and a machine located at 192.168.0.5.

/etc/distcc/hosts
localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9

The above specifies:

  • Use localhost, with two jobs being sent to it (this is what the /2 means).
  • --localslots=2 means that two jobs can't be sent out to the clients and must run locally.
  • --localslots_cpp=2 specifies how many pre-processes will run in parallel.
  • 192.168.0.5/9 targets a client, and it's able to take up to 9 jobs.

More options for this file can be found at the distcc man page.

Starting the server

ln -s /etc/sv/distccd /var/service

Client Configuration

The client configuration is extremely simple. Add the build servers IP address or the networks CIDR address into /etc/distcc/clients.allow.

/etc/distcc/clients.allow
192.168.0.0/24

Start up the distcc server.

ln -s /etc/sv/distccd /var/service

Enable Distcc with Xbps-src

Configure /etc/distcc/clients.allow on both the clients and server.

Append void-packages/etc/conf on the server with the following.

void-packages/etc/conf
XBPS_DISTCC=yes
XBPS_DISTCC_HOSTS="localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9"
XBPS_MAKEJOBS=11

Using Distcc with Cross Compilers and Xbps-src

This section illustrates how to use distcc to build armv6 packages on an original Raspberry Pi. There is a single x86_64 client system which the Raspberry Pi will distribute jobs to.

Use the relevant cross-compiler toolchains for other other architectures.

Server Configuration

Follow the general steps for the server setup above together with the steps for enabling Distcc in Xbps-src.

Xbps-Src Whitelist

Because Xbps-src runs in a chroot, it can't read the systems whitelist directory that was configured in the above the server stage. The directory that will be read is located at <path to xbps-package>/masterdir/usr/lib/distcc/bin. Don't worry though, the Xbps-src bootstrap did most of the heavy lifting already! The following is an example of what that directory should look like.

<path to xbps-package>/metadata/usr/lib/distcc/bin/
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-c++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-cc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-g++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-gcc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 c++
lrwxrwxrwx 1 rpi users   27 Jun  9 09:46 cc
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 g++
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 gcc

Although, this looks fine so far, this configuration is going to tell distcc to call cc instead of arm-linux-gnueabihf-cc. The client server will simply see cc and compile the object using it's native compiler. In this scenario, the produced object will be x86_64 generated binary instead of the expected armv6 binary. The workaround for this comes in two parts.

The first part is a wrapper script, courtesy of the Gentoo Wiki: Distcc Cross Compiling page. Create the following at <path to xbps-package>/metadata/usr/lib/distcc/bin/arm-linux-gnueabihf-wrapper.

<path-to-xbps-package>/metadata/use/lib/distcc/bin/arm-linux-gnueabihf-wrapper
#!/bin/bash
exec /usr/lib/distcc/bin/arm-linux-gnueabihf-g${0:$[-2]} "$@"

Now we want to make the script executable and symlink the generic compiler names to it.

chmod a+x arm-linux-gnueabihf-wrapper
ln -s arm-linux-gnueabihf-wrapper cc
ln -s arm-linux-gnueabihf-wrapper g++
ln -s arm-linux-gnueabihf-wrapper gcc
ln -s arm-linux-gnueabihf-wrapper g++

What this script does is removes the last two letters of the called command, for example c++, and appends it to the end of the compiler path. So c++ will become ++ and the script will turn it into /usr/lib/distcc/bin/arm-linux-gnueabihf-g++ (The 'g' is hard-coded into the script and thus calls the relevant GCC compiler). The "$@" is the rest of the command that was passed to distcc.

To finish up, the whitelist directory now should look like the following.

<path-to-xbps-package>/metadata/usr/lib/distcc/bin/
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-c++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-cc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-g++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-gcc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 c++ -> arm-linux-gnueabihf-wrapper*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:46 cc -> arm-linux-gnueabihf-wrapper*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 g++ -> arm-linux-gnueabihf-wrapper*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 gcc -> arm-linux-gnueabihf-wrapper*
-rwxr-xr-x 1 rpi users   74 Jun  9 10:13 arm-linux-gnueabihf-wrapper*

Start up the Distcc server as normal.

ln -s /etc/sv/distccd /var/service

Client Configuration

If a cross-compiler hasn't already been installed, please follow the directions at the Cross Compiler page.

Just like the server, the clients whitelist directory also needs to be configured. This is done once again by running update-distcc-symlinks. Doing so will result in a /var/lib/distcc' directory similar to below.

<path-to-xbps-package>/metadata/usr/lib/distcc/bin/
lrwxrwxrwx   1 root root     16 Jun  9 22:14 arm-linux-gnueabihf-g++ -> ../../bin/distcc*
lrwxrwxrwx   1 root root     16 Jun  9 22:14 arm-linux-gnueabihf-gcc -> ../../bin/distcc*
lrwxrwxrwx   1 root root     16 Jun  9 22:14 arm-linux-gnueabihf-gcc-8.3.0 -> ../../bin/distcc*

Start the distccd server on the client.

ln -s /etc/sv/distccd /var/service

Building Packages

Building packages will be the same as normal, with the exception that build times should be lessened with distcc active. Look at the Xbps-src page for details on building with xbps-src.

Caveats

Link Time Optimizations

Void linux builds some packages with LTO (Link Time Optimizations) enabled by default. This is because the meson build-style script enables LTO by default. This is a great choice on embedded systems when optimized, faster running packages for our slower hardware. Building with LTO enabled doesn't play well with distcc however. LTO with distcc runs into linker issues which results build failure.

There are two potential options: Try running the build with distcc's pump mode (likely to not work), or you could disable LTO on a per-package basis or globally.

Disabling LTO on a per-package basis means adding -DB_lto=false into the configure_args variable of the the specified packages build template. Alternatively, to disable LTO globally, edit common/build-style/meson.sh and change -DB-lto=true to -DB-lto=false.

The Distcc-Pump method likely wont work, mainly because there is currently a bug with the distcc-pump package on non-x86 systems. It is currently masked from cross-compiling it. When compiled natively on a non-x86 system, pump fails to create a socket which results in no pump server being run. An alternative is to compile distcc from source (distcc github), but pump will still likely result in failed builds.