Difference between revisions of "Distcc"

From Void Linux Wiki
Jump to: navigation, search
m (Enable Distcc with Xbps-src)
m (Configuring Distcc's Whitelist)
Line 12: Line 12:
 
The whitelist directory is located at '''/usr/lib/distcc''' for clients, and '''/usr/lib/distcc/bin''' for servers.  
 
The whitelist directory is located at '''/usr/lib/distcc''' for clients, and '''/usr/lib/distcc/bin''' for servers.  
  
Luckily distcc comes with a {{Pkg|python3|Python 3}}, {{ic|update-distcc-symlinks}} which automatically symlinks all available compilers for us.
+
Luckily distcc comes with a {{Pkg|python3|Python 3}} scipt called {{ic|update-distcc-symlinks}} which automatically symlinks all available compilers for us.
 
{{bc|xbps-install -Sy python3}}
 
{{bc|xbps-install -Sy python3}}
  

Revision as of 12:08, 9 June 2019

Distcc is a utility that allows the user to distribute their compiling jobs across a network. On Void Linux, this can be used with xbps-src. This guide will explain how to setup a build server (the machine you're building on) and a client (the machine that your server will distribute jobs too).

Please keep in mind that you should have matching compiler toolchains on all of your distcc enabled systems. Any mismatches will likely lead to compiler errors and negate any of the potential speed benefits that distcc could offer you. The simplest way to achieve this is by ensuring all hosts are using the same distro (Void Linux in our case), and are all up-to-date.

Installation

Install distcc on both the server and client machines.

xbps-install -Sy distcc

Configuring Distcc's Whitelist

Distcc requires that our compiler names are symlinked in it's whitelist directory. Without these symlinks, the clients won't accept incoming jobs, and the server may fail to actually build.

The whitelist directory is located at /usr/lib/distcc for clients, and /usr/lib/distcc/bin for servers.

Luckily distcc comes with a Python 3 scipt called update-distcc-symlinks which automatically symlinks all available compilers for us.

xbps-install -Sy python3

The script expects there to be a /usr/lib/gcc-cross directory present. On Void Linux however, there isn't one by default. An easy work-around is to simply create the directory and then we can run update-distcc-symlinks.

mkdir /usr/lib/gcc-cross
update-distcc-symlinks
rmdir /usr/lib/gcc-cross

/usr/lib/distcc and /usr/lib/distcc/bin should now look something like this.

/usr/lib/distcc
drwxr-xr-x  1 root root 15 Jun 9 09:40 bin/
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c++ -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c89 -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c99 -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 cc -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 g++ -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 gcc -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-g++ -> /usr/bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-gcc -> /usr/bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-gcc-8.3.0 -> /usr/bin/distcc

Client and Server Configuration

The following configurations are compiling without xbps-src.

Server Configuration

The server configuration files are located in /etc/distcc/.

Clients.allow

This file simply needs the IP address of participating clients, or even a CIDR notation.

/etc/distcc/clients.allow
127.0.0.1
10.0.0.3
192.168.0.0/24

Hosts

/etc/distcc/hosts allows us to specify client behaviour. For a simple configuration, we'll set distcc to use localhost, and a machine located at 192.168.0.5.

/etc/distcc/hosts
localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9

The above specifies:

  • We are using localhost, with two jobs being sent to it (this is what the /2 means).
  • --localslots=2 means that two jobs can't be sent out to the clients and must run locally.
  • --localslots_cpp=2 specifies how many preprocessors will run in parallel.
  • 192.168.0.5/9 targets a client, and it's able to take up to 9 jobs.

You can find more options for this file in the distcc man page.

Starting the server

ln -s /etc/sv/distccd /var/service

Client Configuration

The client configuration is extremely simple. We simply need to allow add the our build servers IP address or the networks CIDR address into /etc/distcc/clients.allow.

/etc/distcc/clients.allow
192.168.0.0/24

Finally we can start up the distcc server.

ln -s /etc/sv/distccd /var/service

Enable Distcc with Xbps-src

Configure your /etc/distcc/clients.allow on your clients and server.

Append void-packages/etc/conf on the server with the following.

void-packages/etc/conf
XBPS_DISTCC=yes
XBPS_DISTCC_HOSTS="localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9"
XBPS_MAKEJOBS=11