Difference between revisions of "Distcc"

From Void Linux Wiki
Jump to: navigation, search
m (Configuring Distcc's Whitelist)
m (Server Setup)
 
(3 intermediate revisions by the same user not shown)
Line 79: Line 79:
 
XBPS_DISTCC_HOSTS="localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9"
 
XBPS_DISTCC_HOSTS="localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9"
 
XBPS_MAKEJOBS=11</nowiki>}}
 
XBPS_MAKEJOBS=11</nowiki>}}
 +
 +
= Using Distcc with Cross Compilers and Xbps-src =
 +
So you might have an old Raspberry Pi around or some other embedded board that you want to build your own packages for. The problem is that you don't have hours and days to wait for that system to compile programs for you. You however happen a significantly faster Laptop or PC laying around which you can utilize.
 +
 +
In this scenario, you have two options: Cross-Compile your programs on your faster computer, or run the compilation from your slower embedded system.
 +
 +
For this example, I will be building armv6 binaries for an original Raspberry Pi (This will be the Distcc server). My cross compiling machine in this instance is an x86_64 desktop which I have installed a cross compiler on (This will be the client machine). If you're cross-compiling for a different system, use the relevant cross-compiler tool chains and native tool chains that match your target architecture.
 +
 +
== Server Configuration ==
 +
Follow the general steps for the server setup above together with the steps for enabling Distcc in Xbps-src.
 +
 +
=== Xbps-Src Whitelist ===
 +
Because Xbps-src runs in a chroot, it can't read our whitelist directory that we configured in the server stage. The directory that will be read is located at '''masterdir/usr/lib/distcc/bin'''. Don't worry though, the Xbps-src bootstrap did most of the heavy lifting already! The following is an example of what that directory should look like.
 +
 +
{{hc|<path-to-xbps-package>/metadata/usr/lib/distcc/bin/|
 +
<nowiki>lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-c++ -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-cc -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-g++ -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-gcc -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:38 c++
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:46 cc
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:38 g++
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:38 gcc</nowiki>}}
 +
 +
Although, this looks fine so far, this configuration is going to tell distcc to call {{ic|cc}} instead of {{ic|arm-linux-gnueabihf-cc}}. The client server will simple see {{ic|cc}} and compile the object using it's native compiler. In our case, it'll produce x86_64 readable binary instead of arm readable. The workaround for this comes in two parts.
 +
 +
The first part is a wrapper script, courtesy of the [https://wiki.gentoo.org/wiki/Distcc/Cross-Compiling#Configuring_distcc_to_cross-compile_correctly Gentoo Wiki: Distcc Cross Compiling] page. Create the following at '''metadata/usr/lib/distcc/bin/arm-linux-gnueabihf-wrapper'''.
 +
 +
{{hc|<path-to-xbps-package>/metadata/use/lib/distcc/bin/arm-linux-gnueabihf-wrapper|
 +
#!/bin/bash
 +
exec /usr/lib/distcc/bin/arm-linux-gnueabihf-g${0:$[-2]} "$@"}}
 +
 +
Now we want to make the script executable and symlink our compiler names to it.
 +
 +
{{bc|
 +
chmod a+x arm-linux-gnueabihf-wrapper
 +
ln -s arm-linux-gnueabihf-wrapper cc
 +
ln -s arm-linux-gnueabihf-wrapper g++
 +
ln -s arm-linux-gnueabihf-wrapper gcc
 +
ln -s arm-linux-gnueabihf-wrapper g++}}
 +
 +
What this script does is removes the last two letters of the called command, for example {{ic|c++}}, and appends it to the end of the path in the script. This will then call {{ic|/usr/lib/distcc/bin/arm-linux-gnueabihf-g++}}. The {{ic|"$@"}} is the rest of the command that was passed to distcc.
 +
 +
To finish up, our whitelist directory now should look similar to this.
 +
 +
{{hc|<path-to-xbps-package>/metadata/usr/lib/distcc/bin/|
 +
<nowiki>lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-c++ -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-cc -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-g++ -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  19 Feb  6 20:40 arm-linux-gnueabihf-gcc -> ../../../bin/distcc*
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:38 c++ -> arm-linux-gnueabihf-wrapper*
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:46 cc -> arm-linux-gnueabihf-wrapper*
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:38 g++ -> arm-linux-gnueabihf-wrapper*
 +
lrwxrwxrwx 1 rpi users  27 Jun  9 09:38 gcc -> arm-linux-gnueabihf-wrapper*
 +
-rwxr-xr-x 1 rpi users  74 Jun  9 10:13 arm-linux-gnueabihf-wrapper*</nowiki>}}
 +
 +
Start up the Distcc server as normal.
 +
 +
{{bc|ln -s /etc/sv/distccd /var/service}}
 +
 +
== Client Configuration ==
 +
If you haven't already installed a Cross-Compiler, please follow the directions at the [https://wiki.voidlinux.org/Cross_Compiler Cross Compiler] page.
 +
 +
Just like the server, we need to setup our client whitelist. This is done once again by running {{ic|update-distcc-symlinks}}. You will end up with a ''/var/lib/distcc''' directory similar to below.
 +
 +
{{hc|<path-to-xbps-package>/metadata/usr/lib/distcc/bin/|
 +
<nowiki>lrwxrwxrwx  1 root root    16 Jun  9 22:14 arm-linux-gnueabihf-g++ -> ../../bin/distcc*
 +
lrwxrwxrwx  1 root root    16 Jun  9 22:14 arm-linux-gnueabihf-gcc -> ../../bin/distcc*
 +
lrwxrwxrwx  1 root root    16 Jun  9 22:14 arm-linux-gnueabihf-gcc-8.3.0 -> ../../bin/distcc*
 +
</nowiki>}}
 +
 +
You should know how this goes by now.
 +
 +
{{bc|ln -s /etc/sv/distccd /var/service}}
 +
 +
== Building Packages ==
 +
Building packages will functionally be the exact same as usual, just the build process will become MUCH faster. Look at the [https://wiki.voidlinux.org/Xbps-src Xbps-src] page for details.
 +
 +
== Caveats ==
 +
=== Link Time Optimizations ===
 +
Void linux builds some packages with LTO (Link Time Optimizations) enabled by default in the meson build-style script. This is a great choice on embedded systems since we want optimized, faster running packages for our slower hardware. Building with LTO enabled doesn't play well with distcc however. You'll run into linker issues which will make your build fail.
 +
 +
There are two potential options: Try running the build with distcc's pump mode, or you could disable LTO for either individual packages or globally.
 +
 +
Disabling LTO on a per-package basis means that you need to add {{ic|<nowiki>-DB_lto=false</nowiki>}} into the {{ic|configure_args}} variable of your packages template. Alternatively, you could disable it globally in '''common/build-style/meson.sh''' by simply changing {{ic|<nowiki>-DB-lto=true</nowiki>}} to {{ic|<nowiki>-DB-lto=false</nowiki>}}.
 +
 +
The Distcc-Pump method likely wont work, mainly because there is currently a bug with the distcc-pump package. It is currently masked from cross-compiling it and when compiled natively, pump fails to create a socket which makes it useless. You could compile distcc yourself from [https://github.com/distcc/distcc the projects github] page, but pump may or may not be a work around for LTO in most cases.
  
 
[[Category:Packages]]
 
[[Category:Packages]]
 +
[[Category:Embedded]]
 +
[[Category:Guides]]

Latest revision as of 23:34, 11 June 2019

Distcc is a utility that allows the user to distribute their compiling jobs across a network. On Void Linux, this can be used with xbps-src. This guide will explain how to setup a build server (the machine you're building on) and a client (the machine that your server will distribute jobs too).

Please keep in mind that you should have matching compiler toolchains on all of your distcc enabled systems. Any mismatches will likely lead to compiler errors and negate any of the potential speed benefits that distcc could offer you. The simplest way to achieve this is by ensuring all hosts are using the same distro (Void Linux in our case), and are all up-to-date.

Installation

Install distcc on both the server and client machines.

xbps-install -Sy distcc

Configuring Distcc's Whitelist

Distcc requires that our compiler names are symlinked in it's whitelist directory. Without these symlinks, the clients won't accept incoming jobs, and the server may fail to actually build.

The whitelist directory is located at /usr/lib/distcc for clients, and /usr/lib/distcc/bin for servers.

Luckily distcc comes with a Python 3 scipt called update-distcc-symlinks which automatically symlinks all available compilers for us.

xbps-install -Sy python3

The script expects there to be a /usr/lib/gcc-cross directory present. On Void Linux however, there isn't one by default. An easy work-around is to simply create the directory and then we can run update-distcc-symlinks.

mkdir /usr/lib/gcc-cross
update-distcc-symlinks
rmdir /usr/lib/gcc-cross

/usr/lib/distcc and /usr/lib/distcc/bin should now look something like this.

/usr/lib/distcc
drwxr-xr-x  1 root root 15 Jun 9 09:40 bin/
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c++ -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c89 -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 c99 -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 cc -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 g++ -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 gcc -> ../../bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-g++ -> /usr/bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-gcc -> /usr/bin/distcc
lrwxrwxrwx  1 root root 15 Jun 9 09:44 x86_64-unknown-linux-gnu-gcc-8.3.0 -> /usr/bin/distcc

Client and Server Configuration

The following configurations are compiling without xbps-src.

Server Configuration

The server configuration files are located in /etc/distcc/.

Clients.allow

This file simply needs the IP address of participating clients, or even a CIDR notation.

/etc/distcc/clients.allow
127.0.0.1
10.0.0.3
192.168.0.0/24

Hosts

/etc/distcc/hosts allows us to specify client behaviour. For a simple configuration, we'll set distcc to use localhost, and a machine located at 192.168.0.5.

/etc/distcc/hosts
localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9

The above specifies:

  • We are using localhost, with two jobs being sent to it (this is what the /2 means).
  • --localslots=2 means that two jobs can't be sent out to the clients and must run locally.
  • --localslots_cpp=2 specifies how many preprocessors will run in parallel.
  • 192.168.0.5/9 targets a client, and it's able to take up to 9 jobs.

You can find more options for this file in the distcc man page.

Starting the server

ln -s /etc/sv/distccd /var/service

Client Configuration

The client configuration is extremely simple. We simply need to allow add the our build servers IP address or the networks CIDR address into /etc/distcc/clients.allow.

/etc/distcc/clients.allow
192.168.0.0/24

Finally we can start up the distcc server.

ln -s /etc/sv/distccd /var/service

Enable Distcc with Xbps-src

Configure your /etc/distcc/clients.allow on your clients and server.

Append void-packages/etc/conf on the server with the following.

void-packages/etc/conf
XBPS_DISTCC=yes
XBPS_DISTCC_HOSTS="localhost/2 --localslots=2 --localslots_cpp=2 192.168.0.5/9"
XBPS_MAKEJOBS=11

Using Distcc with Cross Compilers and Xbps-src

So you might have an old Raspberry Pi around or some other embedded board that you want to build your own packages for. The problem is that you don't have hours and days to wait for that system to compile programs for you. You however happen a significantly faster Laptop or PC laying around which you can utilize.

In this scenario, you have two options: Cross-Compile your programs on your faster computer, or run the compilation from your slower embedded system.

For this example, I will be building armv6 binaries for an original Raspberry Pi (This will be the Distcc server). My cross compiling machine in this instance is an x86_64 desktop which I have installed a cross compiler on (This will be the client machine). If you're cross-compiling for a different system, use the relevant cross-compiler tool chains and native tool chains that match your target architecture.

Server Configuration

Follow the general steps for the server setup above together with the steps for enabling Distcc in Xbps-src.

Xbps-Src Whitelist

Because Xbps-src runs in a chroot, it can't read our whitelist directory that we configured in the server stage. The directory that will be read is located at masterdir/usr/lib/distcc/bin. Don't worry though, the Xbps-src bootstrap did most of the heavy lifting already! The following is an example of what that directory should look like.

<path-to-xbps-package>/metadata/usr/lib/distcc/bin/
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-c++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-cc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-g++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-gcc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 c++
lrwxrwxrwx 1 rpi users   27 Jun  9 09:46 cc
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 g++
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 gcc

Although, this looks fine so far, this configuration is going to tell distcc to call cc instead of arm-linux-gnueabihf-cc. The client server will simple see cc and compile the object using it's native compiler. In our case, it'll produce x86_64 readable binary instead of arm readable. The workaround for this comes in two parts.

The first part is a wrapper script, courtesy of the Gentoo Wiki: Distcc Cross Compiling page. Create the following at metadata/usr/lib/distcc/bin/arm-linux-gnueabihf-wrapper.

<path-to-xbps-package>/metadata/use/lib/distcc/bin/arm-linux-gnueabihf-wrapper
#!/bin/bash
exec /usr/lib/distcc/bin/arm-linux-gnueabihf-g${0:$[-2]} "$@"

Now we want to make the script executable and symlink our compiler names to it.

chmod a+x arm-linux-gnueabihf-wrapper
ln -s arm-linux-gnueabihf-wrapper cc
ln -s arm-linux-gnueabihf-wrapper g++
ln -s arm-linux-gnueabihf-wrapper gcc
ln -s arm-linux-gnueabihf-wrapper g++

What this script does is removes the last two letters of the called command, for example c++, and appends it to the end of the path in the script. This will then call /usr/lib/distcc/bin/arm-linux-gnueabihf-g++. The "$@" is the rest of the command that was passed to distcc.

To finish up, our whitelist directory now should look similar to this.

<path-to-xbps-package>/metadata/usr/lib/distcc/bin/
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-c++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-cc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-g++ -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   19 Feb  6 20:40 arm-linux-gnueabihf-gcc -> ../../../bin/distcc*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 c++ -> arm-linux-gnueabihf-wrapper*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:46 cc -> arm-linux-gnueabihf-wrapper*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 g++ -> arm-linux-gnueabihf-wrapper*
lrwxrwxrwx 1 rpi users   27 Jun  9 09:38 gcc -> arm-linux-gnueabihf-wrapper*
-rwxr-xr-x 1 rpi users   74 Jun  9 10:13 arm-linux-gnueabihf-wrapper*

Start up the Distcc server as normal.

ln -s /etc/sv/distccd /var/service

Client Configuration

If you haven't already installed a Cross-Compiler, please follow the directions at the Cross Compiler page.

Just like the server, we need to setup our client whitelist. This is done once again by running update-distcc-symlinks. You will end up with a /var/lib/distcc' directory similar to below.

<path-to-xbps-package>/metadata/usr/lib/distcc/bin/
lrwxrwxrwx   1 root root     16 Jun  9 22:14 arm-linux-gnueabihf-g++ -> ../../bin/distcc*
lrwxrwxrwx   1 root root     16 Jun  9 22:14 arm-linux-gnueabihf-gcc -> ../../bin/distcc*
lrwxrwxrwx   1 root root     16 Jun  9 22:14 arm-linux-gnueabihf-gcc-8.3.0 -> ../../bin/distcc*

You should know how this goes by now.

ln -s /etc/sv/distccd /var/service

Building Packages

Building packages will functionally be the exact same as usual, just the build process will become MUCH faster. Look at the Xbps-src page for details.

Caveats

Link Time Optimizations

Void linux builds some packages with LTO (Link Time Optimizations) enabled by default in the meson build-style script. This is a great choice on embedded systems since we want optimized, faster running packages for our slower hardware. Building with LTO enabled doesn't play well with distcc however. You'll run into linker issues which will make your build fail.

There are two potential options: Try running the build with distcc's pump mode, or you could disable LTO for either individual packages or globally.

Disabling LTO on a per-package basis means that you need to add -DB_lto=false into the configure_args variable of your packages template. Alternatively, you could disable it globally in common/build-style/meson.sh by simply changing -DB-lto=true to -DB-lto=false.

The Distcc-Pump method likely wont work, mainly because there is currently a bug with the distcc-pump package. It is currently masked from cross-compiling it and when compiled natively, pump fails to create a socket which makes it useless. You could compile distcc yourself from the projects github page, but pump may or may not be a work around for LTO in most cases.