A General Introduction To Package Creation -
This guide will try to explain the building process of a package with the aim of helping you, the user, to understand the procedures behind it and, in the end, create automated scripts to build the desired software. This guide supposes you have zero knowledge about building and creating packages.
A General Introduction
If you have ever tried to build your own package, you will most surely had to follow these three instructions:
$ ./configure $ make # make install
These commands are very well known among the Linux/UNIX community, however, it is usually not that simple, for there are more methods to build packages, and/or running these commands may not work at all, even if the creator of the software you are trying to build says they are all that is needed.
Lets try to understand these three simple to type, yet powerful commands, and we will expand the rest through them.
configure command is part of the GNU Autoconf utility. It is widely used to configure the building process that will take place with the next step.
The reason behind it's existence is the ability to modify the general behaviour of a program before it is built, and to check wether it could be build or not (requirement checking). It generates a file that will retain all the information, a
Makefile, which will be used in the next step.
make is part of the Automake package.
make will read the information stored in the
Makefile and will run the building process according to the information stored in the
make install which is part of the Automake utility also, handles the systemwide installation of the package that
make created. Placing it where the
configure command specified.
Expanding the basis
The introduction gives you a general idea on how the simplest parts work, but we are still far from being able to fully use all their potential, so lets expand on what can they do and how can we control all the options they offer.
Most software is written from scratch, meaning that they do not base their work on anything that was present before it. However, this does not mean a piece of software works all by itself. If you have ever programmed, you will recognize this line on C code
#include <foo.h> . The include statement calls for another file which has valuable information required by the program being written, should this file not be present, the program may fail. The same process happens with dependencies. To be more specific, a dependency, as the name implies, is another piece of software which a program requires in order to work.
However, this is not entirely true, we need to make a distinction between two types of dependencies, which here we will call them hard and soft dependencies. A hard dependency will be one which is required in any given case by the software we are building, should it not be present, the building process will always fail, no matter what we try (apart from patching). On the other hand we have soft dependencies (also known as optional) these are dependencies that we can choose wether to use them and compile the package with them, or not. The downside of not using a soft dependency is that some functionality that a package may give the user, may no longer be present if we decide to package a software without one of it's dependencies. The dependencies will usually be indicated by the software's creator.
As an example:
blender needs to be built with
python as it is noted by the blender's maintainer, this is a hard dependency. Other dependencies, such as
bullet physics are not needed to make the build, however, if the build is done without
bullet physics, then no physics engine will be available.
Now, lets answer the only question you may be asking yourself. Why do you make the distinction between dependencies? Why not package something with all of it's dependencies? It is simpler and you get a lot more features. Yes, that is true, but it presents a lot of drawbacks. First, not every feature may be needed. Some dependencies may be huge (see openjdk-jre) and the feature they offer may not be so needed by the users. Secondly, you need to track more dependencies, which complicates the maintenance, and the worst part, they may change with time as software progresses, which is not ideal. And finally, size, some systems do not have a lot of resources available: defaulting to a lot of dependencies may not be great for those users. However, a nice solution has been found for these issues. Most building systems (noted further in the text) allow us to create subpackages. These special packages are usually meant to partition the features of a bigger packages into smaller ones, allowing the user install the ones they may need (but this topic won't be covered in this guide, it is definetly not a dificult topic, yet it is out of scope for this entry).
Also, lets make another distinction:
hostmakedependencies, makedependencies and (user)dependencies. The host-make-dependencies are dependencies needed by the machine building the software which are not directly needed by the program; for example,
git is a tool used to get code from the internet, but it is not related to the building procedure apart from just using some tools before everything else starts. Make-dependencies are dependencies needed to build the software. And (user)dependencies are dependencies needed to run the program (more correctly called runtime dependencies, but lets keep things simple and funtional here). So, what is the difference between the last two types? Easy, remember
autoconf? It is a program needed to build most software, but it only builds things, after that, it is not required any more, therefore not needed by the user to run the program. This paragraph is just general knowledge, we are getting our selves too ahead, lets continue.
One great way to get more information about how can we control the building options that this command will select, is through the help it prints. So just type:
$ ./configure --help
It will most likely print a long list of options and a few paragraphs detailing some information about
./configure itself and the software we are trying to configure. Read the output carefully, for many problems originate within the information that lies here. Right away (well, depending on the package we are using) you will see things such as
--prefix=$PATH, --enable-FEATURE, CFLAGS, etc
Here is where things start to get dirty. Depending on the complexity of the software we may be trying to build, more or less work needs to be done here, but here lies the core part, get ready, for from here things start to get technical.
We will start with
CFLAGS, CXXFLAGS. These are options that you will generally never touch. Then, why talk about them? Because they are very, very important. They will tell the compiler (the part of the building chain that translates written code to bits understandable by the CPU) how it should do the translation into machine code. Most building systems (portage, xbps, pkgsrc, etc) come with some sane defaults for these options, that is why you should never touch them. Nevertheless, some software requires tweaking. Some software will require speciall flags to be either turned on or off, here is where things start to get fun. Due to some default settings in the building chain, as already mentioned, have flags that won't let software build. The way you detect this issue is during the actual building proces (after you have run
make), when it fails. Fix accordingly. TIP: contact more knowledgeable people (sysadmins) and let them help you, or read the projects documentation to see if they explicitly state a building option should be on or off.
The next bit we will cover will be
--enable-feature bit. This flags are generally related to mayor features the program may offer, and therefore, is strongly related to dependencies, though not always. Be careful with this options, for a program may compile correctly, but when it is used, it may not work as intended due to this step. TIP: if you disable a feature dependent on a dependency, drop the dependency too; read what the software maintainer has written about all these options.
Finally, a quick note about
--prefix=$PATH. This bit will tell the last step
make install where should the software be installed in your system. It will usually be set for
/usr. However, some software may be installed into
/opt path, usually reserved for software that is unpacked (not compiled) and non-standard programs.
Also, it is very important to check the output it prints while it is running, and yes, it is usually a lot, but we don't need to understand all. The important parts are the ones where it checks for files that are not from the program itself. As an example: we may have installed a soft (optional) dependency, however, when we run
./configure it ends with an error pointing to a missing file that comes from our dependency, that would mean it is not being correctly found or that we haven't explicitly enabled the feature. And even if nothing goes wrong, you still want to read the output, while it may not give you errors, but since it is an automated process, it may do something we did not want, like turning off some type of feature.
Please, refer to
cmake later in this wiki entry for more information about it, which is a mixture of
make. It is generally used for much bigger software, and it is usually more complete but more complex to operate.
./configure process has ended we will need to run
make. As it was explained earlier, make takes all the configuration present in the
./configure generated) and does all the building.
There is not much to say about
make directly, apart from what we have said before. Yet this is the core process, if something is going to go very very wrong, it is going to be here. Lets just say it is our debugging part, here we will face most of the most difficult errors to solve, and due to this guide's nature, we won't cover most of them.
The most common error is a syntax error. The programmer may have made a mistake writing the code and therefore it cannot be understood by the compiler and the process cannot continue. Just patch this small error yourself and comunicate it to the project's lead.
The second most common error is a file missing. It may happend that
./configure may have found a needed dependency, but when it comes to the compiler, it may not be the same (even though they use the same variable to search for files: LDPATH). Try to look for the missing file and get it somewhere where it can be found automatically.
Next is the compiling flag argument (yeah, the CFLAGS and CXXFLAGS thing), as written earlier, some flags may be missing or it is not possible to use them. You will have to fix this bit in the
./configure step and rerun it, since the
Makefile needs to be rebuilt.. As it was explained earlier, make takes all the configuration present in the
./configure generated) and does all the building.