Notes on Building Software

#### Beyond Linux® From Scratch - Version 2019-02-27

### Chapter 2. Important Information

- Prev

  Important Information

- Next

  The /usr Versus /usr/local Debate

- Up
- Home

# Notes on Building Software

Those people who have built an LFS system may be aware of the general principles of downloading and unpacking software. Some of that information is repeated here for those new to building their own software.

Each set of installation instructions contains a URL from which you can download the package. The patches; however, are stored on the LFS servers and are available via HTTP. These are referenced as needed in the installation instructions.

While you can keep the source files anywhere you like, we assume that you have unpacked the package and changed into the directory created by the unpacking process (the 'build' directory). We also assume you have uncompressed any required patches and they are in the directory immediately above the 'build' directory.

We can not emphasize strongly enough that you should start from a clean source tree each time. This means that if you have had an error during configuration or compilation, it's usually best to delete the source tree and re-unpack it before trying again. This obviously doesn't apply if you're an advanced user used to hacking `Makefile`s and C code, but if in doubt, start from a clean tree.

## Building Software as an Unprivileged (non-root) User

The golden rule of Unix System Administration is to use your superpowers only when necessary. Hence, BLFS recommends that you build software as an unprivileged user and only become the `root` user when installing the software. This philosophy is followed in all the packages in this book. Unless otherwise specified, all instructions should be executed as an unprivileged user. The book will advise you on instructions that need `root` privileges.

## Unpacking the Software

If a file is in `.tar` format and compressed, it is unpacked by running one of the following commands:

    tar -xvf filename.tar.gz
    tar -xvf filename.tgz
    tar -xvf filename.tar.Z
    tar -xvf filename.tar.bz2

### Note

You may omit using the `v` parameter in the commands shown above and below if you wish to suppress the verbose listing of all the files in the archive as they are extracted. This can help speed up the extraction as well as make any errors produced during the extraction more obvious to you.

You can also use a slightly different method:

    bzcat filename.tar.bz2 | tar -xv

Finally, you sometimes need to be able to unpack patches which are generally not in `.tar` format. The best way to do this is to copy the patch file to the parent of the 'build' directory and then run one of the following commands depending on whether the file is a `.gz` or `.bz2` file:

    gunzip -v patchname.gz
    bunzip2 -v patchname.bz2

## Verifying File Integrity Using 'md5sum'

Generally, to verify that the downloaded file is genuine and complete, many package maintainers also distribute md5sums of the files. To verify the md5sum of the downloaded files, download both the file and the corresponding md5sum file to the same directory (preferably from different on-line locations), and (assuming `file.md5sum` is the md5sum file downloaded) run the following command:

    md5sum -c file.md5sum

If there are any errors, they will be reported. Note that the BLFS book includes md5sums for all the source files also. To use the BLFS supplied md5sums, you can create a `file.md5sum` (place the md5sum data and the exact name of the downloaded file on the same line of a file, separated by white space) and run the command shown above. Alternately, simply run the command shown below and compare the output to the md5sum data shown in the BLFS book.

    md5sum <name_of_downloaded_file>

## Creating Log Files During Installation

For larger packages, it is convenient to create log files instead of staring at the screen hoping to catch a particular error or warning. Log files are also useful for debugging and keeping records. The following command allows you to create an installation log. Replace `<command>` with the command you intend to execute.

    ( <command> 2>&1 | tee compile.log && exit $PIPESTATUS )

`2>&1` redirects error messages to the same location as standard output. The tee command allows viewing of the output while logging the results to a file. The parentheses around the command run the entire command in a subshell and finally the exit $PIPESTATUS command ensures the result of the `<command>` is returned as the result and not the result of the tee command.

## Using Multiple Processors

For many modern systems with multiple processors (or cores) the compilation time for a package can be reduced by performing a "parallel make" by either setting an environment variable or telling the make program how many processors are available. For instance, a Core2Duo can support two simultaneous processes with:

    export MAKEFLAGS='-j2'

or just building with:

    make -j2

Generally the number of processes should not exceed the number of cores supported by the CPU. To list the processors on your system, issue: `grep processor /proc/cpuinfo`.

In some cases, using multiple processors may result in a 'race' condition where the success of the build depends on the order of the commands run by the make program. For instance, if an executable needs File A and File B, attempting to link the program before one of the dependent components is available will result in a failure. This condition usually arises because the upstream developer has not properly designated all the prerequisites needed to accomplish a step in the Makefile.

If this occurs, the best way to proceed is to drop back to a single processor build. Adding '-j1' to a make command will override the similar setting in the MAKEFLAGS environment variable.

### Note

When running the package tests or the install portion of the package build process, we do not recommend using an option greater than '-j1' unless specified otherwise. The installation procedures or checks have not been validated using parallel procedures and may fail with issues that are difficult to debug.

## Automated Building Procedures

There are times when automating the building of a package can come in handy. Everyone has their own reasons for wanting to automate building, and everyone goes about it in their own way. Creating `Makefile`s, Bash scripts, Perl scripts or simply a list of commands used to cut and paste are just some of the methods you can use to automate building BLFS packages. Detailing how and providing examples of the many ways you can automate the building of packages is beyond the scope of this section. This section will expose you to using file redirection and the yes command to help provide ideas on how to automate your builds.

### File Redirection to Automate Input

You will find times throughout your BLFS journey when you will come across a package that has a command prompting you for information. This information might be configuration details, a directory path, or a response to a license agreement. This can present a challenge to automate the building of that package. Occasionally, you will be prompted for different information in a series of questions. One method to automate this type of scenario requires putting the desired responses in a file and using redirection so that the program uses the data in the file as the answers to the questions.

Building the CUPS package is a good example of how redirecting a file as input to prompts can help you automate the build. If you run the test suite, you are asked to respond to a series of questions regarding the type of test to run and if you have any auxiliary programs the test can use. You can create a file with your responses, one response per line, and use a command similar to the one shown below to automate running the test suite:

    make check < ../cups-1.1.23-testsuite_parms

This effectively makes the test suite use the responses in the file as the input to the questions. Occasionally you may end up doing a bit of trial and error determining the exact format of your input file for some things, but once figured out and documented you can use this to automate building the package.

### Using yes to Automate Input

Sometimes you will only need to provide one response, or provide the same response to many prompts. For these instances, the yes command works really well. The yes command can be used to provide a response (the same one) to one or more instances of questions. It can be used to simulate pressing just the Enter key, entering the Y key or entering a string of text. Perhaps the easiest way to show its use is in an example.

First, create a short Bash script by entering the following commands:

    cat > blfs-yes-test1 << "EOF"
    #!/bin/bash

    echo -n -e "\n\nPlease type something (or nothing) and press Enter ---> "

    read A_STRING

    if test "$A_STRING" = ""; then A_STRING="Just the Enter key was pressed"
    else A_STRING="You entered '$A_STRING'"
    fi

    echo -e "\n\n$A_STRING\n\n"
    EOF
    chmod 755 blfs-yes-test1

Now run the script by issuing ./blfs-yes-test1 from the command line. It will wait for a response, which can be anything (or nothing) followed by the Enter key. After entering something, the result will be echoed to the screen. Now use the yes command to automate the entering of a response:

    yes | ./blfs-yes-test1

Notice that piping yes by itself to the script results in y being passed to the script. Now try it with a string of text:

    yes 'This is some text' | ./blfs-yes-test1

The exact string was used as the response to the script. Finally, try it using an empty (null) string:

    yes '' | ./blfs-yes-test1

Notice this results in passing just the press of the Enter key to the script. This is useful for times when the default answer to the prompt is sufficient. This syntax is used in the Net-tools instructions to accept all the defaults to the many prompts during the configuration step. You may now remove the test script, if desired.

### File Redirection to Automate Output

In order to automate the building of some packages, especially those that require you to read a license agreement one page at a time, requires using a method that avoids having to press a key to display each page. Redirecting the output to a file can be used in these instances to assist with the automation. The previous section on this page touched on creating log files of the build output. The redirection method shown there used the tee command to redirect output to a file while also displaying the output to the screen. Here, the output will only be sent to a file.

Again, the easiest way to demonstrate the technique is to show an example. First, issue the command:

    ls -l /usr/bin | more

Of course, you'll be required to view the output one page at a time because the more filter was used. Now try the same command, but this time redirect the output to a file. The special file `/dev/null` can be used instead of the filename shown, but you will have no log file to examine:

    ls -l /usr/bin | more > redirect_test.log 2>&1

Notice that this time the command immediately returned to the shell prompt without having to page through the output. You may now remove the log file.

The last example will use the yes command in combination with output redirection to bypass having to page through the output and then provide a y to a prompt. This technique could be used in instances when otherwise you would have to page through the output of a file (such as a license agreement) and then answer the question of “do you accept the above?”. For this example, another short Bash script is required:

    cat > blfs-yes-test2 << "EOF"
    #!/bin/bash

    ls -l /usr/bin | more

    echo -n -e "\n\nDid you enjoy reading this? (y,n) "

    read A_STRING

    if test "$A_STRING" = "y"; then A_STRING="You entered the 'y' key"
    else A_STRING="You did NOT enter the 'y' key"
    fi

    echo -e "\n\n$A_STRING\n\n"
    EOF
    chmod 755 blfs-yes-test2

This script can be used to simulate a program that requires you to read a license agreement, then respond appropriately to accept the agreement before the program will install anything. First, run the script without any automation techniques by issuing ./blfs-yes-test2.

Now issue the following command which uses two automation techniques, making it suitable for use in an automated build script:

    yes | ./blfs-yes-test2 > blfs-yes-test2.log 2>&1

If desired, issue tail blfs-yes-test2.log to see the end of the paged output, and confirmation that y was passed through to the script. Once satisfied that it works as it should, you may remove the script and log file.

Finally, keep in mind that there are many ways to automate and/or script the build commands. There is not a single “correct” way to do it. Your imagination is the only limit.

## Dependencies

For each package described, BLFS lists the known dependencies. These are listed under several headings, whose meaning is as follows:

- Required means that the target package cannot be correctly built without the dependency having first been installed.

- Recommended means that BLFS strongly suggests this package is installed first for a clean and trouble-free build, that won't have issues either during the build process, or at run-time. The instructions in the book assume these packages are installed. Some changes or workarounds may be required if these packages are not installed.

- Optional means that this package might be installed for added functionality. Often BLFS will describe the dependency to explain the added functionality that will result.

## Using the Most Current Package Sources

On occasion you may run into a situation in the book when a package will not build or work properly. Though the Editors attempt to ensure that every package in the book builds and works properly, sometimes a package has been overlooked or was not tested with this particular version of BLFS.

If you discover that a package will not build or work properly, you should see if there is a more current version of the package. Typically this means you go to the maintainer's web site and download the most current tarball and attempt to build the package. If you cannot determine the maintainer's web site by looking at the download URLs, use Google and query the package's name. For example, in the Google search bar type: 'package_name download' (omit the quotes) or something similar. Sometimes typing: 'package_name home page' will result in you finding the maintainer's web site.

## Stripping One More Time

In LFS, stripping of debugging symbols was discussed a couple of times. When building BLFS packages, there are generally no special instructions that discuss stripping again. It is probably not a good idea to strip an executable or a library while it is in use, so exiting any windowing environment is a good idea. Then you can do:

    find /{,usr/}{bin,lib,sbin} \
        -type f \( -name \*.so* -a ! -name \*dbg \) \
        -exec strip --strip-unneeded {} \;

If you install programs in other directories such as `/opt` or `/usr/local`, you may want to strip the files there too.

For more information on stripping, see http://www.technovelty.org/linux/stripping-shared-libraries.html.

Last updated on 2019-02-26 22:49:50 -0600

- Prev

  Important Information

- Next

  The /usr Versus /usr/local Debate

- Up
- Home