Automating Builds

Automating our builds
Lesson 1: Why Automate Our Builds?

Building software is often a complex process. A single software "product" may be composed of hundreds or thousands of parts, some created by one team, some created by another, some open-source software products, and some proprietary software provided by some vendor. All of these parts must be "assembled" correctly in order to create the final product.

So, as Software Engineers we seek to automate everything. Why? Because:

  • Repeatedly performing such a process by hand is boring.
  • Repeatedly performing such a process by hand is, especially since it is boring, error prone.
  • By automating such a process, we can ensure the inclusion of steps like automated testing and security checks, which are likely to be left out, at least on occasion, if the process is performed manually.
  • The scripts that automate such a process also serve to document it, and in a fashion such that the documentation cannot fall behind the actual process, since it is the actual process!
Lesson 2: Build Tools Comparison

Build tools are programs that automate the creating of the various products involved in some project. Whenever some "product" is constructed by some program(s) from various components that are combined into that product, we should employ a build tool to automate that combining, rather than building that product by running a series of commands "by hand."

The thing built may be a complete program, a portion of a program, documentation, a book, a database configuration, a Docker container: in short, whenever some component of a system we are working on is composed of sub-components, we should seek to automate the building of that component with a build tool.

Make

    make was first created in 1976, but the variants of this build tool are still in use. In make we have to specify a dependency followed by a rule to resolve that dependency. It is a powerful product oriented tool which is highly capable of tracking dependencies within a build and building only those parts/components that have been updated/changed, thus helping in optimizing performance whereby compilation time is crucial in development cycles. The drawback of this tool is that with increase in the rise of complexity, the number of dependencies between components also increases, making for complex rules that are hard to debug.

Ant

    Ant is a task-oriented build tool which was first released in 2000. The limitations found in make led the Java Community to experiment with different solutions such as porting make to Java. At this same time XML was coming up into prominence. Combining the power of both these approaches led to the Apache Ant build tool. Being fully cross platform, a set of instructions are written in Java to address basic operations like compilation and file manipulation. Besides being cross-platform, Ant is also an extremely flexible and powerful system, with Ant Tasks for most things we want to do. Despite being powerful, Ant comes with certain drawbacks, such as the fact that the build scripts have to be written in XML (not quite pleasant for human readability), a lot of time has to be spent in performing tasks (writing boiler plate to compile, create jars, run tests etc) due to lack of a real domain concept. Ant files are poorly factored and tend to get really long, so it won't be surprising to find files thousands of lines long.

Maven

    Maven is a build tool that was released in 2004. Maven was built to overcome problems in Ant. Maven uses XML but less so than Ant does. Maven boasts that it will perform any kind of build, deploy, test and release task with a single command as long as we conform to the structure dictated by Maven. The disadvantage lies in the fact that we have to conform to the Maven structure: if we don't follow the Maven structure then it is almost impossible to make Maven do what we want. Another disadvantage is that it uses an external DSL (Domain Specific Language) written in XML, which means that if we want to extend it then we have to spend a lot of time learning something that language. Maven might also end up creating issues since it continuously tries to automatically upgrade itself every time it runs, this means some plugins might end up failing and we wont be able to reproduce our builds.

PyBuild

PyBuild is a build tool which is aimed at Python projects. PyBuild supports all the standard features of typical build tools. There are many plugins created by Python developers to support building with PyBuild. Plugins exist for the following areas:

  • Unit Testing
  • Measuring Unit Test Coverage
  • Linting Python Source

SonarQube

SonarQube is an open source tool written in Java that helps to analyze the quality of source code. It has the ability to analyze source code in over 20 languages. The code analysis step can be used manually by executing sonar runner but however the best use of SonarQube is during the build process. If SonarQube is integrated with the Jenkins server it can provide continuous integration and reports based on the analysis of the code.

Sources:

  1. PyBuild
  2. Build Concept
  3. Managing projects with make
  4. Java Code Geeks
  5. What's in a build tool
Lesson 3: Make

For our class, we will use make as our build tool. Although there are more modern tools with additional features, make is sufficient for our purposes because:

  • make is widely available on all UNIX-based systems.
  • make allows us to explore the automation of builds in a way that is quite enough for an introduction to such tools.

make executes makefiles. The basic structure of a makefile is:

  • A target, that we seek to build.
  • A list of dependencies, upon which that target depends; and
  • A series of commands to be run, in order to build that target from its dependencies.

A key aspect of make's behavior is that it examines the time at which the target and its dependencies were last updated in order to "decide" whether to execute the commands that build the target from its dependencies. If the timestamp on the target file is newer than any of those on the dependency files, make will "judge" that the target is up-to-date, and does not need to be rebuilt.

A second crucial aspect of make's behavior is that, when it is building a target by executing one or more commands, should any of those commands fail (return a non-zero exit code), make will stop trying to build that target and report the error. It is this feature of make that allows us to insert automated tests into a build, and halt the build should any of those tests fail.

Having examined the logic of a makefile, let us look at an actual instance of one, from one of our projects:

Other Readings
Quiz

    A makefile includes... ?

    1. servers, ports, and inodes
    2. documentation, a version control system, and monitoring
    3. target, dependencies and commands
    4. none of the above

    Ant build scripts are written in...?

    1. Markdown
    2. Python
    3. XML
    4. C++

    Build management includes...?

    1. source repository
    2. build console
    3. package repository
    4. all of the above

    An advantage of documenting a process by automating it versus writing a document about how it is done is that...?

    1. scripts are easier to read than prose
    2. it gives programmers an edge over tech writers in producing the documentation
    3. the script can't be out of date with the process, while a text description can be
    4. all of the above

    Popular build tools include...?

    1. make, Ant, and Maven
    2. grep, awk, and sed
    3. git, StatusCake, and Slack
    4. all of the above

    During the build process 'make' builds...?

    1. all of the components
    2. updated or changed components
    3. make is NOT a build tool
    4. none of the above

    When we are using make and have things to build in subdirectories, a good solution is to...?

    1. recursively call `make` in the subdirectory
    2. turn to Ant or Maven to build your program
    3. write your own build manager
    4. just build things in the subdirectories by hand

    Package repository allows us to...?

    1. group the software into functioning packages
    2. have a place to keep all of our packages
    3. increase the use of external packages
    4. none of the above

    A line in a makefile like `prod: test docker` means...?

    1. There is a target named `prod` that depends on `test` and `docker`.
    2. There is a command named `prod` that gets the arguments `test` and `docker`.
    3. There is a server named `prod` that lives on the `test docker` subnet.
    4. None of the above.

    A manual process that is boring is... ?

    1. good for the programmer
    2. highly paid
    3. suitable for junior programmers
    4. error prone