Lesson 1: Why Automate Our Builds?
Building software is often a complex process. A single software "product" may be composed of hundreds or thousands of parts, some created by one team, some created by another, some open-source software products, and some proprietary software provided by some vendor. All of these parts must be "assembled" correctly in order to create the final product.
So, as in every other process DevOps touches, as DevOps practitioners we seek to automate everything. Why? Because:
- Repeatedly performing such a process by hand is boring.
- Repeatedly performing such a process by hand is, especially since it is boring, error prone.
- By automating such a process, we can ensure the inclusion of steps like automated testing and security checks, which are likely to be left out, at least on occasion, if the process is performed manually.
- The scripts that automate such a process also serve to document it, and in a fashion such that the documentation cannot fall behind the actual process, since it is the actual process!
Lesson 2: Build Tools Comparison
Build tools are programs that automate the creating of the various products involved in some project. Whenever some "product" is constructed by some program(s) from various components that are combined into that product, we should employ a build tool to automate that combining, rather than building that product by running a series of commands "by hand."
The thing built may be a complete program, a portion of a program, documentation, a book, a database configuration, a Docker container: in short, whenever some component of a system we are working on is composed of sub-components, we should seek to automate the building of that component with a build tool.
make was first created in 1976, but the variants of
this build tool are still in use.
make we have to specify
a dependency followed by a rule to resolve that
dependency. It is a powerful product oriented tool
which is highly capable of tracking dependencies within
a build and building only those parts/components
that have been updated/changed, thus helping in
optimizing performance whereby compilation time is
crucial in development cycles. The drawback of this
tool is that with increase in the rise of complexity,
the number of dependencies between components also
increases, making for complex rules that are
hard to debug.
Ant is a task-oriented build tool which was first released in 2000. The limitations found in make led the Java Community to experiment with different solutions such as porting make to Java. At this same time XML was coming up into prominence. Combining the power of both these approaches led to the Apache Ant build tool. Being fully cross platform, a set of instructions are written in Java to address basic operations like compilation and file manipulation. Besides being cross-platform, Ant is also an extremely flexible and powerful system, with Ant Tasks for most things we want to do. Despite being powerful, Ant comes with certain drawbacks, such as the fact that the build scripts have to be written in XML (not quite pleasant for human readability), a lot of time has to be spent in performing tasks (writing boiler plate to compile, create jars, run tests etc) due to lack of a real domain concept. Ant files are poorly factored and tend to get really long, so it won't be surprising to find files thousands of lines long.
Maven is a build tool that was released in 2004. Maven was built to overcome problems in Ant. Maven uses XML but less so than Ant does. Maven boasts that it will perform any kind of build, deploy, test and release task with a single command as long as we conform to the structure dictated by Maven. The disadvantage lies in the fact that we have to conform to the Maven structure: if we don't follow the Maven structure then it is almost impossible to make Maven do what we want. Another disadvantage is that it uses an external DSL (Domain Specific Language) written in XML, which means that if we want to extend it then we have to spend a lot of time learning something that language. Maven might also end up creating issues since it continuously tries to automatically upgrade itself every time it runs, this means some plugins might end up failing and we wont be able to reproduce our builds.
PyBuild is a build tool which is aimed at Python projects. PyBuild supports all the standard features of typical build tools. There are many plugins created by Python developers to support building with PyBuild. Plugins exist for the following areas:
- Unit Testing
- Measuring Unit Test Coverage
- Linting Python Source
SonarQube is an open source tool written in Java that helps to analyze the quality of source code. It has the ability to analyze source code in over 20 languages. The code analysis step can be used manually by executing sonar runner but however the best use of SonarQube is during the build process. If SonarQube is integrated with the Jenkins server it can provide continuous integration and reports based on the analysis of the code.
Lesson 3: Make
For our class, we will use
make as our build tool.
Although there are more modern tools with additional features,
make is sufficient for our purposes because:
makeis widely available on all UNIX-based systems.
makeallows us to explore the automation of builds in a way that is quite enough for an introduction to such tools.
make executes makefiles.
The basic structure of a
- A target, that we seek to build.
- A list of dependencies, upon which that target depends; and
- A series of commands to be run, in order to build that target from its dependencies.
A key aspect of
make's behavior is that it
examines the time at which the target and its dependencies
were last updated in order to "decide" whether
to execute the commands that build the target from its
dependencies. If the timestamp on the target file is newer
than any of those on the dependency files,
will "judge" that the target is up-to-date, and does not need
to be rebuilt.
A second crucial aspect of
make's behavior is
when it is building a target by executing one or more commands,
should any of those commands fail (return a non-zero exit
make will stop trying to build that
target and report the error. It is this feature of
that allows us to insert automated tests into a build, and halt
the build should any of those tests fail.
Having examined the logic of a makefile, let us look at an actual instance of one, from one of our projects:
OK, DevOps participants, the iframe documentation I have read suggests that the contents of this makefile should appear above: but they don't! Let's debug this!