Infrastructure as Code
Lesson 1: What Is Infrastructure as Code?
Infrastructure as code (IaC) as the idea that, rather than manually provisioning servers, or setting up hardware through a point-and-click GUI, the "server room" should itself be managed by code. That code can then be put under version control, tested, deployed with automated build tools, and so on. The code also serves as necessarily up-to-date documentation of what the infrastructure is.
The advantages of IaC can be divided into three main categories:
- Cost savings: By automating hardware provisioning, the time of the people who would have been doing that by hand is freed up for other tasks.
- Speed of deployment: It is much faster to configure infrastructure by running a script than by manually setting a bunch of parameters in a GUI interface.
- Lower error rates: It is error-prone, because it is boring, too configure systems "by hand." A script can be debugged once, and then will run reliably again and again. Furthermore, as code, the infrastructure can be read and reasoned about. It is very hard to do that with a bunch of check-boxes!
(Source: Wikipedia on Infrastructure as Code )
A DevOps principle:
Asking people to behave like automatons
bores and dehumanizes
them. Asking them to devise clever ways
to automate things
interests them, and treats them as the rational
beings that they are!
Aristotle: humans are rational animals.
Lesson 2: Available Tools
- Easy installation
- Supports all majors operating systems
- GUI is user friendly
- Stable and mature solution
- New users must learn Puppet DSL (domain-specific language)
- Remote execution is challenging
- Meant to be used by programmers
- Useful for large-scale development
- Stable and mature solution
- Good version control capabilities
- Complicated tool to use
- Familiarity with Ruby is required
- Documentation can be overwhelming
- Easy and fast deployment
- Secure SSH connection
- Meant for environments that can scale rapidly
- Push and pull models are supported
- Basic support for Windows
- GUI is not very interactive
- Difficult to locate syntax errors with YAML
- Implemented in Python and controlled with YAML files which are simple to understand
- Fast communication between master and client
- Provides high scalability and resiliency
- Vibrant support community
- Difficult to set up
- Salt GUI is under development
- Does not support a variety of Operating Systems
Lesson 3: Running Docker
First thing: Make sure you have Docker installed! You won't get any further in following along on your laptop if you do not.
Please clone (if you have not already) our online DevOps
git clone https://github.com/gcallah/OnlineDevops.git
Once you have cloned that repo, please open two shells: in one, we will look at your local environment, and in the other we will explore the container.
In one of the two shells, in your OnlineDevops directory,
That should put you inside the OnlineDevops container: if that command worked, you should see your prompt change.
If it did, let's explore the shells you are in a little to try to understand better what a container is.
I am going to proceed by showing you the results of running the same command inside and outside the container on my machine: your results will be different, but similar, to mine.
First of all, let's look at the root file system, inside and outside the container:
Outside the container:
Macintosh:OnlineDevops gcallah$ ls / Applications bin net Library cores opt Network debug.txt private Shockwave Log debug.txt.1 sbin System dev tmp User Guides And Information etc usr User Information home var Users installer.failurerequests Volumes logFile.xsl
Inside the container:
root@a5dd222a9812:/home/DevOps# ls / bin dev home lib64 mnt proc root sbin sys usr boot etc lib media opt requirements.txt run srv tmp var
What's of note here: From outside and from inside the container, we see completely different file systems! Inside the container, we are in a chroot file system.
What about our view of what processes are running?
Outside the container we see:
ps -ef | wc -l
Macintosh:OnlineDevops gcallah$ ps -ef | wc -l 667
From outside the container, the OS lists 667 processes as running on my Mac.
Inside the container we see:
root@a5dd222a9812:/home/DevOps# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 Oct25 pts/0 00:00:00 bash root 192 1 0 21:18 pts/0 00:00:00 ps -ef
From inside the container, there are two processes running! The container has process isolation from the host. It has its own process namespace separate from its host's namespace and from the namespace of any other containers running on that host. The separate namespace also provides the container with its own hostname, its own user IDs, and its own inter-process communication names.
Some Docker commands
Now let's look at what some Docker commands are available, and what they do.
This will list your currently running Docker images. When
Prof. Callahan runs it while preparing this lecture, he
Macintosh:OnlineDevops gcallah$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 429ff1a37d21 devops "bash" 11 seconds ago Up 6 seconds 0.0.0.0:32768->8000/tcp stupefied_swanson
(The 0.0.0.0 is the IP address to use for the container, and 32768 is the port it is using.)
docker ps -a
This will list all images that have been run on the system, not just those that are active:
Macintosh:OnlineDevops gcallah$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c560c3729d7e devops "bash" 18 minutes ago Up 18 minutes 0.0.0.0:8000->8000/tcp brave_kirch 429ff1a37d21 devops "bash" About an hour ago Exited (130) 44 minutes ago stupefied_swanson 32b272487675 devops "bash" 5 days ago Created wonderful_heyrovsky a5dd222a9812 devops "bash" 10 days ago Exited (130) About an hour ago naughty_beaver e82e4fb65823 devops "bash" 11 days ago Exited (130) 11 days ago youthful_noyce 0433f774c394 devops "bash" 11 days ago Created stupefied_hawking 1ba39e023554 devops "bash" 11 days ago Created practical_ptolemy 4a6bc8cf7ab0 devops "bash" 11 days ago Created vigilant_curie 47cb14ca5b1b devops "bash" 2 weeks ago Exited (255) 11 days ago 0.0.0.0:8000->8000/tcp determined_goldwasser 7af5f14f88d9 f418f33054e8 "bash" 2 weeks ago Exited (130) 2 weeks ago modest_hypatia e3292cdc1449 indra "bash" 2 weeks ago Exited (130) 2 weeks ago competent_hugle 3ad630752ebf indra "bash" 2 weeks ago Exited (130) 2 weeks ago amazing_meitner fe439d8e15c2 indra "bash" 2 weeks ago Exited (130) 2 weeks a
This command should give you the list of images
that are available on the system. For example
in Prashant's system it looks something like
ENG-EJC369-02:$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE gcallah/jenkins_py3 latest 21130e104ca1 4 weeks ago 742MB jenkins latest cd14cecfdb3a 6 weeks ago 696MB gcallah/indra v7 ad9670e8b27f 2 months ago 946MB python latest efb6baa1169f 5 months ago 691MB ubuntu latest f975c5035748 5 months ago 112MB gcallah/emu86 v4 f6833ae8bf9e 6 months ago 776MB gcallah/django latest 432de70e222d 6 months ago 769MB bash latest 59507b30b48a 6 months ago 12.2MB alpine latest 3fd9065eaf02 7 months ago 4.15MB
In the above listing, the
is not installed. Running the following command will
pull the nginx image from DockerHub, which is like GitHub,
but for docker images.
docker pull nginx
Now that you have installed the nginx image,
just run the docker images command again,
in the list you should see ngnix image as below:
ENG-EJC369-02:$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 71c43202b8ac 7 hours ago 109MB gcallah/jenkins_py3 latest 21130e104ca1 4 weeks ago 742MB jenkins latest cd14cecfdb3a 6 weeks ago 696MB gcallah/indra v7 ad9670e8b27f 2 months ago 946MB python latest efb6baa1169f 5 months ago 691MB ubuntu latest f975c5035748 5 months ago 112MB gcallah/emu86 v4 f6833ae8bf9e 6 months ago 776MB gcallah/django latest 432de70e222d 6 months ago 769MB bash latest 59507b30b48a 6 months ago 12.2MB alpine latest 3fd9065eaf02 7 months ago 4.15MB
(You can put an image into DockerHub using
Download website code
Ok now that you have download the nginx image, let's download the static website that you are going to host inside the docker container. We are going to use the algorithms website for another course. You can find the code for the website here.
Please remember the location where you have
cloned the repository. Prashant has cloned it in
Your location will be different than this, please note
Let's make a container
Ok, so we have downloaded the nginx image and the code of the website which we want to host All we need to do is just make a container out of the image. We will put the code of website inside the container so that the webserver which is nginx in our case can read the html files and host it in local server. The command for that is as shown below.
docker run --name algo_website -p 127.0.0.1:8080:80
Please don't forget to change the location of
algorithms directory in above command.
After you run the command open the browser
and you should be able to see the webpage.
Windows user should use the address
instead of localhost.
After we leave the container, we can get rid of it using
docker rm algo_website.
If we need to remove a container that is still running,
we will have to stop it first with
Our Docker Implementation
We use Docker in our projects for two main reasons, with a third to come:
- To set up a local version of a web server that will be configured "just like" our production server. ("Just like" is in quotes because that is always the ideal, but it may not be fully achieved.)
- To provide our full suite of development tools, such as the correct Python version, make, flake8, various Python libraries, etc., in one simple to build package, so all developer's have a consistent environment.
- Ultimately, we should be deploying the container where we locally test our web servers right into production, guaranteeing that development and production are identical environments. Unfortunately, at the moment, the places we are hosting do not support that. We are exploring other options.
So we need to know how to create the right container for
Each project we work on should have a
Dockerfile consisting of instructions
on how to build the image for that project, a
requirements.txt listing what
external modules need to be included in the image,
and a line in the project's
automating the build of the image.
This is infrastructure as code, since the infrastructure for
the project is built from these files of code.
So, in the
makefile we want something
container: $(DOCKER_DIR)/Dockerfile $(DOCKER_DIR)/requirements.txt
docker build -t indra docker
Here is a sample Dockerfile.
FROM python:3.6.0command says what base image to build our image from.
COPY requirements.txt /requirements.txtline brings the requirements file inside the container.
RUN pip install -r requirements.txtinstalls everything from the requirements file in the container.
ENV user_type TERMINALsets an environment variable (
user_type) that will be available inside the container.
WORKDIR /home/IndrasNet/sets the starting directory inside the container.
- Here is the requirements file it uses.
Our More Advanced Docker Usage
In the Online DevOps course project, we use a more advanced Docker setup: we employ Docker Compose to define and run an application composed of more than one Docker container.
Docker Compose use a YAML file to specify the configuration of a multi-container Docker app.
In the Online DevOps course setup, we are running just two
containers: one to run the MySQL database, and the other
to run our Django web server. But other Docker Compose
setups might include a web server, a load balancer,
a database, an authentication server, and more!
Here is the YAML file that specifies our two-container application.