Continuous Integration / Continuous Deployment

Dave Farley on CI vs. branching
Martin Fowler on Continuous Delivery

Lesson 1: What Is CI/CD?

Continuous integration (CI) and continuous delivery (CD) are a pair of practices that, when done properly, greatly aid an organization's ability to deliver software frequently and reliably to its customers. We often hear the combination of these practices referred to as "the CI/CD pipeline," and they are among the most important practices a Software Engineering team can implement. So what do these practices consist of?

CI : Continuous Integration

Continuous Integration means that every developer should be pushing code to the master branch regularly, at the minimum once per day, and that then that code should be tested to ensure that it integrates with all the other code currently in the master branch. The practice of continuous integration eliminates the huge "integration" phase that typically took place in waterfall model projects, where developers would work on isolated pockets of code for months or even a year or more, and only try to join their code to other teams' code at the very end of the project.

On a push to the master branch, various processes can be launched, such as:

  • Running automatic code quality tests.
  • Building an executable from the code.
  • Reporting on code testing coverage.
  • Building a docker container that can be used to deploy the code.

These simple additions allows the developer to focus on writing the code. The central repository of code is there to receive your changes while automated processes can build, test, and scan the code while providing reports.

CD : Continuous Delivery

Continuous delivery means taking the result of the continuous integration process and delivering it to a "place" in which it can be run. That "place" my be a test server, a staging server, or even production. If the continuous delivery phase is actually putting the new software right into production, then it also qualifies as "continuous deployment." It almost goes without saying that only teams experienced with CI/CD should extend that practice to continuous deployment.

There are many free tools to aid in CI/CD, such as Travis CI and Jenkins, both of which we will discuss below.

References :

  • https://www.infoworld.com/article/3271126/what-is-cicd- continuous-integration-and-continuous-delivery-explained.html
  • https://www.atlassian.com/continuous-delivery/principles /continuous-integration-vs-delivery-vs-deployment
  • https://www.mabl.com/blog/what-is-cicd
Lesson 2: Jenkins

Jenkins is an open source automation tool written in Java. It aids the adoption of CI/CD because through WebHooks, Jenkins can be triggered by, for instance, a push to GitHub. Upon being triggered, Jenkins can directly, or through plugins, run a wide variety of processes that can test, package, document, analyze and deploy your application.

Advantages of Jenkins
  • It is an open source tool with great community support.
  • It has 1000+ plugins to ease your work. If a plugin does not exist, you can code it and share with the community.
  • It is free.
  • It is built with Java and hence, it is portable to all major platforms.
Introduction to Jenkins by Denis Petelin

Jenkins on Wikipedia.

References :

Lesson 3: Travis-CI

Travis CI is a relatively easy-to-setup CI/CD tool. It relies on a YAML file for its configuration, and uses a Webhook to tie to GitHub.

The Travis YAML file (.travis.yml) contains a number of directives as to how the build should be performed. The directives can specify languages to install (e.g., Python 3.6), services to employ (e.g., docker), set environment variables (such as PYTHONPATH), a main script to run (e.g., make tests), an install section to install things like your test tools, a service to notify about the state of the build (such as Slack), addons (such as adding a known host to SSH), and things to do if the build succeeded or if it failed. The YAML file is an example of declarative programming.

Here is the Travis configuration file for this website: 

    language: python
    python:
        - '3.6'
    
    services:
        - docker
    
    addons:
        ssh_known_hosts: ssh.pythonanywhere.com
    
    before_install:
        openssl aes-256-cbc -K $encrypted_4cefc8a5b2ff_key -iv $encrypted_4cefc8a5b2ff_iv -in id_rsa.enc -out ~/.ssh/id_rsa -d
    
    install:
        - pip install -r docker/requirements-dev.txt
    
    env:
        - PYTHONPATH=/home/travis/build/gcallah/OnlineDevops
    
    script:
        - make tests
    
    after_success:
        - ./deploy.sh
    
    notifications:
      slack: devopsnyu:vAFcJBgAP2f73SqhWgb40Gjf

Implementing Travis Deploy to PythonAnywhere

After the Travis YAML file is written, once it is made sure that all the tests are passing, the next step would be to start a cloud deployment. This website is deployed to PythonAnywhere.

For any deployment, one of the ways Travis would establish a connection with the cloud is SSH. But, for an automated deployment by Travis, there is a need for a non interactive SSH connection. One of the ways to set up such a connection is to use a package called SSHPass. This package provides a way to pass the password to the ssh command.

Here are the steps to do that:

  • In the Travis YAML file add the following lines to before_install: 

        before_install:
      - sudo apt-get update
      - sudo apt-get install sshpass

    The first line updates apt-get, and the second line installs the sshpass package on Travis
  • The password to ssh should not be present in any of the scripts. Luckily, there is a way to encript environment variables on Travis. So, the ssh password should be added as an encrypted environment variable (say "pa-pwd") on Travis.(This can be done on the settings tab of Travis.)
  • In the script, where the ssh command is used (say deploy.sh), pass the ssh password from the Travis environment variable set in the previous step by adding the sshpass command as follows: 

        sshpass -p $pa_pwd ssh $project_name@ssh.pythonanywhere.com

    Note: $pa_pwd is available in .travis.yml file. So, if the script " deploy.sh" is being called from .travis.yml file, then make sure that $pa_pwd is passed as an argument to " deploy.sh", which can be used along with the sshpass command.

Lesson 4: Running Jenkins

Note: Below information is about making a Jenkins build triggered by Github changes.

Install the prerequisites
  • Install JAVA if you don't already have it !
  • Install Git if you don't already have it!
  • Install Docker (optional)
  • Create a git repo to test the demo. It should consist of:

      Readme.md

      Jenkinsfile

Install and initialize Jenkins

Jenkins can be installed on various platforms and operating systems. For this demonstration we are using Docker to install Jenkins.

There are several Docker images of Jenkins available. The recommended Docker image to use is the jenkinsci/blueocean image (from the Docker Hub repository). This image contains the current Long-Term Support (LTS) release of Jenkins (which is production-ready) bundled with all Blue Ocean plugins and features.

After downloading the jenkins image you can use below docker command to install and run jenkins on docker.

For macOS/Linux : Open terminal and run the image as container in Docker using below command 


                                docker run \
                                -u root \
                                --rm \
                                -d \
                                -p 8080:8080 \
                                -p 50000:50000 \
                                -v jenkins-data:/var/jenkins_home \
                                -v /var/run/docker.sock:/var/run/docker.sock \
                                jenkinsci/blueocean

For Windows : Open command prompt and run below command 


                                docker run ^
                                -u root ^
                                --rm ^
                                -d ^
                                -p 8080:8080 ^
                                -p 50000:50000 ^
                                -v jenkins-data:/var/jenkins_home ^
                                -v /var/run/docker.sock:/var/run/docker.sock ^
                                jenkinsci/blueocean

Connect Jenkins to Github
Create a project to build your application

References :

  • https://www.mirantis.com/blog/intro-to-cicd-how- to-make-jenkins-build-on-github-changes/
    • Other Material
    Jez Humble on Continuous Delivery
    What Is Jenkins?
    Quiz

      With the use of Jenkins the source code is built and tested...?

      1. Jenkins does NOT build or test the source code
      2. once the entire source code was built
      3. after every single commit
      4. none of the above

      Jenkins uses a __________ architecture?

      1. master-master
      2. master-agent
      3. peer-to-peer
      4. monolithic

      Feature branching is a practice of...?

      1. using version control software
      2. distributing the work on project features among teammates
      3. merging code into the main branch only when a feature is complete
      4. merging code into the main branch as frequently as possible

      The scripting language for Jenkins is called...?

      1. Groovy
      2. Movie
      3. Groupie
      4. Snoopy

      CI developers integrate their work into master...?

      1. at least daily
      2. as infrequently as possible
      3. at the end of the project
      4. never

      Configuration languages are often examples of...?

      1. functional programming
      2. object-oriented programming
      3. procedural programming
      4. declarative programming

      Jenkins is written in...?

      1. Python
      2. Java
      3. C++
      4. Ruby

      The Travis YAML file is an example of...?

      1. imperative programming
      2. declarative programming
      3. procedural programming
      4. object oriented programming

      Software was moved away from release versions due to the...?

      1. rise of agile software practices
      2. growing capabilities of web services
      3. shift from monolithic to service-oriented architectures
      4. all of the above

      If we need to set a path that Travis will use in our tests, it should go in a section called:

      1. env
      2. add_ons
      3. services
      4. install