Complete DevOps Operations (Tool: Azure DevOps)
"It's clear DevOps has become increasingly critical to a team's success." -- Jamie Cool, Microsoft
Lesson 1: Microsoft's DevOps Tools
Here we have Denis Petelin presenting, first, his understanding of what DevOps is, and secondly, how the Microsoft toolkit offers end-to-end support for good DevOps practices:
Below are the key DevOps ideas that Denis presented, with some commentary as to how they relate to other material presented in our course:
This is our incremental development idea stated a little differently. The result of adopting this practice is almost magical: whereas previously, you may have found yourself doing an eight-hour coding spree, and then testing and correcting bugs for another 20 hours, you will find that 12 one-hour sessions, each aimed to get a tiny part of your software working, will get the same amount of work done as the 8-hour session... but will be followed by 20 minutes of debugging.
Everything as code
Infrastructure as code is a part of this principle. But also, your tests should be code (
pytest), and your code reviews should be code (
flake8), and your builds should be code (
This is an extension of the UNIX tool philosophy, which over many years led to the idea of microservices. The basic idea is the code that has the fewest bugs and the minimum development time is the code you don't have to write, since somebody else already wrote it!
Automate everything you can
Automating routine workflows allows us to:
- Turn them into code that itself can be put under version control and tested.
- Remove the drudgery of these repeated tasks from humans and assign it to machines.
- Document these workflows through the very code that automates them.
When we embrace the idea of (very) small releases, and adopt automated testing, there is no need for us to have any "bug backlog" at all: the automated tests should catch most problems, and anything they miss, since our release was a very small addition to the previous code, will be easy to track down.
Develop resilience (be anti-fragile!)
Nassim Taleb teaches in Tandon's finance department. He has developed the idea of anti-fragility. An application of that idea is Netflix's Chaos Monkey tool, part of its Simian Army toolkit.
Use cloud services
This is really an extension of the "Use components" idea: why manage your own server room when there are pros at managing server rooms who will sell you their expertise at a reasonable cost?
Lesson 2: DevOps at DRAFT
Please watch this video, and pay special attention to the discussion about whether DevOps should be a job, or a way of working.
Azure DevOps Ecosystem
Microsoft Azure, as a part of its push towards the DevOps philosophy, provides services aimed at increasing productivity through deep integrations on its platform. The main aim is to enable continuous development (CD) of our production software and continuous integration (CI) of updates to the same. We explore various services provided by Microsoft below:
Boards is a project management service offered by Microsoft aimed at integrating the development activity using easy-to-use interface. Boards is primarily based on the Kanban system, originally designed for lean manufacturing by Toyota. Apart from tracking progress from "idea to release", Boards allows for scheduling of stand- ups, sprints and planning meetings. Boards allows for CI/CD though a feature called Backlogs which allows for dynamic addition of user stories and requirements to product backlog. This feature, when coupled with Scrum and Kanban boards, can dramatically improve productivity and ease the workflow.
Steps to integrate Azure Boards with GitHub
- Sign in to Azure Boards and select your project
- Select Project Settings->GitHub Connections->Connect your GitHub Account
- Azure will prompt for GitHub authorization and inform about access to information
- The user can select repositories where the user is an administrator
Once a connection is made, Azure will link GitHub pull and push requests with work orders on the platform.
Subsequent commit messages can include AB# followed by the work item ID. E.g.: AB#125 will record updates to work item with ID 125. Azure recognizes fix, fixes, and fixed as keywords to close an issue which are subsequently moved to the "Done" section of the Board.
Additional examples can be found here
Artifacts allow package management using existing CI/CD pipelines to enable efficient code sharing across different teams in an organization. Support exists to use npm, Maven and NuGet package managers. A new feature introduced by Microsoft, called Universal Package allows developers to bundle multiple files such as database snapshots, machine learning models, developer tools, 3D models and configuration scripts. Microsoft's compression technology allows savings of 10:1 in network transfers and storage utilization. Native integration with existing CI/CD pipelines enables quick integration in existing projects and production code.
Steps to use Azure Artifacts with YAML (Python - PyPi)
Authenticate Azure Artifacts with Twine
To authenticate twine, we must add the following snippet to our azure-pipelines.yml file
Note: artifactFeeds should contain the name of Azure Artifacts feeds within your organization and externalFeeds should contain one or more connection endpoints including PyPi or other feeds in the organization where the latest package is required to be pushed
- Use custom twine tasks to publish
Steps to use Azure Artifacts with YAML (NPM)
Setting up the connection To publish to an
external npm registry, a service connection must
be created first. To do this, go to Project
Select npm from the option and fill the registry URL and enter the credentials to connect.
- Publishing to NPM
- Install twine
Azure DevTest Labs
Azure DevTest Labs is a service that helps developers and testers quickly create environments in Azure while minimizing waste and controlling cost. You can test the latest version of your application by quickly provisioning Windows and Linux environments using reusable templates and artifacts. Easily integrate your deployment pipeline with DevTest Labs to provision on-demand environments. Scale up your load testing by provisioning multiple test agents, and create pre-provisioned environments for training and demos.
Azure Pipelines helps you implement a build, test, and deployment pipeline for any app.You can use Azure Pipelines to build an app written in any language. You can either use YAML to define your pipelines or use the visual designer to do the same. When you use YAML, you define your pipeline mostly in code (a YAML file) alongside the rest of the code for your app. When you use the visual designer, you define a build pipeline to build and test your code, and then to publish artifacts. You also define a release pipeline to consume and deploy those artifacts to deployment targets.
Steps to Use Azure Pipelines with YAML
You can configure your pipelines in a YAML file that exists alongside your code.
- Configure Azure Pipelines to use your Git repo.
- Edit your azure-pipelines.yml file to define your build.
- Push your code to your version control repository. This action kicks off the default trigger to build and deploy and then monitor the results.
- Your code is now updated, built, tested, and packaged. It can be deployed to any target.
Use Azure Pipelines in the visual designer
You can create and configure your build and release pipelines in the Azure DevOps Services web portal with the visual designer.
- Configure Azure Pipelines to use your Git repo.
- Use the Azure Pipelines visual designer to create and configure your build and release pipelines.
- Push your code to your version control repository. This action triggers your pipeline and runs tasks such as building or testing code.
- The build creates an artifact that's used by the rest of your pipeline to run tasks such as deploying to staging or production.
- Your code is now updated, built, tested, and packaged. It can be deployed to any target.
Azure Repos offers cloud-hosted private and public Git repos accessible through IDEs, web interfaces and any Git client. It offers collaborative code reviews that improve the quality of the code and increase the knowledge of the code base across the team. Repos also allow users to manage access to the code repositories. It is possible to assign granular permissions to users and groups. Azure Repos provide user with semantic code search which is a code-aware search that understands classes and variables. CI/CD pipelines can be set to build on code changes and with the use of branch policies, builds can be checked for quality automatically.
Azure Test Plans
Azure Test Plans provide users to design and execute browser-based planned manual and exploratory solutions to software before being pushed to production. Kanban boards are used to add, view, interact with test cases and progressively monitor the status. It alows users to let you organize tests into test plans and test suites. The test suites can be dynamic or static. Runners can be used to test apps on any platform. The bugs filed during the process will include the diagnostic data automatically. Through the testing process, it collects rich scenario data to enable easier diagnosis of issues.
Azure DevOps tool integrations
Azure provides integration with popular open source and third-party tools and services—across the entire DevOps workflow. Spend less time integrating and more time delivering higher-quality software, faster.Use Visual Studio Team Services or deploy directly to Azure infrastructure from your favorite continuous integration and continuous delivery tools, such as Jenkins. In addition to using Azure Resource Manager for infrastructure as code, you can provision and manage Azure infrastructure directly from your favorite third-party tools, such as Ansible, Chef, Puppet, and Terraform. Azure services, third-party DevOps tools, and related products all work together to help meet the most common business needs and scenarios—including yours. Get started quickly with Azure DevOps solutions that give you access to architectures, tutorials, documentation, examples, templates, partners, and other resources.
Azure Devops Project
Devops Project presents a simplified experience for deploying existing code from a Git repository and create a CI/CD pipeline. With a varied support for different programming languages and git repositories, it also provides direct integration with services like Azure Virtual Machines, Azure SQL, Azure Container Service and Kubernetes, and Azure Fabric. Tutorials for various programming languages are available here
Azure Monitoring Service
Monitoring Service lies outside the Devops platform but provides useful insights about the project by collecting, analyzing and acting on the telemetry collected from cloud and on-premise environments. It helps users better understand the health of the environments and the dependencies their projects contain.
The service collects metrics in real-time and logs from Azure platform, Guest OS and also from the application.
More information about the full suite of Azure Monitor service can be found here
Azure automation provides a way to automate cloud management task by authoring runbooks. Azure automation configuration management feature provides reports when the assigned configuration of node (system) has changed by allowing you to track unwanted changes across services and files. Common scenarios where Azure Automation is used are Build/Deploy resources, configure and monitor VM etc.
More information about Azure Automation can be found here
Azure Service Health
Azure Service Health provides personalized alerts thereby notifying and helping us understand the impact of the Azure service issue affecting the availability of our resources. It integrates into Azure monitor and provides alert through emails and text messages.
More information about Azure Service Health can be found here
Azure Docker Deployment
Different ways of deploying docker to Azure are discussed here:
Azure Container Registry
An Azure container registry is a private Docker registry
in Azure where you can store and manage your private
Docker container images.
Steps to create Azure Container Registry using Portal
Steps to create Azure Container Registry using CLI (Instructions to install the CLI binary can be found here)
You can configure webhooks for your container registry that generate events when certain actions are performed against it. For example, enable webhooks that are triggered when a container image is pushed to a registry, or deleted. When a webhook is triggered, Azure Container Registry issues an HTTP or HTTPS request containing information about the event to an endpoint you specify. Your endpoint can then process the webhook and act accordingly. The following links detail the schema of webhook requests generated by supported events and how to use the webhooks.
Webhook Request Schema
Azure Event Grid
Azure Event Grid is a fully managed event routing service that provides uniform event consumption using a publish-subscribe model. You can use the Azure CLI to create a container registry, subscribe to registry events, then deploy a sample web application to receive the events. Finally, you trigger container image push and delete events and view the event payload in the sample application.
More Information on Azure Event Grid Azure Event Grid event schema
Build and test container images with ACR Tasks in four ways:
Quick task: Build and push container images
on-demand, in Azure, without needing a local Docker
Engine installation. Think
docker pushin the cloud. Build from local source code or a Git repository.
- Build on source code commit: Trigger a container image build automatically when code is committed to a Git repository.
- Build on base image update: Trigger a container image build when that image's base image has been updated.
- Multi-step tasks (preview): Define multi-step tasks that build images, run containers as commands, and push images to a registry. This preview feature of ACR Tasks supports on-demand task execution and parallel image build, test, and push operations.
- A Docker Hub repository can be configured via webhooks to make it an event driven service. As soon as any changes are made to the repository, the URL will be called.
Deploying via Azure Container Instance (one-time)
Create a new container instance
Container instances can be used to deploy serverless Docker instances. The option is accessible through the Azure Portal by clicking on 'Create a new resource' -> searching for 'Container Instances' -> and clicking 'Create'
Define container properties
(Basics) We define properties like 'Container Name', 'Container Image' containing the Docker Tag or URI, 'Resource Group' for resource and policy management and 'Location' for the server.
(Configuration) Attributes like 'OS Type', cores, memory and other infrastructure details can be defined at this step
(Summary) The configuration will be printed once for the user to review and Azure will begin provisioning infrastructure
Deploying via Azure 'Web App for Containers' (Continuous Deployment)
Creating a web app
- Go to portal.azure.com
- Click on "Create a resource"
- Search the Marketplace for "Web App for Containers"
- Click on "Create" button when Azure displays the page for this service
- Provide details like "App Name", "Subscription" (billing), "Resource Group" (create a new group if one doesn't exist), "OS" and "App Service/Location"
- Click the option of "Configure Container" for defining the image source. Select "Docker Hub" to be the source and provide repository details like access type, image and tag information, and "Startup File"
- Click on "Apply" and then "Create" on the web app settings menu. Azure will run validation tests and begin deployment. Deployments can be monitored from the notification button in the top.
Turning on Continuous Deployment
- Go to "App Service" from the Azure Portal
- Select the application for which you wish to turn on Continuous Deployment
Click on "Container Settings" and in the next window,
turn on "Continuous Deployment"
(Example) Deploying a Class Scheduling application
- Deployment HOW-TO Video
Deployment HOW-TO Steps
- Click on "Create A New Resource" from the Azure Portal and search for "Web App for Containers".
- Click on the "Create" button on the application description screen that follows.
- Configure elements of your application like "App Name" (Must be present in the ALLOWED_HOSTS section of your website's settings.py file), "Subscription", "Resource Group", and "App Service Plan".
Click on "Container Settings" and select
Docker Hub (Container source) and enter the
image and tag details
(shaival/testnyuscheduler) in this case.
Default tag selected by Azure is "latest".
Enter the startup file details depending on
container's settings. It is a good idea to
enter absolute paths to prevent confusion.
Click on "Create".
- Access the resource from the Dashboard and click on "Container Settings".
- Turn on "Continuous Deployment" and copy the Webhook URL.
- Paste the URL in Docker Hub's "Manage Repository" settings page and assign a name to the webhook. Click on the "+" symbol to add it to Docker Hub.
- Allow Azure to pull and deploy the image. The application is accessible from the URL as mentioned in the "Overview" page.
Azure Template Deployment (one-time)
Resource deployment can be automated using Azure Template
Deployment. A template can either be created in the online editor
or imported from GitHub. Microsoft also provides some common
templates which can be configured by the user.
The link here contains a reference template to deploy the class scheduler application to Azure
Steps to deploy the script to Azure (Web Portal)
- Go to portal.azure.com and click on "Create a Resource"
- Search the marketplace for "Template deployment" and click "Create" on the product page
- Use the online editor by selecting "Build your own template in the editor"
- Paste the above embedded JSON script in the editor and click "Save"
Review the deployment and select the resource group.
Note: Application name can be customized but one must make the necessary changes in the container to ensure that the application name is in the "ALLOWED_HOST" section of the settings file
- Click on "Purchase" after reviewing and confirming to Azure's terms and conditions
Steps to deploy the script to Azure (Shell Script)
- The shell script is accessible here
Download the shell script and the template file. Make sure to
rename the template file to
Mark the shell script executable using
chmod +x azure_infrastructure_deploy.shand place the
template.jsonin the same directory as the script
Execute the shell script and pass parameters like your
deploy.sh -i lorem -g ipsum -n lorem -l ipsum
Create a Azure function
Create an HTTP interface to access the function
- Enable POST method on the function
Get the function URL
Now open Docker hub repository and paste the URL in
the webhook section.
Any changes to the Docker Hub repo will now be reported to
the Azure. Below is the log that gets generated in Azure
functions when a push was made to the Docker Hub repo
AWS Lambda and ECS
Connecting AWS and Docker Hub using webhooks
Create a lambda function
- Create an API Gateway. Make a post method in the API Gateway
Link the post method to the lambda functions
Deploy the API gateway and save the generated URL
Now open Docker Hub repository and paste the URL in
the webhook section.
Any changes to the Docker Hub repo will now be
reported to the AWS Lambda function.Below is the log
that gets generated in AWS when a push was made to
the Docker Hub repo
- Create a lambda function
Create EC2 Cluster
Go to ECS from menu and click on create Cluster
Select EC2 with networking
Make sure you select a completely new VPC, Security
group and IAM role and create.
- Go to ECS from menu and click on create Cluster
- Create Tasks on ECS
Click on task definition and create tasks with specific
docker and settings.
- Click on task definition and create tasks with specific docker and settings.
- Connect Lambda to ECS
Write code to access the cluster and specific
- Write code to access the cluster and specific Tasks
CI/CD with Azure Pipelines
Select New Pipelines (Pipelines -> New -> New build
pipeline) on the Azure Devops page.
Select the location of source code and the desired repository.
If the repository contains the
azure-pipelines.ymlfile, then you'll see a Run button. If not, you'll be asked to pick a starter template. Select Save and Run, then select commit directly to the master branch, and then select Save and Run again.
Update the YAML file appropriately depending on whether you are using Docker Hub or Azure Container registry.
- Wait until the build finishes.
Now that the build is finished, the next step is to configure and run some tests, create the container image and push it to Azure Container Registry.
Now its time to create a release pipeline and Deploy it to Azure Web
App for Containers.
Steps to create a Release pipeline and deploy the App
Azure Monitoring Services
By Default Azure provides some monitoring services for all its applications,including Docker which can be found here
- Go to Azure potal, and click on monitoring services
- Go to metrics, it will look like this
- Azure DevOps
- Introducing Azure DevOps
- Microsoft Azure DevOps: What You Need to Know
- Azure Lounge
- Docker Enterprise for Azure
- Deploy to an Azure Web App for Containers
- Multi-Container app using Docker Compose
- Monitoring Azure Services and External Systems with Azure Automation
- Monitoring Tools for Azure Services
- Application Insights support for Microservices and Containers
- Azure Monitor: Creating Alerts
- Azure Monitor: Creating Metrics
- Azure Monitoring Tools Explained
There is no quiz available for this module.