How to use Terraform to create a small-scale Cloud Infrastructure

Introduction

This tutorial gives and describes the instructions on how to use the software Terraform to create various types of Cloud infrastructures, namely Infrastructure as a service (IaaS), Platform as Service (PaaS) and Software as a Service (SaaS) and using these as a method to help the reader learn more about the use of Terraform and cloud technology in general.

This tutorial-document will provide a step-to-step guidance for anyone with no knowledge about Terraform and some of cloud technology but does have an interest in learning more about it in a small-scale test environment.

Mainly focused on IT students, this tutorial will only provide the needed knowledge most relevant for installing, configuring and using Terraform with use cases focused on real practical solutions and benchmarking these. All this while keeping in mind that it should stay accessible for the reader, and intuitive steps to follow while learning about Terraform.

As described before, this guide will cover the three types of clouds. This will be achieved using hypervisors and service providers. The following three Application Programming Interfaces will be used in this guide:

1. Kernel Based Virtual Machine (KVM) Hypervisor for IaaS

2. Docker Container Platform for PaaS

3. Azure cloud services provider for SaaS

Table of Contents

1. Introduction

2. Context

Chapter 2 will provide context on all the used software. This will describe some information about Terraform and service providers that are going to be used to create a small scale cloud infrastructure. Also, an explanation about cloud technology and orchestration software will be found here.

3. Prerequisites

Will describe some hardware requirements some of which are prerequisite to replicate the instructions, and some are recommendations to make it easier to do certain tasks.

4. Downloading and Installing Terraform

Gives the necessary instructions to install Terraform on your system. In our example, we use Ubuntu 18.04.

5. Creating your first Terraform Project

This chapter is a simple example configuration to print “hello world” to verify the installation. Furthermore, more about the usage of Terraform syntax and the functionality of resource lifecycle management will be discussed here.

6. Installing KVM and Terraform provider for libvirt

7. Create a Virtual Machine using Terraform and KVM

8. Provisioning and Orchestration

Chapters 6 to 8 cover deployment of an example SaaS project “PiCalc” using the Linux KVM hypervisor via the Terraform libvirt provider. This example also demonstrates the use of load balancing using multiple VM’s.

9. Installing Docker and Terraform provider for docker

Chapter 9 will cover the installation of the docker provider plugin, built from source using Go.

10. Deploying Docker examples with Terraform

Chapter 10 demonstrates the deployment of the example project, PiCalc, in a container provided via the Docker provider.

11. Installing Azure CLI and Terraform for Windows

12. Provision a Function app On Microsoft Azure using Terraform

Chapter 11 to 12 will cover installation and usage of the official Terraform Azure provider. This chapter is meant to display the functionality of remote provisioning that Terraform provides where the same PiCalc project will be deployed as a Function Application.

13. Troubleshooting

14. Bibliography

Context

Go back to the table of contents

This chapter will cover the software used in this guide. The goal of this chapter is to provide basic information about the software and what cloud technology in general means. The software described is mainly used to create and provision virtual machines and containers. In this guide they will be used to demonstrate some of Terraform’s functionalities and capabilities.

Cloud Technology
When talking about ‘The Cloud’ in general, this is actually a definition of various services that are provided to the user through a set platform, offered by a provider.

As described in (Mell .P & Grance .T, The NIST Definition of Cloud Computing p2–3”, 2011a) the definition of cloud computing is:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models.

This entails anything from storage and computing services to databases, networking, software and analytical services (“Microsoft — What is Cloudcomputing”, 2019a).
These technologies can be offered through a flexible and on-demand service stream to the user / customer.
Technologies like these need technical configuring to attain functionality, and can be set exactly to specific needs. Combining and configuring cloud technologies will result in a full functional cloud-infrastructure, in the end servicing the customer / user.

This is where you, as a future systems-engineer, will play an important role in realizing a cloud-infrastructure by configuring it.

Terraform
Terraform is a software which can help the user manage, build and change cloud infrastructure using code (“HashiCorp, Introduction — Terraform”, 2019a). This can be done on many popular Cloud services, such as Amazon Web Service or Microsoft Azure Services. Different types of cloud infrastructures can then be built and managed on the service of choice.

Terraform can interact with various API’s using so called ‘Provider’ plugins. These providers are a sort of middleman between Terraform and the said API. Terraform can be used to manage the Lifecycle of various created ‘resources’ of the said API. Resources can be all types of virtual devices but can also be simple file or directory. In the figure below, figure 1, an illustration can be found of Terraform software can be found with the goal to show relations between providers their resources and how Terraform manages the lifecycle but also creation and destruction of these resources.

Figure 1 — Terraform software based on: (“HashiCorp, Introduction to HashiCorp Terraform with Armon Dadgar”, 2018a)

To run Terraform software the command terraform init is executed. This command will search in the current directory for Terraform configuration files and read them. These are files with the extension .tf .Based on the configuration it will then search for the used providers. As described earlier, Terraform can be used to manage resources. Terraform can do this by creating a tree of the resources which describes which resources they are dependent on. This tree is created when the command terraform plan is called. The result of the command is a description on what Terraform will change(create or destroy) in the infrastructure. This tree is saved in the directory terraform init is called from, otherwise said, the current directory. To apply changes the command terraform apply is used. This will send configuration to the corresponding providers to create the resources defined in the code. The user can then decide what he wants to change in the infrastructure, mainly by adding code, and running the terraform apply command again. To remove an infrastructure terraform destroy can be called. This is a requirement when a certain resource is to be removed from the infrastructure completely, or requires major changes.

The user can program what infrastructure Terraform has to build and manage based These configurations can be written in two formats: HashiCorp Configuration Language (HCL), or JavaScript Object Notation (JSON). HCL is a language created by HashiCorp, it is machine-friendly yet easy for humans to read and will be used throughout this document. It heavily inspired by nginx type configurations (“HashiCorp, README.md — HCL is the HashiCorp configuration language”, 2019b).

Terraform is mainly used from the command line, there are also possibilities to run Terraform from GUI. But these are still in their early stages and not very accessible yet. An example GUI tool for Terraform is Terraform UI(“glitterlabs, Terraform UI”, 2019a).

Kernel-based Virtual Machine (KVM), QEMU and Libvirt
KVM is an open-source virtualization system for Linux systems (“Redhat, What is KVM?”, 2019a). It is a hypervisor that can be used to create virtualized, hardware accelerated virtual machines. It can achieve this using the built-in QEMU hardware emulator to emulate processors. QEMU and KVM are both configured using the configuration language, XML, (“libvirt, KVM/QEMU hypervisor driver”, 2019a) which can become a tedious task when many virtual devices must be configured. Generally, KVM/QEMU are used together in combination with a virtualization library called libvirt. Libvirt is an application user interface that can be used to manage KVM and is accessible from all types of programming languages such as Python, C#, Go and many others. One of its very useful utilities is the command line interface called ‘virsh’(“libvirt, The virtualization API”, 2019b). This tool is used to make calls to the libvirt API from the command line.

Terraform-provider-libvirt
The terraform-provider-libvirt is a provider for Terraform which makes to create resources for a virtual machine. The provider is developed by Duncan Mac-Vicar P. This software can be found on Github (“Duncan Mac-Vicar P., dmacvicar/terraform-provider-libvirt”, 2019a). Using the terraform-provider-libvirt plugin, the libvirt library can be accessed from within Terraform code and used to create and manage various virtual machines and their computing resources.

Docker
Docker is a software that delivers platform as a service software on OS-Level virtualization. The visualization instances are called containers and are basically packages of multiple software libraries and dependencies so that applications can run reliably when run on different environments (“Docker, What is a Container?”, 2019a). Many popular organizations make use of containers to achieve different types of tasks.This is mainly because docker containers are very lightweight and will reduce the amount of management and development an infrastructure will need(“Docker, Why Docker?”, 2019b). An example is PayPal using containers to manage the 200 transactions that need to be processed every second (“Docker, PayPal Use Case with Docker Enterprise”, 2019c).

Terraform-provider-docker
The Terraform provider for docker uses the Docker API to manage images, networks and life cycles of containers (“HashiCorp, Docker Provider”, 2019c). Furthermore, docker swarm, a docker orchestration platform, can be used within the code to create docker services and pools of multiple containers. The services can be used to deploy application services that can easily be scaled, updated or replaced. The provider can be found in GitHub and is developed by the Terraform community.

Azure
Microsoft Azure is a well-known cloud computing service. It provides the use of different cloud services, including those for compute, analytics, storage and networking (“Miscrosoft, What is Azure”, 2019b). We are using it mainly using Azure, because of the possibility to create a student account. Students can use “Azure for Students”, they will receive €100,- credits to spend on Azure resources.

Terraform-provider-azurem
Terraform Microsoft Azure provider makes it possible to deploy and change resources within your subscription (“HaschiCorp, Azure Provider”, 2019d). The provider will be automatically downloaded when Terraform is run. It is not a third party provider like libvirt where the provider has to be downloaded separately. The provider can access an Azure user by authenticating using a tool called Azure CLI (“HaschiCorp, Azure Provider: Authenticating using the Azure CLI”, 2019e).

Example architectures
To illustrate useful comparisons and overviews of all the providers we’re going to use in this tutorial, various architecture-diagrams are shown throughout the tutorial to give theoretical examples. Also, the fundamental basics of virtualization and Terraform will be illustrated as well.

The diagrams are created through the Archimate modelling language provided by the The Open Group Architecture Framework (“The Open Group, Welcome to the ArchiMate® 3.0.1 Specification, an Open Group Standard”, 2019a). This is an Open source modelling language used all over the world by industries and universities and is known for being very versatile and accessible (“The Open Group, Archi — Open Source ArchiMate Modelling”, 2019b).

In the first illustration (figure 2), a basic setup is shown where the Libvirt virtualization is used from the Host OS environment, thus providing KVM functionality, providing a faster and optimized addressing of the PC’s hardware.

Heed the architecture below. This is an example architecture (figure 2) illustrates KVM virtualization running on a host machine; in the context of this guide, your own laptop or desktop PC.

Figure 2 — Basic architecture KVM running on a PC (made with Archi)

The user (student) creates infrastructure within the host machine using Terraform code. This code consists of data that is required to create and configure a new virtual machine. With the help of the provider, Terraform can be used to make calls to the libvirt virtualization API, which Terraform can do with the help of the earlier mentioned terraform-provider-libvirt plugin. The libvirt API will then make the functional calls to the KVM hypervisor which can allocate the needed resources to support the new machine. It provides an abstraction layer where machines can run with different kinds of Operating systems. Hardware instructions of the virtual system are translated via the QEMU hardware emulator. It achieves this by transforming the instructions so that it can be run on a different processor. QEMU can also emulate al kinds of devices, such as Network and USB.

3. Prerequisites

Go back to the table of contents

To be able to follow the instructions described in this document accurately it is important to meet the requirements described in this chapter. It is possible to derive from the requirements described below, but no instructions of other distributions and or versions of the used software can be found in this document.

It is expected that the reader has a fundamental understanding of the GNU/Linux CLI.

The instructions described in this document are all tested on a Virtual Machine created with VMware Workstation Player 15. This Virtual Machine(VM) containing Terraform will be called the “Host Machine“ throughout this document. It should also be possible to follow these instructions on a VirtualBox host, or even directly on a physical “bare-metal” host, but this is not tested by us. In both cases, whether running on a physical or VM, it will be called the “Host Machine” as this is the machine where Terraform will be run on. This is mainly to prevent confusion later on in the document. For clarification heed the illustration below (Figure 3). Asume your “Your PC” is a Linux or Windows OS on which we have VMware. On VMware we create the machine which we call the “Host Machine”. We name it like this because we will use Terraform to create various VMs or other virtual devices, which we will call “Guest Machines”.

Figure 3 — Nested VMs (made with draw.io)

The OS used in the Host Machine is Ubuntu 18.04.3 Bionic 64-bit. This version was chosen because it is a Long Time Support(LTS) distribution of Ubuntu and will mainly come in use when Terraform is going to be used for a longer period. Another reason is that many packages used in this document have stable builds for this distribution of ubuntu.

System specifications
These are the minimum specifications for Your PC:

8 GB RAM
64-bit processor with 4 vCores
200 GB storage

These are the recommended specifications we used for our Host Machine:

4 GB RAM
64-bit processor with 4 vCores
60 GB storage

If available, a faster CPU with more vCores will speed up the Terraform workflow significantly. A mandatory requirement is a CPU that supports hardware/nested accelerated virtualization (Intel VT-x or AMD-V required). This is required to be able to create new virtual machines within our Host Machine efficiently. Nested virtualisation is required if the guide is being done within a Virtual Machine. Support for the appropriate CPU instructions can be tested on a bare-metal Ubuntu machine as follows:

$ lscpu | grep Virtualization

Our physical machine (Your PC) is Intel based, and we fully support hardware accelerated virtualisation. AMD machines would display AMD-V. The output is blank if the physical machine has no support for accelerated and nested virtualisation.

In case the Your PC is using Windows 8 or 10 64-bit, nested virtualisation within VMWare Workstation should already be accessible.

In case the Your PC is using a GNU/Linux machine, we have to instruct KVM (the Hypervisor) on the physical machine to enable nested virtualisation for us (only possible with VT-x or AMD-V). Make sure all other VM’s are closed, then enable and verify support as following:

$ sudo rmmod kvm-intel && sudo modprobe kvm_intel nested=1
$ cat /sys.module/kvm_intel/parameters

In case the physical machine has an AMD CPU, replace kvm_intel with kvm_amd.

If the output prints ‘Y’ or ‘1’, it is safe to proceed with the guide

4. Downloading and Installing Terraform

Go back to the table of contents

This chapter describes how Terraform is downloaded and installed on a Linux host system. The software will be downloaded from Terraform’s official site:

www.terraform.io/

The downloaded file is a zip file. To unarchive this file a program called unzip will be required. The resulting file is a stand-alone executable file. This file will then be moved to the ‘/usr/local/bin’ directory so that the program can be called widely across the system.

Also we made a video of the first provider KVM. At every chapter you will find a video with the exact link to the chapter in the video.

1. Download Terraform for Linux 64-bit by going to www.terraform.io/ and pressing on download CLI.

2. After pressing the download button, a tab will show up. Click on save file. This will save the file in ‘~/Downloads/’ directory by default.

Unarchive the download and install Terraform
The file needs to be unarchived before it can be used. This will be done using unzip program which is not installed automatically.

1. Start a console

2. Install unzip using the following command

$ sudo apt install unzip

3. Go into the downloads folder and extract the downloaded file using the command below. This will extract an executable called terraform in the Downloads directory.

$ cd ~/Downloads/
$ unzip terraform_0.12.12_linux_amd64.zip

Test the installation
Move ‘terraform’ to a directory for user programs called ‘/usr/local/bin’. This directory is reserved for user installed software. This action will make it possible to run Terraform from any directory in your system.

1. Move the unarchived file, which is an executable, to ‘/usr/local/bin’. This directory is where normal users will run terraform from. Placing the ‘terraform’ executable file will make it possible to call it from anywhere in the system. This is possible because the directory automatically has a path export.

$ sudo mv terraform /usr/local/bin/terraform

2. Test if terraform is installed correctly by running the following command. If the version of terraform is outputted, then this means the executable is correctly linked and can be found.

$ terraform -v

5. Creating your first Terraform Project

Go back to the table of contents

This chapter describes how a simple Terraform project can be setup where a small test code will be used to test Terraforms functionality. This test configuration will print ‘hello world’. After that a guide on how Terraform is used to manage infrastructure will be discussed with a simple example where files will be managed. The purpose of this whole chapter is to install, and test Terraform to display some of its code syntax and usage.

This chapter describes how a simple Terraform project can be setup where a small test code will be used to test Terraforms functionality. This test configuration will print ‘hello world’. After that a guide on how Terraform is used to manage infrastructure will be discussed with a simple example where files will be managed. The purpose of this whole chapter is to install, and test Terraform to display some of its code syntax and usage.

1. Create a new directory and go into that directory. This directory is where we will place configuration files needed for creating a new infrastructure.

$ mkdir ~/terraform_project/
$ cd ~/terraform_project/

2. Initialize the project for Terraform using the command below. This will make terraform search for configuration files with an extension ‘.tf’ in the directory it’s called in. In this case the directory being ‘~/terraform_project/’. Furthermore, this will generate some hidden files in our workspace that will be used by Terraform to save and make changes in the infrastructure.

$ terraform init

Create a new Terraform configuration
This section covers a part of the syntax of the Hashicorp scripting language which is used for creating Terraform code. A terraform configuration file is going to be created with which ‘hello world’ is going to be printed. After that you will learn how to read a file. This will be useful later for loading configuration files.

1. Go into the project and create a new file using your favorite editor. For terraform to recognize a new configuration file, the file must have a ‘.tf’ extension.

$ nano test.tf

2. Copy and paste the code below in the opened file and save it. Note that the syntax requires the curly brackets placed as shown below. This is also called a scope. The code will output the value of the ‘text’ as a string. The quotes behind output saying “text” can be changed to whatever the user prefers, if it isn’t defined earlier in the code.

output “text” {
 value = “hello world”
}

3. Run terraform with the new configuration after initializing the project. The output should be same as output below.

$ terraform apply

4. It is also possible to read files. Do this by editing the test.tf file and adding the code below. This code will read a file from a given path and put its content as a string in output. The contents are read using the file(“/example_path/”) function that can be found within the curly brackets. The function requires a path. As you can see below, it expects a file in our project workspace. Note that ${} is used to call functions within a string. In this case its a function that returns a path to the directory of the test.tf file. Besides these functionalities note the hashtag (#). This is used to write comments in code and is negated when running the code.

# test.tf
output “text” {
 value = file(“${path.module}/text.txt”)
}

5. Save the test.tf file and create the text.txt file that has to be read. Make sure to add the text and save this file as well.

$ nano ~/terraform_project/text.txt
Output: Hello world from text

6. Run the following command again to see the output.

$ terraform apply

Managing resources with Terraform
As described throughout this document, Terraform is used to manage the life cycle of various resources of cloud Infrastructures. Resources are nodes which can be storage disks, virtual machine, network interfaces and many others. In this section we will assume that files are the resources we want to manage. By creating and destroying files using Terraform we can learn more about the use of Terraform.

A resource in Terraform code is written as follows:

resource “<resource_type>” “<resource_name>” {
}

The “<resource_type>” corresponds to a type of resource which must be provided by a sort of provider. Since we do not have providers yet, we will use a resource of type “null_resource”. This resource type corresponds to the “null” providers and is automatically installed. This enables Terraform to track the lifecycle of an unknown object (“HashiCorp, Null Resource”, 2019f). The “<resource_name>” is a user defined name. This name must be unique in the scope of your code. Lets say its called, “node1”, then this would mean that no other resource of type “null_resource” with name “node1” may exist.

We will use the description above to create a simple example resource that creates a file using local provisioners. These are functions that enable us to call commands in our host machine from code.

1. Go into the project

$ cd ~/terraform_project/

2. Edit the test.tf file. In this file we will define a new resource.

3. Remove any other existing code in the file. Add the code below to the file. This will make it possible to create a resource.

# test.tf
resource “null_resource” “node1” {
 # add provisioners here
}

4. It is already possible to run Terraform now but this will of course do nothing other than manage a “null_resource”. To make this useful we will add provisioners. Add the following code below the “# add provisioners here” so that its within the scope of “node1”. Go to the next step for explanation.

# add provisioners here
 provisioner “local-exec”{
 command = “echo >> ${path.module}/node1.txt”
 }
provisioner “local-exec”{
 command = “rm ${path.module}/node1.txt”
 when = destroy
 }

5. Now two local provisioners are added. These, as described before, are functions used to execute commands on the host machine. The‘provisioner’ functions used here are of type “local-exec”. It expects a command which it will run on an interpreter, on default being ‘/bin/sh’ on ubuntu OS. In the case of the example, the first function runs a command to create a new text file called ‘node1.txt’ in the directory of our ‘test.tf’ file and the second function runs a command to remove a file called ‘node1.txt’ in the directory its called from. Note that the second function is only called when Terraform destroy is called. This is configured with ‘when = destroy’

6. Save the file and run “terraform init”. You will see in the output that Terraform will search for the provider “null” to create the type “null_resource” since we didn’t install it before in our project.

$ terraform init

7. Apply the Terraform configuration with “terraform apply”. You will see that a new resource is going to be added. Type ‘yes’ to confirm this.

$ terraform apply

8. Now search for the created file “node1.txt” in the project directory with the command below to verify if the creation of the file was successful. The command will also list some other files and a hidden directory. These items (.terraform/, terraform.tfstate and terraform.tfstate.backup) are used by Terraform to manage the life cycle of the resources. Deleting these will result in Terraform not knowing which resources were created, consecutively meaning you will have to remove all resources manually if so needed.

$ ls -al ~/terraform_project/

9. Finally, destroy the resource using ‘terraform destroy’. You see that a single resource is marked for destruction. Type ‘yes’ to confirm the destruction.

$ terraform destroy

In this chapter you learned how to use Terraform to manage a single resource. Terraform code is most useful when multiple resources that need configuration comes into play. Furthermore, no actual resource providers where used in this chapter. In the chapters from now on, the concept of creating and destroying infrastructure with Terraform will be used a lot.

6. Installing KVM and Terraform provider for libvirt

Go back to the table of contents

Now that Terraform is installed and some tests are done with it. Further steps can be taken so that a virtual machine can be provisioned. To achieve this, the KVM hypervisor will be used. Furthermore, the interaction with the virtualization library (libvirt) used to manage KVM needs to be added as a provider in Terraform.

Download/Install KVM and enable libvirt-service
KVM is a standard downloadable package for ubuntu 18.04. There are several packages that KVM is depends on. The main package is qemu-kvm and libvirt-bin. These packages are required for virtualization. This chapter will guide you through the installation of KVM and its dependencies.

1. Start a console and update your system to get new package sources.

$ sudo apt update

2. Download/Install KVM and the required packages with following command. This will install a lot of packages and could take quite some time. Besides the KVM packages some other useful tools will be installed. Bridge-utils is useful for debugging issues in a network, mainly a bridge network. Virtinst is a tool that can be used to create virtual machines libguestfs-tools has some dependencies for cloud init.

$ sudo apt -y install qemu-kvm libvirt-bin virt-top libguestfs-tools virtinst bridge-utils

3. Enable the “vhost_net” kernel module. This module allows libvirt to directly call into subsystems instead of using calls from userspace. This will mainly increase the performance of the virtual machine and is not a prerequisite to use the libvirt service.

$ sudo modprobe vhost_net

4. Verify if the “vhost_net” module is loaded using ‘lsmod’, which will list the currently loaded modules, and ‘grep’(‘Global Regular Expression’) to find the ‘vhost_net’ module in the module list. Run the following command and compare the output and verify if the module is activated.

$ sudo lsmod | grep vhost

5. Start the libvirt service and enable auto starting on next startup using the following command.

$ sudo systemctl enable — now libvirtd

6. Verify libvirt status by running the following command and searching for active (running).

$ sudo systemctl status libvirtd

7. Install virt-manager with the command below. This is a GUI program which can be used to manage various virtual machines created with libvirt. You do not need to install this program if running from console only.

$ sudo apt -y install virt-manager

8. Restart your host machine.

Install GOlang and Git
Golang is an open source programming language made by Google (“Google, The Go Programming Language”, 2019). It is known for its convenience as it provides cross-platform support. For these instructions it will be used to build various terraform providers from GitHub. We only use the GO compiler to build and install the plugin.

1. Run the following command to install GOlang and Git.

$ sudo apt -y install golang git

Install Terraform libvirt provider
To be able to interact with KVM using Terraform a provider is required. The provider will handle our terraform configurations and call the required functions of the libvirt library which will then tell KVM to do certain tasks. The provider will be installed from Github like other providers. The installation is also very similar to other providers. Before doing this some packages specific to terraform-provider-libvirt will need to be installed.

1. Install libvirt-dev dependency for installing terraform-provider-libvirt with go.

$ sudo apt -y install libvirt-dev

2. Export GO paths using the following commands. These are directories the provider will search for so that it can be built. The function ‘export’ is a function that can be used to export certain paths for your current shell. Many paths already automatically added on startup from ~/.bashsrc.

$ export PATH=$PATH:/usr/lib/go/bin
$ export GOPATH=/usr/lib/go/

3. Create and go in to a new directory within GOPATH’s directory using the command below. The format described in the following code seems to be a standard of installing terraform providers as the provider will search for the build files in the directories shown below.

$ sudo mkdir -p $GOPATH/src/github.com/dmacvicar/
$ cd $GOPATH/src/github.com/dmacvicar/

4. Clone the official git repository.

$ sudo git clone https://github.com/dmacvicar/terraform-provider-libvirt.git

5. Go into the provider’s directory and build it with the following commands. The ‘make’ command will compile the provider and ‘make install’ will install it in our GOPATH. This will take up to 30 seconds.

$ cd terraform-provider-libvirt
$ sudo make install

6. The result of the build is an executable which is saved in $GOPATH/bin. Go into this directory and check the version with the following commands. The output should be similar to the picture below.

$ cd $GOPATH/bin/
$ ./terraform-provider-libvirt -version

Add the provider as a plugin
We want to use libvirt as our Terraform provider. The plugin needs to be specifically added into ‘~/.terraform.d/plugins’ directory. This is the directory in which terraform will search for third party plugins when initialized.

1. Make a new directory called ‘plugins’ in ‘~/terraform.d’ and link the ‘terraform-provider-libvirt’ executable from ‘/usr/lib/go/bin/’. Again, command ‘ln’ is used to prevent duplicate executables in our system. Furthermore, when the actual file in ‘/$GOPATH/bin/’ is changed no further step will have to be made.

$ mkdir ~/.terraform.d/plugins
$ sudo ln -s /usr/lib/go/bin/terraform-provider-libvirt ~/.terraform.d/plugins/
$ ls -l ~/.terraform.d/plugins/

7. Create a Virtual Machine using Terraform and KVM

Go back to the table of contents

In this chapter, instructions will be given on how to create a new virtual machine, allocate computing resources to support the virtual machine and finally configure the virtual machine using the terraform-libvirt-provider plugin for terraform.

The configurations are mainly the resources we want to put in to the machine such as number of cores and size of the memory. Besides those, the users in the machine will have to be configured for us to login after creation. To this end we will be using SSH keys. After creating the machine and login in, a test server which can calculate Pi to a given precision will be run on the created virtual machine to show its capabilities.

Setup permissions
First we need to make sure the libvirt service has the rights to create, read and write user files so that space for the virtual machine can be allocated.

1. Libvirt requires the host machine’s user rights to be able to allocate computing resources. Run the following command to add your user to groups libvirt and KVM. Change ‘ubuntu’ to your host’s user.

$ sudo usermod -aG libvirt,kvm ubuntu

2. The groups are not yet enabled for our current section. Re-login to your user with the ‘su — ${user}‘ command. After that list the groups the user is a member of with the id command.

$ su — ubuntu
$ id -nG

3. Turn off SElinux security driver for libvirt by editing a file that can be found at ‘/etc/libvirt/qemu.conf’. The security driver needs to be turned off because it will prevent libvirt from creating a libvirt-domain. Open the file.

$ sudo nano /etc/libvirt/qemu.conf

4. Search with CTRL+W and change the line ‘security_driver=”selinux”’ to ‘security_driver = “none”’.

5. Exit and save with CTRL+X and press Y then Enter

6. Finally, restart libvirt service to make the changes effective

$ sudo systemctl restart libvirtd

Setup the project workspace
The project is a directory located at ‘~/kvm_project/’. The directories that are going to be added are images and downloads. These directories will be required later. The images directory is where new virtual machines will be saved. The downloads directory is where images will be cloned from.

1. Go into the project directory and create two new directories called ‘downloads/’ using the following commands.

$ mkdir ~/kvm_project/
$ cd ~/kvm_project/
$ mkdir downloads

2. Create a new directory to save images. This where the disks of the virtual machine will be found. Therefore it is required to put this in a place not easily accessible.

$ sudo mkdir -p /libvirt_images/

Download an image
An image of the OS you would like to run on the Guest machine is required. Any OS can be used. For this guide we’ll be using ubuntu 18.04 cloud image. We recommend using cloud type images as this could significantly reduce the time to download and also when cloning it.

1. Download the image to downloads directory in the project using the following command.

$ cd ~/kvm_project/downloads
$ wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img

Download URL: https://cloud-images.ubuntu.com/bionic/current/

2. The download is going to take a while. You can continue with the other steps by simply starting a new console.

Generate an SSH key
An SSH key will be needed to log in to the guest virtual machine. SSH keys come in pairs. A public key and a private key. The private key is never shared and saved under the ~/.ssh directory. The public key is also saved in directory ~/.ssh. Unlike the private key, the public key is meant to be shareable. When a user wants to login into a guest virtual machine with SSH, that machine will have to have the public key authorized. This is when the public key is added to a list of public keys in ‘~/.ssh/authorized_keys’ file of the guest machine. To create the keys a program called ‘ssh-keygen’ can be used.

1. Run the following command to generate the SSH keys. Press enter twice when asked for a passphrase. A password will not be required simply press enter. No options are specified as the required files and keys will be put in ~/.ssh/.

$ ssh-keygen

2. Look into the permissions of the created keys to verify if permissions changed correctly by running the following command and comparing the output. You will see two keys. One of the keys is a public key and a private key. The difference can be seen in the .pub extension of the public key. Since this is the one we are going to share it also requires read writes.

$ ls -al ~/.ssh/

Setup a terraform Configuration to provision a guest virtual machine
As discussed before in chapter 5, ‘.tf’ files are used in Terraform to configure an infrastructure. In this section a new terraform configuration file will be created. The goal of the code is to create a new virtual machine and allocate the resources needed to support the virtual machine. To create the VM, an image is required. This is an image downloaded in the first section of this chapter. Furthermore, the users in the to be created VM need to be configured as well. This will be done in the next section, but the call to the user configuration file will be done in this section.

1. Go into the project directory.

$ cd ~/kvm_project/

2. Create a new terraform configuration file using your favorite editor.

$ nano libvirt.tf

3. The first thing that is required is the provider ‘terraform-libvirt-provider’. To enable this in the code, a call needs to be added. This is done by adding the code lines below. This code will connect to QEMU Hypervisor. Do not close the editor after adding the lines, more instructions will follow.

# libvirt.tf
# add the provider
provider “libvirt” {
 uri = “qemu:///system”
}

4. Create a libvirt pool. This a quantity of storage set aside for use of virtual machine images. The name argument is used to identify a pool. The pool type ‘dir’ provides the means to manage files within a directory. These files can be a special disk format such as ‘qcow’ or ‘vmdk’. Finally, the path argument is where the storage is and where the images of the machine are going to be saved. As the path we choose ‘/libvirt_images/ubuntu-pool/’. When the code is run, this will create a new directory if not existing. (The default pool directory is “/var/lib/libvirt/images”. In the guide the directory “/libvirt_images/” will be used)

# create pool
resource “libvirt_pool” “ubuntu” {
 name = “ubuntu-pool”
 type = “dir”
 path = “/libvirt_images/ubuntu-pool/”
}

5. Create a new resource for the virtual machine image disk. Add the following lines to create a new image resource. This will create a new image by cloning it from a given source and saving it in the path to the pool created in the last step. The source can be a directory or even an HTTP URL to the actual download. Note that the given format should match the pool type.

# create image
resource “libvirt_volume” “image-qcow2” {
 name = “ubuntu-amd64.qcow2”
 pool = libvirt_pool.ubuntu.name
 source =”${path.module}/downloads/bionic-server-cloudimg-amd64.img”
 format = “qcow2”
}

6. Create a cloud init disk and load a cloud init configuration. This disk will be used to share user data found in the ‘cloud_init.cfg’ file with the guest VM. An image will be created and saved in the same storage pool as the virtual machine we are going to create. More explanation on the cloud init file will come in the next section where the cloud_init.cfg file will be created. Add the following lines.

# add cloudinit disk to pool
resource “libvirt_cloudinit_disk” “commoninit” {
 name = “commoninit.iso”
 pool = libvirt_pool.ubuntu.name
 user_data = data.template_file.user_data.rendered
}
# read the configuration
data “template_file” “user_data” {
 template = file(“${path.module}/cloud_init.cfg”)
}

7. Create a new libvirt domain. This where resources to support the guest machine are added. This is basically where the virtual machine is created and will determine the performance of the created machine but also the networking functionality. You can change the name of the memory and the vcpu depending on the resources available for you. The network used is the ‘default’ network created by libvirt, which already has DHCP functionality. It is recommended to leave the console settings as they are. More about this will come in the next chapter. Append the following lines to your configuration file.

# Define KVM domain to create
resource “libvirt_domain” “test-domain” {
 
 # name should be unique!
   name = “test-vm-ubuntu”
   memory = “1024”
   vcpu = 1
 # add the cloud init disk to share user data
   cloudinit = libvirt_cloudinit_disk.commoninit.id
# set to default libvirt network
   network_interface {
   network_name = “default”
 }
console {
   type = “pty”
   target_type = “serial”
   target_port = “0”
 }
disk {
   volume_id = libvirt_volume.image-qcow2.id
 }
graphics {
   type = “spice”
   listen_type = “address”
   autoport = true
 }
}

8. This script can now be used to create a new virtual machine. Save it and continue with ‘setup up cloud-init..’ section to make it possible to login into the virtual machine using cloud init configuration when its created.

Setting up cloud-init for configuring the guest virtual machine system on creation
In this section, cloud init will be used to configure a user on the guest virtual machine. A short description of what cloud init is:

“Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization. It is supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations.”

(“cloud-init, cloud-init 19.2 documentation.” 2019a)

First we will start by creating a file called ‘cloud_init.cfg’. This file is not automatically loaded by Terraform. To load the data a call needs to be made from a ‘.tf’ configuration file. In the last section, chapter 7.5 step VI, you can find the call to ‘cloud_init.cfg’. The syntax of cloud init is ‘yaml’. The syntax cannot handle tabs. Therefore, use only spaces to create a structure in your code. Instead of tabs its recommended to use two spaces. This is important to note as Terraform won’t handle the configuration faults of cloud init configuration.

1. Go into the project directory and create a new file called ‘cloud_init.cfg’.

cd ~/kvm_project/
nano cloud_init.cfg

2. Add the following code and heed the explanation below.

#cloud-config
users:
 — name: terraform_guest
 sudo: ALL=(ALL) NOPASSWD:ALL
 groups: users, admin
 home: /home/terraform_guest
 shell: /bin/bash
 ssh_authorized_keys:
 — ssh-rsa XXXX

3. This code is used to set the users of the virtual machine which we are going to create. Make sure to change all bold marked parts.

The name argument is the name of a user that is going to be added. Adding more name arguments will result in multiple users. Under the name argument the user data is to be set. Most importantly is the ‘ssh_authrozed_keys’ argument. This argument requires a list of public keys which are going to be added under the ‘authorized_keys’ in directory ‘~/.ssh/authorized_keys’ of the guest machine. This way we will be able to login into the user using our private key in the same directory.

To get the public simply run: cat ~/.ssh/id_rsa.pub on your host machine.

4. Lastly, we will have the cloud init download some packages for us when the virtual machine is created. Heed the configuration below. This will download the package and python and git.

# install packages
packages:
 — qemu-guest-agent
 — git

Provisioning the guest virtual machine
Now that the project is set up a new virtual machine can be created using Terraform. This section will run you through the commands to get the machine up and running.

1. Initialize the project workspace.

$ terraform init

2. Apply the changes with following command. Make sure to type ‘yes’ to confirm the changes. As you can see there are four resources that are going to be added. These are the libvirt pool, the libvirt disk, the cloud init disk, the libvirt volume. After applying Terraform will save these resources and remember them with the goal to manage the lifecycle of these objects.

$ terraform apply

3. After the creation of the machines is done. The machine consists of four resources as can be seen in the output. We would like to verify if all our resources are created correctly. We will go through a set ‘virsh’ commands. This is a tool used from command line to create and manage libvirt resources. First run the following command to list all virtual machines (Domains) created with libvirt. One of the entries in the Domains list has the same name as the one created with Terraform code. In this case ‘test-vm-ubuntu’

$ virsh list — all

4. There are a few other resources we created for the Domain to exist. These are the libvirt pool, libvirt volume and the cloud init disk. As discussed in chapter 7.5, the machine’s disk (the libvirt volume) and the cloud init disk need to be in the same pool to share cloud init configuration. To this end we created a pool called ‘ubuntu-pool’. Run the following command to list existing pools.

$ virsh pool-list — all

5. To list the existing volumes run in a certain pool run the following command. As you can see these are the ‘libvirt_volume’ and ‘libvirt_cloud_init_disk’ we defined in the code in chapter 7.5. They are also in the directory we wanted them to be. This directory being ‘/libvirt_images/ubuntu-pool/’.

$ virsh vol-list ubuntu-pool

6. List the volume files in directory ‘/libvirt_images/ubuntu-pool’.

$ sudo ls /libvirt_images/ubuntu-pool/

7. Now we are sure the code is working as expected we can proceed to try and login. The only way to login into the machine we created is using SSH. To this end we need to know which network the Machine is in and what IP address it has. List all libvirt networks with the command below. This will list the ‘default’ network provided by libvirt. We configured our Virtual machine would connect to the default network in chapter 7.5 under the network interface of the Domain.

$ virsh net-list –-all

8. Finally try pinging the guest machine. First find the IP-address DHCP leased to it. This won’t always happen immediately, since the machine could still be starting up. Use the following command, it requires the network name. We set this to ‘default’ in chapter 7.5. The network can of course also be found using the name gotten from the last step.

$ virsh net-dhcp-leases default

9. Ping the ‘IP-address’ found in the last step. We will use this to make user if the guest can find the host. Note down this IP address as it will be used in the next section.

$ ping 192.168.122.79

Running a test server on the guest virtual machine
In the last section a virtual machine was created. There are two ways to login that we configured. One of them being only using the username and password and the other way is using the SSH key configured in the cloud init section of this chapter. We will be using the latter method to login into the machine to download, install and run a test program called PiCalc.

PiCalc is a HTTP server with which Pi can be calculated to a given precision. This will be achieved using the Python math library named mpmath and the web framework, Bottle. This is a lightweight web framework for python with no dependencies. The program will be downloaded from Github:

https://github.com/krebsalad/PiCalcPy.git

1. Start a terminal emulator.

2. First the IP-address of the virtual machine is required to login with SSH. Run the following command on the host machine to see leased IP-addresses.

$ virsh net-dhcp-leases default

3. Login to the guest VM using the guest’s username configured in the cloud init and the IP-address gotten from the last step. Make sure to terraform_guest matches a user made under the cloud init section of this chapter. Also, the <192.168.122.79> should be the guest machine’s IP-address.

$ ssh terraform_guest@192.168.122.79

4. Check if git is installed as we specified in the cloud_init file and python in the guest machine if not already installed. These are going to be used to create the python server. You will notice that git is already installed. This thanks to the cloud init package update argument we added in the cloud init section.

$ dpkg -s git | grep Status
$ dpkg -s python | grep Status

5. Clone the PiCalc server from Github. This has a python project which will be used to install and run the PiCalc server.

$ git clone https://github.com/krebsalad/PiCalcPy.git

6. Install the dependencies of the server by going in ‘PiCalcPy/’ and running the commands below. This will install mpmath, a math library and bottle. Bottle, as mentioned before, is very light weighted as it doesn’t require other dependencies than python already has. This can be seen in the fact that it only requires a single python file to run. Mpmath will need to be installed which requires sudo rights.

$ cd PiCalcPy/
$ sudo python install_picalc.py
$ ls -a

7. Open the file called ‘picalc_server.py’. This file contains the code to run a service.

$ nano picalc_server.py

8. Heed the contents of ‘picalc_server.py’. The contents can also be found below. This script will wait for an HTTP request to the route http://<IP>:<port>/PiCalc/<p>. The IP is set in the last line ‘run(host=ip_addr, port=8080)’.The p stands for precision and is used to determine how many decimals of PI we want. The script will automatically get your current IP using a standard library of python called ‘socket’.

# picalc server.py
from bottle import route, run, template
from mpmath import mp
import socket
import time
hostname = socket.gethostname()
ip_addr = socket.gethostbyname(hostname)
@route(‘/PiCalc/<p>’)
def index(p):
 # set time
 start_time = time.time()
# read precision value
 precision = int(p)
# handle non ints
 if(str(type(precision)) != “<type ‘int’>”):
 print(str(type(precision)))
 return str(“error, precision was not an int”)
# set precision val and calculate pi
 mp.dps = precision
 pi = mp.pi
# end time
 processing_time = time.time() — start_time
# return txt
 return str(“Server ip: “+ip_addr+ “\nTime took: “+ str(processing_time) + “ seconds\nPI = “ + str(pi) + “\n”)
# function to run picalc on port 8080 by default
def run_server(port_num=8080):
 run(host=ip_addr, port=port_num)

9. Exit the editor and run the test program with the following command.

$ sudo python run.py mode=server

10. Send a request to the server from your host machine by starting a new terminal and entering the following command. (You can also send a request via browser if you have GUI on your host machine. Simply type in the following command in the browser without ‘curl’)

$ curl 192.168.122.79:8080/PiCalc/100

11. Now we understand how to provision a virtual machine we can go more in depth on how to provision multiple virtual machines. For now, exit the SSH session on the guest machine by typing ‘exit’.

12. Finally, destroy the domain and its resources with the following command on the host machine. To do this correctly it is required to be in the directory terraform was initialized in. In our case ‘~/kvm_project/’. Type ‘yes’ when requested to delete everything.

$ terraform destroy

8. Provisioning and Orchestration

Go back to the table of contents

This chapter will cover different aspects of cloud technology such as automatic creation of servers, also called provisioning, and combining different virtual machines to create a single service, a form of orchestration. This will be done using an example where a total of three virtual machines will be created. Two machines will provide a web service while one will load balance requests for this service. This means that all requests will go through the machine with load balancer, which will then decide on a machine which will process the request in round robin style.

Figure 4 — Architecture Provisioning Multiple VMS (made with Archi)

See the architecture above. This is a variant on the architecture used in chapter 2 but displays more about our current situation. As can be seen, the goal is that the user (student) or any other can send HTTP requests to the guest machine. This request will firstly be available in your machine at the network interface card. The NIC will then search based on the destination of request where the packet must be sent. The destination network should already be known by hardware with the help of the hypervisor as it creates a so called ‘bridge’ between the NIC of the host machine and the virtual network interface of the hypervisor. On this bridge Network Address Translation (NAT) will be enabled so that the virtual machines can be accessed from the host network. Furthermore, the bridge will function as a middelway for Domain name solving (DNS) and will provide ip addresses for the virtual machines using Dynamic Host Configuration Protocol (DHCP). In this chapter we will try to achieve this by provisioning all this from Terraform code. This will show a good example on how Terraform can be used to create such cloud infrastructure with a simple load balancing example.

The first section of this chapter will cover how to provision a virtual machine that runs a server on boot. The same PiCalc program will be used to test the virtual machine. After that, in the second section, more about the virtual network that libvirt provides will be discussed where a custom bridged networking will be created. The third, fourth and fifth section will teach on how to use modules which can be reused from a centralized place with the goal to make the configuration more manageable. Finally, the sixth section will combine all the information to create multiple virtual machines that load balance the PiCalc service.

To follow the instructions in this chapter it is prerequisite that chapter 7 was done successfully because in this chapter, we will build further on top of the code created in that chapter. We will be adding some code in these files or moving the said files.

Provisioning a server with a virtual machine
To be able to create the eventual goal of this chapter, a load balanced PiCalc service, it would be useful if the PiCalc server in the machine is automatically created and run. This way we won’t have to log into the virtual machine to start the server. To achieve this some edits will have to be done in the cloud_init.cfg file. We will be adding some packages which the PiCalc server requires to install and run but also we will be adding configuration to run the server after installation.

1. Edit cloud_init.cfg and the following yaml configuration. This will attempt to install packages after the machine is created. This functionality was displayed before, in chapter 7.8, where git was installed on the virtual machine. Note that we added python wget, two programs used to install the PiCalc service.

# cloud-init configuration
# install packages
packages:
 — qemu-guest-agent
 — git
 — python
 — wget

2. To start the server after installing, we will have to run some bash commands. Fortunately, cloud init also can do this. Add the configuration below as well. As you can see this will run the same commands we used in chapter 7.8 to create the PiCalc server.

# cloud-init configuration
# run command after boot
runcmd:
 — [“cd”, “/home/terraform_guest”]
 — [“git”, “clone”, “https://github.com/krebsalad/PiCalcPy.git"]
 — [“cd”, “PiCalcPy”]
 — [“python”, “install_picalc.py”]
 — [“python”, “run.py”, “mode=server”,”&”]

These additions should be enough to run the server automatically when it’s started. Test this by initializing Terraform in the project and applying the configuration. When the created machine is booting the packages in step I will be installed. When this is done, the installation of the server should automatically start, where after it will run. If you want to see the output of all the installation until the moment it starts, you can go into the virtual machine’s console using the commands below. (recommended to start a new terminal for this). As shown before, the first command will list the machines on your system. Note down the id of the machine you want to log into and replace the number behind the ‘virsh console’ by the id of the machine. Now you will be able to see the output of the installation.

$ virsh list — all
$ virsh console 2

When the installation is done, test the server by sending a HTTP request to the IP address from the CLI or from your host’s browser. You can find the ip with which you can find with the ‘virsh net-dhcp-leases default’ command.

$ curl VM_Guest_ip:8080/PiCalc/100

When done testing the server, make sure to destroy the created infrastructure before continuing with the next instructions as we will be building changing more configuration.

Creating a libvirt bridged network
In this section, a virtual bridged network will be created using libvirt with Terraform. A bridge is a networking device that provides the ability to combine several communication networks into a single network. The network device will be used to provide IP addresses to the virtual machines we are going to create. This will be done using DHCP. Heed the sketch below where a scenario that is going to be created in this section can be found. Here can be seen that there is a virtual network and a guest machine that is connected to the network. The bridge will provide an IP address in a given rang the toe machine. The bridge can communicate with the outside network via the host. This is possible using NAT. All the communications on the outside will see the virtual network as the host machine.

Figure 5 — Bridged network made with draw.io

Until now we have been using the ‘default’ network that libvirt provides. This network is also a NAT based bridged network with DHCP. We will be basing our configuration on the ‘default’ network with the goal to understand more about this before we go more in depth in the next sections.

The libvirt network is defined using XML files. The syntax of XML requires that you define sections as following </section>. Note that the / at the beginning is used to end the tag. If it is not present all the sections below will be a member of that section. To end this behavior, repeating the section and closing it will also work. For example:

<section> arguments </section>

Fortunately, libvirt will look for errors in our syntax, which will help in debugging faults in our configuration. Look into the configuration of the default network by running the command below. This will print the configuration of the network.

$ virsh net-dumpxml default

This configuration is in XML format. Copy the output, as we will be using this configuration to create our own network. Create a new file in your project directory called ‘libvirt_network_config.xml’ and paste the configuration of the ‘default’ network copied from the last step. This file will be used to define a new network. We will have to edit some values so that it works for our new network.

$ nano ~/kvm_project/libvirt_network_config.xml

The first XML tag we will have to change is the name of the network. This is between by the tags ‘<name>’ and ‘<name/>’. Change the name ‘default’ to ‘picalc_net’. The line should look as following

<name>picalc_net</name>

Now remove the <uuid> tag entirely. This tag is generated by libvirt when the network is created.

The <forward mode = ‘nat’> tag does not need to be changed. This will provide Network Address Translation (NAT) for our network. This is required to communicate with other virtual networks in our host machine and therefore also needed to be able to communicate with the internet.

The <bridge name..> must be changed. The bridge is the network interface where all ingoing and outgoing packets of the network go through. The option that needs to be changed is the ‘name=’ option as it is used to identify bridges and should be unique in your machine. This will have to change from ‘virbr0’ for example ‘virbr1’. Besides this option, there is the ‘stp=’on’’ tag. Make sure this is on, as it helps to prevent any loops in the network, but will also have the packets choose the most reliable path to its destination. We don’t need to change this option.

<bridge name=’virbr1' stp=’on’ delay=’0'>

The <mac address…> tag is used to identify the virtual network interface on machine level. The address itself must be unique. This means that the mac address that we copied from default will have to change as well. Change the mac address to a valid mac address. Heed the example below.

<mac address=’52:54:00:6c:3c:01'/>

The <ip address…> tag is used to set an IP address for the network interface. This IP should be the gateway address for the machines in your network. Furthermore, the netmask must be a valid mask. In our case we will be using the same netmask as the default network, which is /24, but we will be changing the subnet of the network to network 192.168.180.0. Change the tag as follows.

<ip address= ‘192.168.180.1’ netmask=’255.255.255.0'>

As you can see there is also the child member tag <dhcp> within the IP address tag. This tag will provide an IP address for all clients in the network we defined. It is important that the ranges in this tag are valid addresses within the subnet we defined. In our case this will look as following.

<dhcp>
   <range start=’192.168.180.2' end=’192.168.180.254'/>
</dhcp>

Lastly, we will configure a DNS server. This is required because the virtual network within our host will not be able to solve domain addresses because its hiding behind a bridge interface. Fortunately, the interface itself is within our host machine and can find a DNS server. This means the clients in our network will be able to solve domain names by simply using the network interface. The only thing we will need to configure is that the resolve request from our clients in the virtual network get passed on by the interface. This is done by defining our network interface IP as a DNS server.

<dns enable=’yes’/>

Now that the configuration is done. Save the file, define and start the network using the following commands.

$ virsh net-define ~/kvm_project/libvirt_network_config.xml
$ virsh net-start picalc_net
$ virsh net-list — all

To test the network, you can make a virtual machine using the project ‘~/kvm_project’. But before initializing some changes will have to be made in the Terraform configuration so that the guest machine’s network interface is connected to the virtual network we called “picalc_net”. This was discussed briefly in chapter 7 step VII. To change the network in the configuration, edit the ‘libvirt.tf’ file and change the ‘network_name = “default”’ under ‘network_interface’ in the ‘libvirt_domain’ resource to ‘network_name = “picalc_net”’.

Finally, initialize and apply the Terraform configuration. When the guest machine is created. Run the command below to get the DHCP leased IPs. The leased IP should be within the range that was defined earlier which is 192.168.180.2/24 to 192.168.180.254/24.

$ virsh net-dhcp-leases picalc_net

Make sure to destroy the machine with Terraform when done testing. Do not delete the network configuration file as this will be used in the next section and further.

To stop a running libvirt network run the following command.

$ virsh net-destroy picalc_net

To remove an existing network which is inactive, run the following command.

$ virsh net-undefine picalc_net

To search for existing networks, active or inactive, run the following command.

$ virsh net-list — all

Creating a custom Terraform module for a libvirt network
In this chapter, Terraform will be used to create the same network in the last chapter. This is so that the network’s life cycle can be managed using Terraform. The terraform-provider-libvirt does provide a libvirt network resource, but it does not support the use of static hosts for DHCP as of October 2019, which we will need to provision the load balanced PiCalc service.

Get Nitesh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To be able to create a network from a file and manage its life cycle we will have to create a module. A module is a directory in which configuration files of Terraform are placed. A module requires at least one file with the ‘.tf’ extension. This module will then be callable from another Terraform configuration files by calling the directory.

Create a directory ‘/modules/’ in your project and then create a new module directory within the modules directory called ‘/libvirt_picalc_network/’.

$ mkdir -p ~/kvm_project/modules/libvirt_picalc_network/

Move the network configuration XML file we created in the last chapter(8.2) to this new directory. We will be using this network configuration file to create the same network from Terraform code.

$ mv ~/kvm_project/libvirt_network_config.xml ~/kvm_project/modules/libvirt_picalc_network/libvirt_network_config.xml

Add and edit a new file called ‘picalc_net.tf’ in the ‘/libvirt_picalc_network/’ directory. This file will have the configuration needed to define and run the network but also destroy and undefine the network when instructed.

$ nano ~/kvm_project/modules/libvirt_picalc_network/picalc_net.tf

Add the code below. This is to make sure libvirt is running before the network can be created.

# make sure qemu is running
provider “libvirt” {
 uri = “qemu:///system”
}

To have terraform do the required calls when ‘apply’ or ‘destroy’ is called we have to create a resource. To this end Terraform provides a resource called ‘null_resource’ this resource has a unique ID which Terraform will manage. Within the resource a ‘provision’ of type ‘local-exec’ is added. This is a useful function that can be used to execute commands locally. There is also the option to choose which interpreter is used. We will use the standard ‘/bin/bash’ to call some virsh commands. Note that there two provisioners. One of the provisioners will be run when ‘terraform apply’ is run and the second one when ’terraform destroy’ is run.

# let terraform manage the lifecycle of the network
resource “null_resource” “picalc_network” {
 
 # when terraform apply
 provisioner “local-exec” {
 command = “”
 interpreter = [“/bin/bash”, “-c”]
 }
# when terraform destroy
 provisioner “local-exec” {
 when = “destroy”
 command = “”
 }
}

Add the following command in the provisioner that is used when applying the terraform configuration. This is a one-line command that combines multiple commands for defining and starting a libvirt network. The ‘&&’ means that the following command will only execute when the command before was successful. The commands used where discussed in chapter 8.2.

command = “virsh net-define ${path.module}/libvirt_network_config.xml && virsh net-autostart picalc_net && virsh net-start picalc_net”

Now add some commands to destroy the network. Again, these are the commands used earlier in chapter 8.2 to undefine an existing network but now in one single line. Use the following line to do just that.

command = “virsh net-undefine picalc_net && virsh net-destroy picalc_net”

Save the configuration file. The final structure of the project should look as following

To test the module, we can simply initialize and apply terraform in the directory of the module. This will not do anything to the project’s configuration.

$ cd ~/kvm_project/modules/libvirt_picalc_network
$ terraform init
$ terraform apply

Verify if the network is created with the following command

$ virsh net-list -–all

If you want to test the virtual machine on this network, go into the project directory init and apply the changes. Note that the network name of the virtual machine’s network interface should be picalc_net. This was discussed in the last chapter as well and should already be configured.

To destroy the network, call ‘terraform destroy’ in the directory ‘libvirt_picalc_network’. In this chapter we only created the module from it directory. But it is also possible to call it from different files. This method will be discussed later.

Configuring guest machines with static IP’s
The eventual goal is creating a service of multiple virtual machines, where one of the virtual machines load balances requests sent to the other virtual machines. To achieve this, the load balancing machine needs to know the IP addresses of the said machines. As you could have noticed, the IP addresses are randomly given within a given range by the DHCP server. This would mean that even if we provision a machine with load balancing services, it won’t immediately know which IP addresses it should load balancer for and therefore we would have to manually search the IP and connect it to a load balancer.

Fortunately, DHCP has the option to assign fixed IP addresses to hosts based on MAC addresses. In this chapter we will use this behaviour to statically set the IP addresses of the virtual machines we are going to provision by assigning MAC addresses before creating the machines. The sketch below displays the situation we want to configure.

Figure 6 — DHCP static hosts based on MAC addresses made with draw.io

Using the hosts functionality in DHCP we can have a client with a fixed MAC address get a predefined IP. In the case of provisioning it would mean that the MAC address should be set statically when the virtual machine is created if we always want the virtual machine to get the same IP address. This chapter will cover this by configuring the MAC address in Terraform code.

To do this section correctly, knowledge on how to create a libvirt network is required. Heed the chapters 8.2 and 8.3 for these topics. It is assumed that the same configuration created in chapter 8.3 is used.

1. Edit the ‘libvirt_network_config.xml’ file.

2. Add the the bold lines below within the DHCP tags. This will ensure that the IP addresses 192.168.180.102, are leased to only the machines that match the given MAC address, which in this case is 52:54:00:6c:3c:02.

<dhcp>
   <range start=’192.168.180.2' end=’192.168.180.254'/>
   <host mac=’52:54:00:6c:3c:02' ip=’192.168.180.102'/>
</dhcp>

3. The virtual machine also needs to be set to the MAC addresses above. This can be done within the terraform configuration file used to create the virtual machine. In this document, until now, it was called libvirt.tf. Edit this file and add the bold line below under the ‘network_interface’ in the ‘libvirt_domain’ module. The MAC address added here should match the host mac address added in the last step.

network_interface {
network_name = “picalc_net”
mac = “52:54:00:6c:3c:02”
}

4. Now test the project. Make sure the picalc_net we created in the last chapter is running. After that test this configuration with a virtual machine by initializing and planning in the main project workspace.

5. Verify if the IP is leased as expected with the following command.

$ virsh net-dhcp-leases picalc_net

Using modules and variables
This chapter will describe how to create a Terraform module for provisioning a virtual machine. The goal is to create a template with which different ubuntu machines can be created by simply changing values. With the use of variables it will be possible to reuse the module and set them up from a centralized place.

In this chapter we will be using the ‘libvirt.tf’ file used in earlier chapters to create a virtual machine. We will be making this file into a module that is reusable. Therefore it is a prerequisite to have done all the instructions in this chapter, before doing this section so that the guide can be followed accurately.

Create a new module directory called ‘ubuntu-module’. This directory will have the required files to create a virtual machine.

$ mkdir -p ~/kvm_project/modules/ubuntu-module/

Move the file ‘libvirt.tf’ to the module directory. We will be changing the ‘libvirt.tf’ file so that its reusable.

$ mv ~/kvm_project/libvirt.tf ~/kvm_project/modules/ubuntu-module

The tree structure of the kvm_project should look as following:

Edit the ‘libvirt.tf’ file with your favourite editor. We will start with adding variables. Variables are used to set values. These variables can be set from other Terraform modules besides the file it self. It can also be done from other ‘.tf’ files. Heed the following code. This code defines a variable called ‘machine_name’ . The variable we defined is of type string. This can be defined using the ‘type’ argument. The description argument is useful when many variables are required. It should describe what the variable will be used for. Add the variable definition below, to the top of ‘libvirt.tf’. This variable will be used to set various values of the file.

#libvirt.tf
variable “machine_name” {
}

When reusing a module, variables are useful as there are some values that need to be unique so that the module is reusable. In the ‘libvirt.tf’ file there a number of values will have to be changed using variables. We will cover all these values below that have to be unique in the file.

First, start with the ‘libvirt_pool’. The name of the pool must be unique when creating multiple machines. Furthermore, the directory in which the pool will save its images in must be unique. To use a variable ‘var’ argument is used. This provides access to all variables in our current module. Using this we can set the values from a different file. Take the example below. If the variable ‘machine_name’ was set on initialization to ‘ubuntu_1’, then the name of the pool would be ‘ubuntu_1_pool’ and the path would be ‘/libvirt_images/ubuntu_1_pool’.

resource “libvirt_pool” “ubuntu” {
 name = “${var.machine_name}_pool”
 type = “dir”
 path = “/libvirt_images/${var.machine_name}_pool/”
}

The second resource that has to be changed in the ‘libvirt.tf’ file is the ‘libvirt_volume’. Heed the example below. The only value that needs to be dynamic is the name value since its used to manage the life cycle of the machine and is the name of the image file. Changing on the basis of a variable will help prevent name collisions. We will change the path value as well. This is because we moved the directory of the module. Because of this, the path.module function won’t return the path to our download. Change the values as shown below

resource “libvirt_volume” “image-qcow2”{
 name = “${var.machine_name}_image.qcow2”
 pool = libvirt_pool.ubuntu.name
 source = “${path.module}/../../downloads/bionic-server-cloudimg-amd64.img” 
 format = “qcow2”
}

The third resource that must be changed is the ‘libvirt_domain’. Heed the example below. The name should be changed again as it must be unique. It is possible to create new variable for this resource. For example, creating variables for setting the memory.

resource “libvirt_domain” “test-domain” {
   name = “${var.machine_name}_domain”
   memory = “1024”
   vcpu = 1
   …

Before editing the more components, we will create three new variables which will be used to set the network interface of the ‘libvirt_domain’ and the user configuration of the machine. As described in the last chapter we can set the MAC address of the network interface so that our machine gets a fixed IP. To make this possible for multiple machines, it will be useful to have a variable with which we can set the MAC address and network. Besides the variables for the network interface, it is also important to be able to set different user configurations for the virtual machines. To this end we will use the variable ‘user_data_path’. Add the following variables.

variable “network_name” {
}
variable “mac_address” {
}
variable “user_data_path”{
}

Now use the two newly added variables to set the ‘network_name’ and ‘mac’ of the ‘network_interface’ which can be found under the ‘libvirt_domain’. Heed the example below

network_interface {
   network_name = “${var.network_name}”
   mac = “${var.mac_address}”
}

The ‘user_data_path’ variable will have to point to the ‘cloud_init.cfg’ file. To this end, we have to change the place where the user data file is loaded so that it can be loaded from a variable path. Heed the example below

data “template_file” “user_data” {
   template = file(“${var.user_data_path}”)
}

Finally, save the ‘libvirt.tf’ file and close. The module can be reused without name conflicts. To make use of the module a new file will be created in the main directory of the project. This file will be used to call multiple modules, therefore it will be called main.tf. Go into the directory and create the file.

$ cd ~/kvm_project/
$ nano main.tf

In this file we will call the module and set the variables we added earlier. Heed the example below. The same values we used in earlier chapter to set the mac address and network are reused here. We also set the machine name to “ubuntu_1”. The path to the cloud init is in the directory of this file. This means we can use ‘path.module’ function to get our current directory.

module “ubuntu-module-1” {
   # load the module
   source = “./modules/ubuntu-module/”
   # set the variables
   machine_name = “ubuntu_1”
   network_name = “picalc_net”
   mac_address = “52:54:00:6c:3c:02”
   user_data_path = “${path.module}/cloud_init.cfg”
}

Make sure to save the file after adding. The final structure of the project should look as follows.

Test the code by running ‘terraform init’ and ‘terraform apply’ in the project workspace where the ‘main.tf’ file is. This should create a new virtual machine. The result should be similar to the last chapter, where the created machine gets an IP leased based on the given MAC address. This will of course only work if the network is already running. When done testing, make sure to destroy the infrastructure. Furthermore, destroy and undefine the libvirt network, as from now on we should be able to create the network from the ‘main.tf’ file as well.

Orchestration example: Load balancing
In this chapter, the module that was created in chapter 8.3 called ‘libvirt_picalc_network’ and the module created in chapter 8.5 calle ‘ubuntu_module’ will be combined to create a single PiCalc service. The ubuntu module will be used to create three individual virtual machines which will be placed in the same network. One of the virtual machines will be the load balancer for the other two machines and will receive the HTTP requests for the PiCalc servers and then decide which server will process the request in round robin style. Heed the sketch below the explanation following it.

Figure 7 — Load Balancer network interface configuration (made with draw.io)

To be able to provision a load balancing service for the virtual machines running the PiCalc server, the use of static IP addresses will be useful. The way to do this using the DHCP protocol was discussed in chapter 8.4. Here, the host was set to a MAC address before creation so that it gets a fixed IP address when available on the network. This can also be seen the sketch above and will be done for all machines.

The load balancing service will be run from a single Terraform configuration file called ’main.tf’. This file was created in the last chapter. Have its contents cleared before starting the instructions. The new additions will be one call to the ‘libvirt_picalc_network’ to create the network, three calls to the ‘ubuntu-module’ to create three virtual machines. Besides these addition in the main file, we will create a new user configuration file specific to the load balancer so that its service is launched on creation. This will be done in a new file called ‘cloud_init_lb.cfg’.

1. Edit ‘main.tf’ and clear its contents.

2. Add the network module definition so that the network is created and managed in the same ‘main.tf’ file.

# run the network
module “picalc-network-module” {
   source = “./modules/libvirt_picalc_network/”
}

3. Add the three modules with the example code below. They will be used for creating virtual machines. The first module will create a virtual machine where we run a script for load balancing. This will be discussed in the following steps. The second and third module will create a PiCalc server. This is possible because our current user data configuration, cloud_init.cfg, is configured to setup a PiCalc server on startup as created in chapter 8.1. This is also why the config file for the load balancer is different since it won’t need the PiCalc server to run on start. Besides the user configuration file, note the mac addresses added to each machine. These can of course be any other valid mac addresses.

# load balancer
module “picalc-lb” {
   # load the module
   source = “./modules/ubuntu-module/”
   # set the variables
   machine_name = “loadbalancer”
   network_name = “picalc_net”
   mac_address = “52:54:00:6c:3c:02”
   user_data_path = “${path.module}/cloud_init_lb.cfg”
}
# picalc server 1
module “picalc-server-1” {
   # load the module
   source = “./modules/ubuntu-module/”
   # set the variables
   machine_name = “server_1”
   network_name = “picalc_net”
   mac_address = “52:54:00:6c:3c:03”
   user_data_path = “${path.module}/cloud_init.cfg”
}
# picalc server 2
module “picalc-server-2” {
   # load the module
   source = “./modules/ubuntu-module/”
   # set the variables
   machine_name = “server_2”
   network_name = “picalc_net”
   mac_address = “52:54:00:6c:3c:04”
   user_data_path = “${path.module}/cloud_init.cfg”
}

4. The virtual machine for the load balancer can already be created with the above example. We only need to create a new cloud_init.cfg which is specific to the load balancer as we don’t want the same PiCalc server to run in the machine. Do this by copying and pasting the ‘cloud_init.cfg’ file to the same directory as ‘cloud_init_lb.cfg’.

$ cp ~/kvm_project/cloud_init.cfg ~/kvm_project/cloud_init_lb.cfg

5. Open and edit the ‘cloud_init_lb.cfg’ file.

6. Change the section under ‘runcmd:’ so that a load balancer gets created on startup. Use the code example below to achieve that. As can be seen, the same PiCalc git repository is used as earlier instructions. The way the program is used is slightly different. To install the load balancer an extra ‘lb’ option is needed. Furthermore, to run the server we have to set the mode to lb and add some configuration using the lb_config option. More about the lb_config option in the next step.

# run command at boot
runcmd:
   - [“cd”, “/home/terraform_guest”]
   - [“git”, “clone”, “https://github.com/krebsalad/PiCalcPy.git"]
   - [“cd”, “PiCalcPy”]
   - [“python”, “install_picalc.py”, “lb”]
   - [“python”, “run.py”, “mode=lb”, “lb_config=      [options]\nbuffer_size=4096\n\n[mappings]\n80=192.168.180.103:8080,192.168.180.104:8080”, “&”]

7. The load balancer is configured very easily. The load balancer used is called PumpkinLB (“Savannah T., PumpkinLB A simple, fast, pure-python load balancer”, (2019a) and can be installed using the PiCalc example cloned from github. The load balancer works as following. first some option can be given behind the [options] tag. We set the buffer size to 4096 in bytes. After the options follows the [mappings] tag. This expects a port to expose for incoming message and one or more IP addresses including the port to listen to. In our case we need to change these to match IP addresses statically configured in the network definition. Do this with the following step.

8. Open the network configuration to add two new static hosts based on MAC addresses defined in step III and the load balancer configuration in the last step.

$ nano ~/kvm_project/modules/libvirt_picalc_network/libvirt_network_config.xml

9. Add the bold mark lines below to your configuration so that the newly added hosts also get a fixed IP address leased by DHCP. Note that the IP address added are the IP addresses of the servers. We already added the address for the load balancer virtual machine before. In any case, make sure the hosts’ MAC addresses match the corresponding addresses given in step III and that the hosts’ IP addresses match the ones set in step VII. Heed the sketch figure 7 for the full hosts table and corresponding hosts.

<dhcp>
   <range start=’192.168.180.2' end=’192.168.180.254'/>
   <host mac=’52:54:00:6c:3c:02' ip=’192.168.180.102'/>
   <host mac=’52:54:00:6c:3c:03' ip=’192.168.180.103'/>
   <host mac=’52:54:00:6c:3c:04' ip=’192.168.180.104'/>
</dhcp>

10. Now, save the file and exit.

11. The Terraform configuration should be ready to run. But before running the ‘main.tf’ file it is very useful to make sure no domains, networks or pools with the same name as the ones we are going to create exist. Do this by using various debug commands learned throughout this document. More about these commands can also be found under chapter 13. Troubleshooting. Furthermore, it will be useful if your host machine has all its resources available as three guest machines will be run at the same time. This should work for even less than 4 cores, but some process swapping would occur a lot and make it very slow. In case of the host being a virtual machine, you could expect it to crash when there is too much process swapping.

12. Plan the configuration by going into the main workspace ‘~/kvm_project’ and running ‘terraform init’ and then ‘terraform plan’. You will see that a total of 13 resources are going to be created of which four are for the first PiCalc server, another four for the second PiCalc server, another four for the load balancer and finally one the libvirt network. If you do not remember what these resources are, heed chapter 7.

13. Run the configuration by running ‘terraform apply’ in the main workspace. When you type ‘yes’ the infrastructure will be created.

14. Continue here after the machines are created and Terraform is done. Start three new terminals to monitor the boot of the virtual machines using ‘virsh console <n>’. Change <n> by the number of the console which can be found using the ‘virsh list — all’ command. Open multiple consoles for each of the virtual machines created. Here you will have to wait until the machines start completely. Do not press enter when done, as you can see if the server is running when the cloud init installation on boot is done.

15. Make sure if all the machine got there ip leases using ‘virsh net-dhcp-leases picalc_net’

16. Finally, test the PiCalc load balanced service by sending a HTTP request to ‘192.168.180.102:80/PiCalc/100’. This is the iP address we set for the load balancer. Do this multiple times and you will notice that the returned IP address differs after on consecutive requests.

9. Installing Docker and Terraform provider for docker

Go back to the table of contents

In this chapter the usage of the Docker provider plugin with Terraform is explained. Docker is mostly used within a cloud service over the internet to containerize various apps. This makes it a perfect candidate to automate it with Terraform because the docker API is readily available and can be used with Docker Swarm and Docker Compose. The (official) Docker provider plugin makes it possible to interact with the Docker engine through Terraform.

We will continue using the same host environment in this chapter (Ubuntu 18.04) and assume that Go and Git is installed (as previously covered in chapter 6.3).

Figure 8 — Static Hosts (made with Archi)

See the above diagram. In this chapter we will create 3 replicated containers using the Docker provider plugin. These containers will be created by Terraform to run as a Docker service, within

an overlay network. The goal is to deploy a docker image and scale it horizontally. Container replicates are commonly used to keep uptime as high as possible. The image must be created beforehand as Terraform does not offer automated generation of images based on Dockerfiles. The procedure to make our architecture will comprise of generating our docker Image containing an example application (for instance, PiCalc). Next, we will configure and plan our Terraform service to use this image and replicate it, and create a Docker network.

Installing the docker engine
The first step required is installing the Docker engine on our Ubuntu 18.04 installation using the Docker repository, followed by building and installing the Terraform provider from source. We need to add the repository manually because Ubuntu does not have Docker in it’s repositories.

1. Open a terminal emulator and add the official Docker repository to APT (for this we download the Docker public key, add it to APT and then install ‘docker-ce’

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ | sudo apt-key add -
$ sudo add-apt-repository \
“deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable”
$ sudo apt update
$ sudo apt install -y docker-ce

2. Enable and start the Docker engine service:

$ sudo systemctl enable — now docker

3. Verify that the Docker service is up and running

$ sudo systemctl status docker

Systemd should give results about the container engine running:

4. From here it is possible to start new containers, but before we do we have to configure our Terraform user to have permission to use Docker service. For this we have to add our user to the docker group.

5. Add our Terraform user to the docker group to let Terraform use Docker without root. Change ‘ubuntu’ to your host’s user running Terraform.

$ sudo groupadd docker #in case its not created by default
$ sudo usermod -aG docker ubuntu

6. It is strongly recommended to logout and login back to the system (or start a new session from your current one) and then verify the user is added to the group by checking the output of groups.

$ su — ubuntu
$ groups

7. Verify that the user is allowed to run Docker. For this, a small ‘hello world’ test will be executed.

$ docker run hello-world

This should output the typical Docker ‘Hello from Docker’ text.

Compiling and installing the Docker provider

1. Export our PATH variables, in case this hasn’t been done before.

$ export PATH=$PATH:/usr/lib/go/bin
$ export GOPATH=/usr/lib/go

2. Create the source directory according to our PATH’s. We will clone the source code in these directories similar to chapter 6.4. (if this chapter has been done already, the directory ‘terraform-providers’ should already exist).

$ sudo mkdir -p $GOPATH/src/github.com/terraform-providers
$ cd $GOPATH/src/github.com/terraform-providers

3. Clone the official git repository.

$ sudo git clone github.com/terraform-providers/terraform-provider-docker.git

4. Go into the provider’s directory and build it with the following commands. In this makefile, the ‘make’ default target installs the provider.

$ cd terraform-provider-docker
$ sudo make

5. Verify the plugin is installed by running it. The GO executable path is already in our PATH environment so it can be directly run from any directory.

$ terraform-provider-docker -version

This command should output a warning message about it’s usage:

6. Symlink the plugin to the Terraform plugins directory so that Terraform will use it.

sudo ln -s /usr/lib/go/bin/terraform-provider-docker ~/.terraform.d/plugins/

10. Deploying Docker examples with Terraform

Go back to the table of contents

This chapter will demonstrate some of the essential functionalities that Terraform has to offer with the Docker provider plugin, by guiding the user through the steps and configuration files.

Interacting with Docker via Terraform stands out from the traditional usage of Terraform, since it is no longer about Infrastructure as a Service, but rather deploying containerized applications directly on the same host running Terraform. This makes sense if people seek to unify deployment of loose applications with the rest of their infrastructure spread over multiple Terraform modules, or if other Terraform modules depends on some application that happens to be in a Docker container during early deployment or later.

With the installation of Docker on our Ubuntu 18.04 host machine completed as described in chapter 9, we will continue with two examples:

• A simple containerized Apache web server with the default test page.
• Our own generated image, which will be initialised as a Docker Service with replicas (and thus are automatically load-balanced). In our example, a NodeJS application called ‘PiCalc’ will be used to distribute requests.

It must be noted that at the time this document was made, Terraform offers no integration with Docker Compose and usage of Dockerfiles. This means we cannot build our image based on configuration. Therefore, the second example we will use Docker directly to build our image via Docker Compose and run it with Terraform. Alternatively this can be achieved roguely using the remote-exec or local-exec provisioner as earlier demonstrates in chapter 5.3. We will not use this and instead use Docker directly.

Building a simple webserver example using Docker Hub
The first example demonstrates that only very little configuration is actually required to get a Docker container up and running, and that the used Docker image can be substituted with many other images depending on the use-case. A resource will be created similar to chapter 5.3.

1. Create a new folder in the Terraform working directory, we will call it ‘docker_web’

2. Go into the project directory and create a new file called ‘cloud_init.cfg’.

$ cd ~/docker_web/
$ terraform init

The output of the last command will notify the user that Terraform is successfully I initialised in the current empty directory, and we are ready to proceed.

3. Create a new terraform module with your preferred editor, we will call it ‘main.tf’, as we follow Terraform’s recommended file names for the minimal module.

$ nano main.tf

4. Append the following snippets of code to the file as we explain them.

#main.tf
provider “docker” {
   host = “unix:///var/run/docker.sock”
}
#…

5. To begin with, we set our provider to a host running the Docker daemon. Theoretically any external docker host can be used (like an Azure, AWS or libvirt virtual machine), but for the sake of simplicity, we directly connect to our local Docker daemon as configured in chapter 9.

resource “docker_image” “httpd” {
   name = “httpd:2.4”
}
#prints the used contained ID for demonstration, optional
output “Docker Image” {
   value = “Using ${docker_image.httpd.latest}”
}
#…

6. Next, we define a Terraform resource for the docker image containing the app we want to use. The arguments that are accepted here are identical to docker run. This means that if the image specified is not locally installed, it will automatically fetch it from Docker Hub. We pull the official Apache image version 2.4 for this example.

resource “docker_container” “webserver” {
   name = “webserver”
   image = “${docker_image.httpd.latest}”
   count = 1 #we come back to this optional variable later!
   ports {
      internal = 80
      external = 80
   }
}

7. Lastly we create the resource that makes our Docker container, using the docker_image resource. The attribute latest of our docker_image.httpd resource object contains a string with the SHA256 ID that represents the name of the image as seen in the output of the command ‘$docker images’. Terraform only accepts this format as a valid input for the ‘image = ‘ property. Exercise caution not to create ambiguous image names when adding your own local image, and also ensure Terraform won’t delete any images (more about this in the second example)

8. The ‘ports’ block specifies which ports Docker will forward from the container to the outside world. Only one ports block can be used in the current version of the provider plugin.

9. For reference and better understanding, the complete above script will achieve the exact equivalent of:

#docker run — name webserver -p 80:80 apache:2.4

10. Apply the Terraform configuration by running:

$ terraform apply

11. After entering ‘yes’ when being prompted, Terraform will now create the container and print its full SHA256 ID. This can be used when inspecting the output of for example ‘$ docker ps’ as well as ‘$ docker stats’ to know the container is up and running.

12. The result from a browser connecting to IP the Terraform should produce the following. Note that the IP to be entered can vary depending on your Host Machine network configuration.

13. Before proceeding to the next example, remove the infrastructure using the following command in our project folder:

$ terraform destroy

14. This will stop and cleanup any Docker containers. Note that it will also clear previously pulled images both locally or remotely via Docker Hub.

Building a scalable Docker Service with custom container and load balancing.
We have seen in the first example how to easily create containers based on downloadable containers from Docker Hub. However, it offers low flexibility and customization.

While it is possible to create multiple containers using the ‘count = ‘ property, it is not very useful in our case and could result in port conflicts and offer no load balancing.

To address this, a second example will be used where we create our own local Docker image running the aforementioned PiCalc NodeJs example application. It will also demonstrate more advanced and useful blocks of code that are commonly used in Docker Compose. The result will be 4 Docker containers working together, each with a 25% maximum CPU load and 100MB of assigned RAM to compute PiCalc requests simultaneously. This increases performance by reducing queue time, and achieves enhanced availability thanks to redundancy.

Because Terraform is completely script driven, it makes sense to compose our own Docker application based on a script (in other words, a Dockerfile). However, it is currently not possible to automate the image building process and we will therefore run $ docker image create manually.

The example will consist of the following steps:

1. Generate our Docker image based on the example Dockerfile

2. Create, and plan (but not yet apply) our Terraform configuration file.

3. Add the host machine to a Docker Swarm.

4. Apply (run) the Terraform module and verify the results in the browser.

1. Create a new project folder for our example:

$ mkdir terraform-docker-picalc
$ cd terraform-docker-picalc

2. In order to create our own image, we have to generate one based on a Dockerfile. It’s a text document that contains all the necessary shell commands to assemble an image. It also contains special fields to derive images from existing ones, and also the command that will execute our task.

3. Create a new file with your preferred editor called Dockerfile, and append the following content to it:

#Dockerfile
FROM node:10
MAINTAINER terraform_user
WORKDIR /app
RUN git clone https://github.com/pb12/PiCalc .
RUN npm install
CMD [ “node”, “server.js” ]

4. In short, this Dockerfile will pull an existing (Ubuntu Linux based) image from Docker Hub, and clones the official Git repository into a folder /app. Because a base image has been used designed for using NodeJS, no extra steps are required and the NodeJS application can immediately be executed with the CMD statement.

5. Built and install the image for local usage. We name this image “picalc” and assign a tag to it with “git” for our version.

$ docker image build -t picalc:git .

6. The output should end with “Successfully built a46004371aeb and “Successfully tagged picalc:git”. Because the unique SHA256 ID of the docker container will vary each time the image an image is built, the output varies. Terraform will use this ID as seen in chapter 10.1 step IV.

7. Now we have built our image manually, because the provider plugin offers no such functionality as of writing this guide. However, we can also instruct Terraform to run this command for us in the future by using the “null_resource” as seen in chapter 5.3:

#optional code snippet, not required for the next step
resource “null_resource” “create_local_image” {
   provisioner “local-exec” {
      command = “docker image build -t picalc:git .”
   }
}

8. This will create our image whenever Terraform creates our Docker infrastructure. But because the image creation process is not always guaranteed to always be successful, it could abruptly fail the Terraform apply process in the future, and we therefore do not recommend this kind of workaround.

9. Create a new file with your preferred editor called “main.tf”, add the following snippets of code to the file as we explain them in detail.

$ nano main.tf

First snippet:

#main.tf
resource “docker_image” “picalc_image” {
   name = “picalc:git”
   keep_locally = true #We don’t want Terraform to remove our local           image
   #depends_on = [null_resource.image_create_local] #if the NR is   used
}
#…

The first resource is similar to the one in the previous example in chapter 10.1. This time however, a different image will be used for the containers, namely the one we just built. Similar to the Docker CLI, it will search for containers locally first, and secondly search Docker Hub if the image does not already exist locally.

Whenever local images are used with Terraform, the property keep_locally has to be set to true (defaults to false). If the default value is used, the local kept version of the Docker image (either downloaded or built) will be removed after Terraform destroy’s the infrastructure.

Second and last snippet:

resource “docker_service” “app-server” {
   name = “picalc-service”
task_spec {
   restart_policy = {
      condition = “on-failure” #if it crashes, container will respawn
  }
   container_spec {
      image = “${docker_image.picalc_image.latest}”
}
   resources {
      limits {
         nano_cpus = 250000000 #~25% load, 1*10^-9 CPU shares in ns
         memory_bytes = 100000000 #100MB, same as 100*10⁵ bytes
    }
  }
}
mode {
   replicated {
    replicas = 4
  }
}
  endpoint_spec {
    ports {
      target_port = “80”
      published_port = “80”
    }
  }
}

The last resource actually defines our Docker service. It consists of several required and optional blocks. It must be noted that these blocks must have these exact names and contain minimal required properties. A minimal docker service consists of at least one task specification (task_spec) and an endpoint specification (endpoint_spec) block.

10. Within task_spec, we set our restart policy for Docker and specify which Image we use for the containers in our service. We also use a block resources to prevent containers from taking up all CPU execution time. This is necessary for PiCalc to process 4 requests simultaneously with as much CPU utilization as possible.

The mode block specifies to make 4 replicas of the container to use within our Service. It differs from the count = used in our previous example, because this time it will be created within aa service as opposed to being separate. The replicas work together behind a virtual load balancer (thanks to Docker Swarm working in the background for us). Lastly we define our ports to forward within the ports block similar to our previous example.

11. For reference, the Terraform configuration now achieves the same direct Docker command:

# docker service create — publish 8080 \
 — name picalc-service \
 — cpus 0.25 \
 — memory 100M \
 — restart-condition=”on-faillure” \
 — mode replicated \
 — replicas=4 \
picalc:git

Or, it will completely replace this Docker Compose file if it would be used:

#docker-compose.yml, for reference only
version: “3”
services:
   myservice:
      image: picalc:git
      deploy:
      replicas: 4
      resources:
      limits:
         cpus: “0.25”
         memory: 100M
      restart_policy:
      condition: on-failure
      ports:
      - “80:80”
      networks:
      - net1
networks:
   net1:version: “3”
services:
   myservice:
   image: picalc:git
   deploy:
   replicas: 4
   resources:
   limits:
      cpus: “0.25”
      memory: 100M
   restart_policy:
   condition: on-failure
   ports:
   - “80:80”

12. Start our infrastructure. To do this we need to initialize Terraform in our project folder. It could some missing plugins.

$ terraform apply

Make our Host Machine the Docker Swarm manager, this must be done manually because the provider cannot

$ docker swarm init

Now apply the configuration, this can take some time. Enter ‘yes’ when prompted.

$ terraform apply

13. Now it is time to check the final result in the browser. We will send one request to calculate the first 5000 decimals of Pi as a demonstration workload for our service. While performing the request, inspect the output of:

$ docker stats

14. This command monitors the containers resource utilisation.

15. Now send the request, by entering your host machine IP address followed by /?n=decimals. Where decimals is the amount of decimals of π to be calculated.

One of the containers is now processing the request, at the CPU cap configured in our Terraform configuration. Its possible to open multiple browser tabs and send more requests to see the workload getting distributed.

11. Installing Azure CLI and Terraform for Windows

Go back to the table of contents

In this chapter we install the needed tools to use Terraform to deploy resources in Azure provider. The required tool is Azure CLI. This will be used to authenticate ourselves through the Azure provider which is readily available for Terraform. We will install these tools before going in Depth in the next chapter. Furthermore, in this chapter also has instructions for installing Terraform on Windows 10. It should still be possible to follow these instructions on the same machine used throughout the guide.

Install Terraform (Windows 10)
In this section we will install Terraform for windows 10. Using Terraform on Windows offers an easier workflow when working with Azure. We install Terraform from the official site and then we will have to set it as a global path.

1. Download Terraform for Windows 10

The download contains a single executable file. Place it in any folder you like and note down the path.

2. Go to Control Panel -> System and Security -> System -> Advanced system settings -> Environment Variables.

3. Search for the variable “Path” and double click it.

4. Click new and add the path of Terraform (without the .exe file) after adding semicolons(;) on the last entry.

For example: C:\Users\<user-name>\Terraform\

Figure 9 — configure environment variable (screenshot)

5. Now you can go to Powershell and use Terraform.

NOTE: If you receive an error that Terraform is not installed. The PATH environment variable was not set up properly.

Install Terraform (Windows 10)
We are going to need Azure CLI for Terraform to initialize your account. With this it is possible to directly use your account with Terraform, without going to the Azure portal.

1. Download Azure CLI and install it.

2. Restart Powershell. Restarting just Powershell will not always work, it is recommend to also restart your computer or vm.

3. Run the login command: az login

The default browser will open and from there you can login with your Azure credentials.

12. Provision a Function app on Microsoft Azure using Terraform

Go back to the table of contents

In this chapter we are going to create the PiCalc App within Microsoft Azure on Windows. We have chosen Windows this time, because we already did everything in Linux. So we want to show it is also possible in Windows. Of course you can also do this on Linux, but you will have to do that yourself.

Microsoft Azure is a well-known cloud computing service. It provides the use of different cloud services, including those for compute, analytics, storage and networking. We are using it mainly using Azure, because of the possibility to create a student account. Students can use Azure for Students, they will receive €100,- credits to spend on Azure resources. This will of course be a prerequisite for this part of the guide.

The use of Azure will help you learn more about remote provisioning using Terraform.

Create a Terraform configuration for Azure
As described before, configurations made for Terraform are being done within a configuration file with extension .tf. These configuration can be written in two formats: HashiCorp Configuration Language (HCL), or JavaScript Object Notation (JSON). HCL is a language created by HashiCorp, it is machine-friendly yet easy for humans to read. JSON is a well known language with the same aspects of HCL, but sometimes preferable when configurations are generated by machine. You can choose which one you want to use, but in this chapter we will keep using HCL.

We are going to make a basic Azure provider block.

1. Create a new project workspace called kvm_project/

2. Create an empty file named main.tf within the project directory

3. The following code will tell what provider we are going to use

provider “azurerm” {
   version = “=1.34.0”
}

The version argument is optional and is used to constrain the provider to a specific version or even a range. This will prevent the provider to downloading a new provider, which potentially can break changes.

4. In the next section we will make a resource block. A resource block has two string parameters as shown below.

resource “<resource_type>” “<resource_name>” {
   name = “HelloWorld”
   location = “westeurope”
}

5. Copy paste the code from step 2 and 3 in the file we created in step 1 main.tf.

Do not forget to change the resource type and resource name. Change the <resource_type> in azurerm_resource_group and <resource_name> in whatever name you want to give it. We will call it rg.

Provisioning an Azure resource using Terraform
The three important commands of Terraform are: terraform init, terraform plan and terraform apply. The first command is used for a new configuration, which initializes local settings and data.

With the second command it is possible to see all the changes you are going to make. Terraform apply will execute all the changes described in the configuration file.

1. Go to your project folder where you made the configuration file

2. Use the command: terraform init Terraform will download the provider plugins.

3. Now use terraform apply, we do not have to run terraform plan, because we are not changing the infrastructure. Make sure you have logged in with az login else the command will fail.

4. Go to portal.azure.com and click on the Resource groups panel on the left. You will find the resource group you defined in the last chapter here. In our case we named it HelloWorld.

5. Destroy the resource group created with terraform destroy. This should start destroying the resource. After a while the resource group will also disappear in Azure portal.

Create a PiCalc server as an Azure Function App with Terraform
In this section we are going to create an Azure Function App on which we run the PiCalc server. For this chapter we are going to use two different docker images we have created which have both a different implementation from one another. One of them is based on the NODE|10 Docker image and the other one is based on the latest release of the Python Docker image. These images are configured to launch a PiCalc server on boot and are going to be the configuration of the Azure Function App, which uses Linux containers as a site configuration.

The implementation of the servers is different in both images, the Python based Docker image will run the same PiCalc python server we used throughout this document. On the NODE variant will run the PiCalc app created by my teacher sir Pieter Burghouwt, we have to thank him for creating this app. Below we will show the Dockerfiles which were used to create these images which can run a PiCalc server on startup and of course the Terraform code used to create an Azure Function App based on these images.

Dockerfiles
First of all we are going to look into the two Dockerfiles used to create the Images. The use of a Dockerfile is already explained in chapter 10. Heed the docker files below and there explanations following them.

Nodejs

FROM node:10
MAINTAINER Nitesh
EXPOSE 8080
COPY /PiCalc ./PiCalc
WORKDIR /PiCalc
RUN npm install
CMD [ “node”, “server.js” ]

First we download an image called node:10. Then we copy the app files to the container and change the directory of the shell instance into the PiCalc app directory. The RUN command is overall being used to install the application or packages. The CMD command allows you to set a default command, which will be executed only when you run container without specifying a command.

Python

FROM python
MAINTAINER Nitesh
EXPOSE 8080
COPY /PiCalcPy ./PiCalcPy
WORKDIR /PiCalcPy
RUN python install_picalc.py
CMD [ “python”, “run.py”, “mode=server” ]

Basically, what we are doing here is almost the same as above. This flow installing is also very similar on how we configured a virtual machine to run PiCalc in chapter 8 with cloud init configuration. Here we also called the same commands to run the PiCalc server.

After we execute those two Dockerfiles we created two new images of them. Which you can find here on DockerHub: PiCalcPy and PiCalc19. In the next section these will be used to create an Azure function App.

The PiCalc code can be found on GitHub: PiCalcPy and PiCalc19.

We are not going to explain the code, it’s very simple, basic Python and JavaScript. An essential part is already explained in chapter 7. Also heed the README.md files in the said repositories for more information.

Configure a Terraform module to deploy the PiCalc Function App
In this section we are going to create a new configuration. This configuration is going to be a Terraform module containing a .tf file which will contain code to deploy an Azure Function App. The module directory will be called modules/azure_picalc_function/ and the main code of the module will be called deploy.tf.

Create the above-mentioned file and directory. Edit the modules/azure_picalc_function/deploy.tf file. In this file we will be adding Code.

First create an Azure resource group. This resource group will contain the information on where our service will be hosted. In our case we will be setting the ${var.location} to ‘westeurope’. The resource group won’t add to your costs in Azure.

resource “azurerm_resource_group” “main” {
   name = “${var.prefix}-resource”
   location = “${var.location}”
}

Add an Azure storage account.

‘The storage account provides a unique namespace for your Azure Storage data that is accessible from anywhere in the world over HTTP or HTTPS’. (“Microsoft, Azure storage account overview”, 2019c)

This is where function App will be deployed. Note that the resource_group_name should match the name of the resource. In the code below this is ensured by using the name variable of azurerm_resource_group.

resource “azurerm_storage_account” “main” {
   name = “${var.prefix}sa”
   resource_group_name = “${azurerm_resource_group.main.name}”
   location = “${azurerm_resource_group.main.location}”
   account_tier = “Standard”
   account_replication_type = “LRS”
}

Add an Azure App service plan within the resource group. This will be decided which consumption plan you take on Azure and will greatly define the performance of the to be created service but likewise will greatly increase the costs. Note that the kind below is “FunctionApp” this means that the image we are going to run must be a variant of the Linux distribution. The other options are FunctionApp, Linux and elastic or Windows. When the kind is set to Linux the reserved must be set to true

Furthermore it is possible to change the tier and size, this will of course change the price you are paying at the moment. See the images below for the different options:

resource “azurerm_app_service_plan” “main” {
   name = “${var.prefix}-asp”
   location = “${azurerm_resource_group.main.location}”
   resource_group_name = “${azurerm_resource_group.main.name}”
   kind = “FunctionApp”
   reserved = “true”
sku {
    tier = “Standard”
    size = “S1”
 }
}

Add a function app. This will deploy the PiCalc service. In the code below, under site_config { can be seen that a linux_fx_version value is set. This will pull a Docker image and run it as the Function App. The docker image installed in this case is the NODE variant.

resource “azurerm_function_app” “main” {
   name = “${var.prefix}-function”
   location = “${azurerm_resource_group.main.location}”
   resource_group_name = “${azurerm_resource_group.main.name}”
   app_service_plan_id = “${azurerm_app_service_plan.main.id}”
   storage_connection_string = “${azurerm_storage_account.main.primary_connection_string}”
# set picalcpy image as server example
   site_config {
   linux_fx_version = “DOCKER|ineshi/picalcpy:latest”
   #linux_fx_version = “DOCKER|ineshi/picalc19:latest”
 }
app_settings = {
   “WEBSITES_ENABLE_APP_SERVICE_STORAGE” = “false”
   “DOCKER_REGISTRY_SERVER_URL” = “https://index.docker.io"
  }
}

As discussed in chapter 8, modules can be used so that variables of a Terraform configuration can be set from a centralized place. In the code above we added several variables but never defined them. In this chapter we will use a different approach where we create a separate ‘.tf’ file within the module directory which will contain the variables and values. For example, we can call this new file modules/azure_picalc_function/variables.tf

Create the above-mentioned file and paste the code that can be found below in variables.tf. The syntax of variables is the same. Note that we set a default value for all the variables. This will ensure that the code can run even if the variables are not set from where the module is called.

variable “prefix” {
   description = “The prefix used for all resources in this example”
   default = “picalc” 
}
variable “location” {
   description = “The Azure location where all resources in this example should be created”
   default = “westeurope” 
}

To run the configuration, you will have to call the module. Edit your main.tf and add the following code to launch the module.

module “function_app_test” {
   source = “./modules/azure_picalc_function/”
}

Finally run the configuration with ‘terraform init’ and ‘terraform apply’. Go to Azure portal then panel ‘Resource groups’ and press on the resource group called ‘picalc-resource’. You will see the following:

Test the PiCalc by starting a browser and searching for one of the following URLs depending on which PiCalc image you used. It will take a long time to load, because the Function App has to initialize all sorts of things.

(In the example code given in this chapter we used “DOCKER|ineshi/picalcpy:latest”, but you can also use “DOCKER|ineshi/picalc19:latest” as the site config.)

# PiCalcPy
https://picalc-function.azurewebsites.net/PiCalc/100
PiCalc19
https://picalc-function.azurewebsites.net/?n=100

The output of the above URLs should look similar to the following output (Ignore the IP in the following image):

13. Troubleshooting

Go back to the table of contents

This chapter will cover common problems that can occur when using Terraform with providers mentioned in this guide. This will be done in three sections where solutions for problems than can happen when using a provider can be found. In these sections some tools that can be used for debugging these problems will be discussed as well.

Debugging Terraform provider libvirt
Most of the errors that occur when using the libvirt provider have to do with the creation of libvirt resources. When creating a resource, the provider can fail because of various problems. Therefore, it is useful to know some commands that can help debug these problems. Heed the commands below and their description.

Common problems and possible solutions

Problem 1

Error: virError(Code=38, Domain=7, Message=’Failed to connect socket to ‘/var/run/libvirt/libvirt-sock’: No such file or directory’) on libvirt.tf line 1, in provider “libvirt”: 1: provider “libvirt” {

This problem can happen when libvirt is not started. Verify that libvirt is started with the following command:

$ sudo systemctl status libvirtd

If it’s not running make sure to start it. Using the systemctl start command.

Problem 2

Error: virError(Code=38, Domain=7, Message=’Failed to connect socket to ‘/var/run/libvirt/libvirt-sock’: Permission denied’)

• Is libvirt running?
• Is your user in the libvirt group? if not redo chapter 7.1 setup permissions
• If on a virtual machine and you just installed libvirt for the first time, make sure to restart the machine and try again.

A common problem from Chapter 7
When deploying a KVM through libvirt with Terraform as described in Chapter 7, certain problems can occur and follow up on eachother, even if just one error is made. The known solutions for those are described here. Problems 3 until 7 are all related, makes sure if one occurs, you use all the known solutions from each one first before using Terraform Apply again.

Problem 3

Error: Error creating libvirt domain: …. Could not open ‘/tmp/terraform_libvirt_provider_images/image.qcow2’: Permission denied’)

This problem can occur when applying the Terraform plan on Libvirt provider.

• This means the file cannot be accessed. Make sure the directory of the file that is denied has user permissions. Is the directory existing? as terraform can’t create files unless its in /tmp/.

You can change the permissions of the images folder with the chmod command.

• Make sure the security_driver in /etc/libvirt/qemu.conf is set to “none” instead of “selinux”

Don’t forget to restart the libvirt service if you made changes:

$ sudo systemctl restart libvirtd

Problem 4

Error: Error defining libvirt domain: virError(Code=9, Domain=20, Message=’operation failed: domain ‘ubuntu-domain’ already exists with uuid 56a0fcc3-e8a2–4eae-a9e0-b94b9fc2e284')

This problem can occur when applying the Terraform plan on Libvirt provider. Often happens in Chapter 7.

• This means that the resource that you are trying to create is already existing. Make sure to destroy the said resource using one the virsh undefine commands. For example, when a domain is already existing, as in the example above, run the following commands.

$ virsh destroy ubuntu-domain

Problem 5

Error: Error creating libvirt volume: virError(Code=90, Domain=18, Message=’storage volume ‘ubuntu-amd64.qcow2’ exists already’)

and

Error:Error creating libvirt volume for cloudinit device commoninit.iso: virError(Code=90, Domain=18, Message=’storage volume ‘commoninit.iso’ exists already’)

This error can occur when trying to remove a faulty Terraform plan. Often happens in Chapter 7.

• The Volumes created by Libvirt are still attached to the images, and prevent a new Terraform being applied with the same Volume names. Therefore removal of these is required:

$ virsh vol-delete commoninit.iso — pool ubuntu-pool && virsh vol-delete ubuntu-amd64.qcow2 — pool ubuntu-pool

Problem 6

Error: Error storage pool ‘ubuntu-pool’ already exists

Make sure you delete the created pool as well, first by halting it and remove it afterwards. Often happens in Chapter 7.

• Remove the libvirt pool that was created during the Terraform process:

$ virsh pool-destroy ubuntu-pool && virsh pool-undefine ubuntu-pool
Pool ubuntu-pool destroyed

Problem 7

Error: Error your_vm_name already exists

• Remove the running VM

$ virsh undefine your_vm_name

Bibliography

“cloud-init, cloud-init 19.2 documentation”, (2019a). Retrieved 15 October 2019, from https://cloudinit.readthedocs.io/en/latest/

“Docker, What is a Container?”, (2019a). Retrieved 10 October 2019, from https://www.docker.com/resources/what-container

“Docker, Why Docker?”, (2019b). Retrieved 10 October 2019, from https://www.docker.com/why-docker

“Docker, PayPal Use Case with Docker Enterprise”, (2019c). Retrieved 10 October 2019, from https://www.docker.com/customers/paypal

“Duncan Mac-Vicar P., dmacvicar/terraform-provider-libvirt”, (2019a). Retrieved 30 September 2019, from https://github.com/dmacvicar/terraform-provider-libvirt

“glitterlabs, Terraform UI”, (2019a). Retrieved 20 October 2019, from http://www.glitterlabs.com/terraform_ui/index.html

“Google, The Go Programming Language”, (2019). Retrieved 10 October 2019, from https://golang.org/

“HashiCorp, Introduction — Terraform”, (2019a). Retrieved 30 September 2019, from https://www.terraform.io/intro/index.html

“HashiCorp, Introduction to HashiCorp Terraform with Armon Dadgar”, (2018b). Retrieved 11 October 2019, from https://www.youtube.com/watch?v=h970ZBgKINg&t=148s

“HashiCorp, README.md — HCL is the HashiCorp configuration language”, (2019b). Retrieved 11 October 2019, from https://github.com/hashicorp/hcl

“HashiCorp, Docker Provider”, (2019c). Retrieved 10 October 2019, https://www.terraform.io/docs/providers/docker/index.html

“HaschiCorp, Azure Provider”, (2019d). Retrieved 20 October 2019, https://www.terraform.io/docs/providers/azurerm/index.html

“HaschiCorp, Azure Provider: Authenticating using the Azure CLI”, (2019e). Retrieved 20 October 2019, https://www.terraform.io/docs/providers/azurerm/auth/azure_cli.html

“HashiCorp, Null Resource”, (2019f). Retrieved 10 October 2019, https://www.terraform.io/docs/providers/null/resource.html

“libvirt, KVM/QEMU hypervisor driver”, (2019a). Retrieved 15 October 2019, from https://libvirt.org/drvqemu.html

“libvirt, The virtualization API”, (2019b). Retrieved 30 September 2019, from https://libvirt.org/

“Miscrosoft, What is cloudcomputing”, (2019a). Retrieved 30 September 2019, from https://azure.microsoft.com/nl-nl/overview/what-is-cloud-computing/

“Miscrosoft, What is Azure”, (2019b). Retrieved 20 October 2019, https://azure.microsoft.com/en-us/overview/what-is-azure/

“Miscrosoft, Azure storage account overview” (2019c), Retrieved 23 October 2019, from https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview

“Mell .P & Grance .T, The NIST Definition of Cloud Computing p2–3”, (2011a). Retrieved 30 September 2019, from https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf

“Redhat, What is KVM?”, (2019a). Retrieved 30 September 2019, from https://www.redhat.com/en/topics/virtualization/what-is-KVM

“Savannah T., PumpkinLB A simple, fast, pure-python load balancer”, (2019a) Retrieved 20 October 2019, https://github.com/kata198/PumpkinLB

“The Open Group, Welcome to the ArchiMate® 3.0.1 Specification, an Open Group Standard”, (2019a). Retrieved 11 October 2019, from https://pubs.opengroup.org/architecture/archimate3-doc/

“The Open Group, Archi — Open Source ArchiMate Modelling”, (2019b). Retrieved 11 October 2019, from https://www.archimatetool.com/