SaltyVagrantChapter 7
The Master Server
In building our system we need to start somewhere. It is tempting to start with a development environment but we will instead start by building a server to provide some basic facilities.
- A version control repository—into which all our sources will be placed
- A ‘bug’ tracker—to keep track of issues and defects.
- A build system—a ‘doer of things’ to automate the transformation of source into product and ultimately to build our infrastructure.
Although I said we will be starting by building server rather than a development environment we will still be creating a virtual server for development and testing.
7.1 Preliminaries
If you have not done so already, get the material for this book to make it easier to follow along (see §1.3.1)
7.2 Base server and operating system
The master server will be an x86 machine with 8GB RAM, 4 cores, and 250GB SSD1. Onto this hardware we will install the Debian operating system. At the time of writing the latest stable version of the Debian operating system is Debian-10 codenamed ‘Buster’.
The Vagrantfile for our base server will therefore look like this;
1# -*- mode: ruby -*- 2# vi: set ft=ruby : 3 4Vagrant.configure("2") do |config| 5 config.vm.box = "bento/debian-10" 6 config.vm.box_version = "202010.24.0" 7 config.vm.provider "virtualbox" do |vb| 8 vb.memory=8192 9 vb.cpus=4 10 end 11end
Try it now. Create a new directory, move into that directory, create a new Vagrantfile and enter the content above, save the file and then vagrant up. You should end up with a new Virtualbox server set up with the Debian operating system installed.
The keen-eyed amongst you will have noticed that there is no mention in this Vagrantfile of disk size. The Vagrant box will provide the initial system disk, changing this size of this disk involves more than a simple directive in the Vagrantfile. For now we will live with the disk provided and address this issue if the need arises.
Another key feature of this Vagrantfile is the config.vm.version line. This ‘locks’ our configuration to a specific version of the base box. This ensures that any configuration work we do after this is building on a known starting configuration. It also means that anyone using our Vagrantfile is sure to also deal with the correct starting point.
If we omit line 6 then Vagrant will use the latest bento/debian-10 available on the Vagrant Cloud box catalogue. As we develop our initial solution this may be an option we want to use, but as soon as we are to share our configuration with others providing the version is important.
‘Locking’ our version like this is important for consistency later but has an associated downside. As our configuration ages this version of the Vagrant box becomes increasingly out of date with respect to the Debian releases. We can deal with these issues in a number of ways, among them the following.
- Review and update the ‘locked’ version of the box periodically.
- Build our own custom Vagrant box.
- Update the Vagrant VM as part of our subsequent configuration.
Each of these has pros and cons and we will consider each later. This ‘lock’ versus ‘free’ version issue will be a recurring one. For now we have a potentially more serious issue to deal with, the Vagrant box we rely upon is hosted on the Vagrant box catalogue hosted by HashiCorp and there is no guarantee that this box will remain available indefinitely. Worse, we have no control over that box’s availability, the Chef team or HashiCorp may choose to deprecate it. This raises a general issue; we should, so far as practicable, make copies of all resources we rely upon such that we can maintain direct control over them. For the box it would simply mean cloning the bento/debian-10 box into our own Vagrant box catalogue and then using this local Vagrant box catalogue as our primary source. Since we do not currently have such a catalogue we should add this to our backlog.
It is also important to emphasise that our development and test configuration is based on a virtual machine configuration suitable for use with Vagrant on our desktop machine, it is not necessarily a precise match with the base configuration we will have on our target system. How should we deal with this? One obvious solution is to carefully control matters such that these differences are eliminated. Another, perhaps more pragmatic, approach for our purposes here is to test for the important features that we must have in our base configuration and to make our configuration appropriately adaptable. We will investigate these options as we proceed.
For practical purposes part of the Vagrant specification for a box requires that an SSH server be provided to allow Vagrant to communicate with the VM to both provide access (via vagrant ssh) and for further configuration, to which we now turn our attention.
While Vagrant requires SSH access we will likely not install SSH access on the majority of our ‘real’ servers. So, one of the major discrepancies we have between our Vagrant development and test system, and our formal test and production system is the presence of SSH. We can minimize the impact of this discrepancy by severely limiting access to SSH. We will address this shortly.
Vagrant boxes will generally not have much security by default. This is by design. Vagrant is intended primarily as a development tool and is not suitable for controlling or deploying production systems, consequently you will find that most generic Vagrant boxes, such as bento/debian-10, are fairly ‘bare bones’, consisting of a largely default installation. However we want our local system to reflect our formal test and production environments as closely as practicable. We will secure our servers as much as is practicable given the aforementioned Vagrant requirements.
7.3 Vagrant SSH
Vagrant boxes set up SSH with a non-privileged user account vagrant. The vagrant account is configured for both password (also vagrant) access and a public/private key pair is preloaded to allow Vagrant SSH access to the server without the need for messy password integrations.
The upshot of this setup is that Vagrant controlled servers are inherently insecure by default as the username/password pair (vagrant/vagrant) is common knowledge and the SSH key pair used is also publicly available (and common to all default Vagrant setups).
The normal use-case for Vagrant renders this insecurity moot because we would not normally make these machines accessible outside our host computer and certainly not on a public network. The difference is significant to our current setup because as we secure our server we need to ensure access to the vagrant account via SSH so that Vagrant continues to work.
We could secure our Vagrant setup further by changing the public/private keys used and changing the vagrant account password, but frankly this is overkill for our purposes. Our Vagrant systems are intended only for use in development on individual host computers, which should themselves have host firewalls preventing unwanted network access to the Vagrant systems.
The most important thing we need to do though is isolate, so far as practicable, any Vagrant specific configuration such that it does not interfere adversely with our development. In other words, we want to avoid making assumptions that are only applicable in our development Vagrant environment. Since our users (developers) may be unaware of these issues, we need to isolate them as far as practicable as part of our configuration.
7.3.1 Vagrant provisioning
As our first step in configuration let’s do something simple. We will need the Git system installed on our new server so this is a good simple thing for use to install.
Modify your Vagrantfile, inserting the config.vm.provision instruction.
1# -*- mode: ruby -*- 2# vi: set ft=ruby : 3 4Vagrant.configure("2") do |config| 5 config.vm.box = "bento/debian-10" 6 config.vm.box_version = "202010.24.0" 7 config.vm.provider "virtualbox" do |vb| 8 vb.memory=8192 9 vb.cpus=4 10 end 11 config.vm.provision "shell", inline: "apt install -y git" 12end
The new line (11) instructs Vagrant to use its SSH connection to run the shell command apt install -y git on the guest operating system (our new server).
By default Vagrant will run all such provisioning shell commands sudo, that is with elevated ‘super user’ privileges. (The vagrant account is an unprivileged account but is configured as a sudo user, see sudo users.)
Assuming you still have your Vagrant VM running from earlier, you can have vagrant ‘re-provision’ the VM rather than needing to destroy it and start over. On your host computer, while in the same directory as the Vagrantfile2.
1vagrant up --provision
By default (without the --provision option) vagrant up will start a halted or suspended Vagrant VM without running any provision entries. With the --provision option the VM is started (if halted or suspended) and any provision commands in the Vagrantfile are run.
In our example the inline shell command apt install -y git will be run and the Git package will be installed on the VM.
If the VM is already running, that is not halted or suspended, the provision entries in the Vagrantfile are still run against the running VM.
If the VM does not exist (it has not been previously created with vagrant up or it has been destroyed with vagrant destroy) then the VM will be created as normal and the provision entries will be run as part of the creation of the VM.
This simple approach to configuring our VM seem okay for simple things but has one major problem, it is not portable. Suppose we want to apply this configuration to another server, one not controlled by Vagrant. We would need to somehow extract all the config.vm.provision directives to apply the relevant configuration. Not very practical.
What we need to do is decouple the configuration actions from the Vagrant mechanism that invokes those actions.
7.3.2 Vagrant provision by script
Modify your Vagrantfile again, replacing the config.vm.provision directive.
1# -*- mode: ruby -*- 2# vi: set ft=ruby : 3 4Vagrant.configure("2") do |config| 5 config.vm.box = "bento/debian-10" 6 config.vm.box_version = "202010.24.0" 7 config.vm.provider "virtualbox" do |vb| 8 vb.memory=8192 9 vb.cpus=4 10 end 11 config.vm.provision "file", source: "scripts/configure", destination: "/tmp/configure" 12 config.vm.provision "shell", inline: "sh /tmp/configure"" 13end
We have added line 11 to copy a file (scripts/configure) from the host computer to the VM (into /tmp/configure).
Line 12 then executes the configure script on the VM.
Next we need to create the configure script. On the host computer, in the directory containing the Vagrantfile.
1mkdir scripts 2vi scripts/configure
Use whatever your preferred text editor is (I’m using vi here).
Enter the following into the configure file.
1#!/usr/bin/env bash 2# vi :set ft=bash: 3 4set -euo pipefail 5 6apt install -y git
The first line ensures that Linux invokes the correct interpreter when none is specified. The second line is a ‘mode’ line telling vi that this is a bash script (not important if you do not use vi, but I do so adding this mode line is habit for me). Line 4 ensures the script fails early and hard if it has any errors (not strictly useful in such a short script, but a good habit to acquire).
Line 6 is the important line and reproduces the install of the Git package.
This may all seem rather overkill, and it is for such a trivial example, but it illustrates an important principal. Moving our configuration instructions into script means we can copy that script to any system we want to configure and run it. This configuration is now independent of Vagrant, relying solely on Linux script interpreters. The only parts of our configuration process that are tied to Vagrant are the provision directive, these contain no configuration information other than which script to upload and run for this particular VM.
7.4 What versus How
Notice that our configuration is really a script detailing ‘how’ to impose our configuration. To use this configuration we need to know some things about our target system. For example, we need to know that the system supports installation of packages using apt, and we need the system to run bash shell scripts.
The second requirements (the need to run bash) could be a simple prerequisite, we are using bash as our configuration tool. The former though is more questionable.
What if I want to run this configuration on an Arch based distribution? These distributions use pacman rather than apt. What about on RedHat distributions where dnf is preferred?
The issue is this; my configuration is really saying ‘I want to ensure Git is available’. This requirement is independent of the underlying operating system or distribution or that operating system. All I really want is to have Git available after I have applied my configuration.
Configuration management tends, therefore, to be based on a declarative system. The configuration is a statement of ‘what’ should be true on the system if it is configured correctly. The configuration manager is unconcerned with ‘how’ the configuration is asserted and only concerned ‘that’ it is asserted. In other words, I don’t care about which installation method is used to install Git I only care that once my configuration is applied Git is available.
In a trivial sense my script could be something like the following (if you’re following along there is no need to make these changes, I’m just illustrating a point).
1#!/usr/bin/env bash 2# vi :set ft=bash: 3 4. config-tool.sh 5 6set -euo pipefail 7 8git-installed
Our configuration now just states ‘after running this configuration the system must have git-installed’, or put another way, ‘any system that meets the requirements of this configuration must have git-installed’. All of the detail about how the configuration tool should verify that Git is in fact installed or how if should be installed if it is not already, is irrelevant to the configuration itself.
A naive implementation of config-tool.sh might look something like the following.
1#!/usr/bin/env bash 2# vi :set ft=bash: 3 4set -euo pipefail 5 6 7# Figure out package tool 8PKG="" 9for pm in "apt dnf pacman"; do 10 if command -v "${pm}"; then 11 PKG="${pm}" 12 break 13 fi 14done 15 16git-installed () { 17 # If git IS installed, we're done 18 command -v git && return 19 20 # Otherwise try to install it 21 case "${PKG}" in 22 "apt") 23 apt -y install git 24 ;; 25 "dnf") 26 dnf -y install git 27 ;; 28 "pacman") 29 pacman --sync --noconfirm git 30 ;; 31 *) 32 echo "No package manager found" >2 33 exit 1 34 esac 35}
Obviously this script is deficient in many ways (not being very generalised, not accounting for different package names, not covering many distributions, not handling errors, etc.) but in principle it allows us to say git-installed (the thing we want to be true) in our configuration and leave all the messy details (of how to make it true) to be figured out by the configuration tool.
As you might imagine, we are not going to be writing our own configuration management system!
7.5 Our core configuration tool
In the previous section we converted our initial configuration steps into a Bash script. This decouples our initial configuration from a Vagrant specific format (the Vagrantfile) and places it into a more portable form that can be used to provision not ony Vagrant machines but also cloud servers or physical machines. This has the benefit of making our configuration something defined independent of the underlying implementation of our server.
We also showed the separation of the configuration from how to acheive that configuration. This is an important abstraction allowing us to focus on ‘what’ our system should look like rather than ‘how’ to make our system look the way we want. (This is obviously an ideal and reality being the complex mess it is seldom this clean cut in real life. But, hey, that’s what we’re here to learn about!)
The Bash script is certainly a move in the right direction and there will be many more such scripts required to set up our servers, but Bash scripts are tough to get right and are seldom concise in expressing all the minor variations required when configuring multiple servers. Fortunately there are specialised tools for managing our server configurations.
There are many tools to choose from in the configuration management space. Which you choose may depend on a number of factors.
- Does your team already have experience using a particular tool? This could result in a default decision based on current experience, the tool may be adopted because people are comfortable using it, or rejected because of past problems with the tool.
- Does your team have deep knowledge of a particular language? This can influence tool choice because particular tools are written using, or are designed to integrate with, specific languages. For example Puppet is Ruby based, while Saltstack is Python based.
- Cost.
- Supported platforms.
- Legacy configuration. Either a need to adopt existing configuration or to migrate from a legacy configuration.
We will use Saltstack for the following reasons:
- It is freely available.
- It is based on Python and Python is pretty much the defacto standard scripting language on Linux (and is preinstalled on most Debian installations, including this bento/debian-10).
- It is so much more than a configuration management tool, it can be used for monitoring and self-healing of systems, features we will use much later in this course.
- It has tools for deploying cloud servers, which we will use later in this course.
- It can be used standalone, over SSH, or as a full bus-oriented master/minion system. These options are discussed briefly in later sections and more fully in Saltstack from Scratch[Boo20d].
7.5.1 Installing Salt
We have seen how trivial installing a Debian package can be when we installed Git. Debian repositories do have a set of Salt packages but they tend to be older versions of Salt and we would prefer to have a more recent version (and have the option to keep our systems up-to-date with the latest releases), so we will not be using the Debian package repository version.
Fortunately SaltStack provide a shell script for installing various SaltStack components. In a similar approach to that used in §7.3.2 we can provide the SaltStack script and run it to install the Salt components we require. We will start with a naive implementation and gradual refine it into a more robust implementation, along with discussion of each refinement.
Edit the Vagrantfile.
1# -*- mode: ruby -*- 2# vi: set ft=ruby : 3 4Vagrant.configure("2") do |config| 5 config.vm.box = "bento/debian-10" 6 config.vm.box_version = "202010.24.0" 7 config.vm.provider "virtualbox" do |vb| 8 vb.memory=8192 9 vb.cpus=4 10 end 11 config.vm.provision "file", source: "scripts/configure", destination: "/tmp/configure" 12 config.vm.provision "shell", inline: "sh /tmp/configure"" 13 config.vm.provision shell, inline: "cd /tmp && curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com && sh bootstrap-salt.sh -M git master" 14end
We are adding just one line (line 13) but it’s doing a lot of work. There are three commands to be run; change to the /tmp directory, download (curl) the script from the SaltStack website, and finally run that script.
As before we can run this additional provision line using the --provision option. On your host computer, in the same directory as the Vagrantfile.
1vagrant up --provision
This run will take some time as the Salt installation script does a lot of work for us.
While it works, let’s consider what we just did (and why it’s not a particularly good approach).
We downloaded a script from the internet and ran is into our VM without any checks. This is a security risk on two levels; the source of the script (the bootstrap.saltstack.com website) could have been compromised and the script could have been tampered with, secondly, we have no idea what the script is actually doing, where is it sourcing the installation from, what (if any) precautions are taken to ensure the script installs the proper files.
This level of trust may be okay for our ‘quick and dirty’ development environment, but they are unacceptable for a production environment.
In Chapter 13 we discuss repositories in more detail but for now we should note that sourcing directly from a public website is a security risk. Whether we consider it a reasonable risk comes down to our risk tolerance (see §11.1.1) and this will vary according to context (we may accept more risk in an isolated development environment but less in a live production environment).
The first step in reducing our risk is simple in principle, we download the script to a local file system, review it, and then use this reviewed copy as the source for our configuration. Simple in principle, more complex in practice. Performing such a review is a non-trivial exercise. It is important that this process results in a controlled copy of the script that can be used for delivery but also for comparison when updates to the upstream (original source) are made.
Furthermore, in this case the script itself refers to other repository objects and if we are to be highly risk averse these must also be vetted and local repositories used. Chapter 13 discussed these observations in more detail, for now we will set most of them aside in order to progress with our configuration work (rest assured though, we will return to this issue).
7.5.2 Additional Salt setup
The current Salt setup will not work!
Salt, as installed, is running a Master/Minion setup where each Minion controls a target configuration (in this case the new VM) and the Master coordinates a set of Minions. In order for Minions to find the appropriate Master they use a DNS lookup. By default the Minion will look for the domain name salt.
The Master and Minion are running as two services on the VM. We can view their current state using systemctl.
1systemctl status salt-master 2systemctl status salt-minion
You will see that the Minion has failed to start, this is because we have not yet configured a domain name salt, so the Minion will be unable to find the Master (even though it happens in this case to be the same VM).
To fix this we need to define a domain name salt that the Minion can find and resolve, and then we need to restart the Minion so that is can find and connect to the Master.
To fix this quickly we can add a line to our /etc/hosts file to resolve domain name salt to the VM itself. As the Master and Minion are running on the same VM we can use the local loopback device lo, this has the IP Address 127.0.0.1 (this IP Address is a standard ‘this machine’ IP address3). We could simple edit our /etc/hosts file, but this would be useless to anyone building this VM from our Vagrantfile in the future. Following the principals of infrastructure as code, this change must be written into our configuration. The modification to the /etc/hosts should exist before we install Salt so that it is available before the Minion tries to find the Master for the first time. The obvious place to do this in our current configuration is the configure script we started earlier.
1#!/usr/bin/env bash 2# vi :set ft=bash: 3 4set -euo pipefail 5 6apt install -y git sed 7 8sed -i -e '/[[:space:]]salt\([[:space:]]\|$\)/ {:l;n;bl}' -e '/localhost/ s/$/ salt/' /etc/hosts
sed is a fairly standard Linux tool available in most distributions (it is available in the bento/debian-10 box already). So, why add it to the install on line 6? This is a precaution. If our configure script is run on a distribution that does not have sed installed we want to ensure that it is installed before trying to use it on line 8. If Git or sed are already installed attempting to install them a second time will not cause any error.
Line 8 needs some explaination. This is the line that adds salt as a domain name. It may be tempting to try something line echo "127.0.0.1 salt" » /etc/hosts to append a suitable line to the /etc/hosts file. But consider what this would do if the configure script were run multiple times (as it has been already). Every run would add another 127.0.0.1 salt line to our /etc/hosts file. Not good.
The more complex sed command avoids this problem. The salt domain name is added only if no suitable entry already exists.
This idea that the script should result in the same output each time is called ‘idempotence’. A fancy word meaning ‘repeated application without change to the result beyond the first run’. Put another way, any idempotent operation has the same result on our system as the first time we run it.
Our configuration is not entirely idempotent yet. If we run this repeatedly there is a possibilty of different results for the following reasons.
- apt install will install the latest available version of each package, so if either Git or sed packages are updated in the repository between runs then the version installed on our VM will be upgraded; our configuration results in different versions of these packages being installed, breaking idempotence.
- We have not specified a specific version to the Salt installation script. As with the packages this may result in the Salt version changing if the source repository is updated between our runs of the isntallation script.
The bootstrap-salt.sh and configure scripts can be combined into one script by adding the call to the bootstrap-salt.sh into the configure script, then remove the config.vm.provision line that calls this script from the Vagrantfile. As a final bit of cleanup for this version we change the name of the configure script to bootstrap-masterserver. The resulting files are shown next.
1#!/usr/bin/env bash 2# vi :set ft=bash: 3 4set -euo pipefail 5 6apt install -y git sed 7 8sed -i -e '/[[:space:]]salt\([[:space:]]\|$\)/ {:l;n;bl}' -e '/localhost/ s/$/ salt/' /etc/hosts 9 10pushd /tmp 11curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com 12sh bootstrap-salt.sh -M git master 13popd
1# -*- mode: ruby -*- 2# vi: set ft=ruby : 3 4Vagrant.configure("2") do |config| 5 config.vm.box = "bento/debian-10" 6 config.vm.box_version = "202010.24.0" 7 config.vm.provider "virtualbox" do |vb| 8 vb.memory=8192 9 vb.cpus=4 10 end 11 config.vm.provision "file", source: "scripts/bootstrap-masterserver", destination: "/tmp/bootstrap-masterserver" 12 config.vm.provision "shell", inline: "sh /tmp/bootstrap-masterserver" 13end
We now have one clean script to take a base Linux (Debian) install and add in basic tools to facilitate our full configuration.
On the plus side, this script is simple enough that it can be modified for alternate base systems without much difficulty and it does so little that debugging it, should the need arise, will be trivial.
On the downside, we are reliant on bootstrap-salt.sh which, while not catastrophic, leaves us open to some risks. We are currently pulling the latest version direct from SaltStack’s server which means it could change without warning, it also means any compromise to this script or the SaltStack domain could compromise any server built from this bootstrap-masterserver script. For now I think these risks are within my comfort zone so let us proceed.
7.6 Something is missing?
The eagle-eyed amongst you will have noticed a significant omission from our progress so far. Requirements. Well, more specifically we’ve dived in to building our system and no one has laid out exactly what we are trying to build. Sure we have some vague notion of what we’re aiming at but most projects produce depressing reams of requirements before a line of code is written, let alone building any support infrastructure.
The odd thing is that, at least in my experience, these project requirements seldom cover basic project infrastructure. There tends to be an implicit assumption that the project team will just build ‘stuff’ they need. Most of the time this seems to work…More or less. Often these support infrastructure elements are an invisible cost to the project. A ‘build manager’ or even ‘build team’ (the title varies) will spend many hours tending the support elements, fixing problems as they arise, building out additional capacity, rebuilding broken systems, etc. Without any requirements it is impossible for the project to assess whether needs are being met in an efficient manner, and so any associated cost is absorbed into the ‘build team’ overhead.
Does this mean we should spend an age developing details requirements? No. But it does seem odd that there is a tendency for teams to promote the short life cycle, rapid feedback, Agile approach to customers but fails to adopt the same standards for the internal systems.
Let’s fix that.
1You might be thinking, “that’s a pretty weird specification for a server!” And you’re right. It just happens to be the specification of a spare machine I have lying around (recall, we’re working with what we have available.)
2Technically you can be in the same directory as Vagrantfile or any of it’s sub-directories, but that’s a lot to type every time.
3Technically the address block 127.0.0.0/8 is reserved for loopback addresses ([see CVH13, table 4])