Fast Puppet or Chef development using Vagrant

vagrantIf you are developing infrastructure-as-code using Puppet modules or Chef cookbooks then you must checkout Vagrant.

Vagrant has been around for a few years but is today, one of the DevOps favourite tools for automated provisioning of development environments or, for testing infrastructure-as-code, before it gets pushed to production environments.

In this example we’ll be focusing on using Puppet as the provisioner.
It uses puppet apply, saving the need of having to setup a puppetmaster and certificates. (If you prefer, you can provision with a puppetmaster).

Installation

Vagrant uses VirtualBox for provisioning VMs so firstly you need to install it:

Then go to http://downloads.vagrantup.com to download and install the latest version of Vagrant.

Adding a box

Next you need to import a “box” which is a ready to use base image of your favourite Linux distribution; copy the url of a box at http://www.vagrantbox.es or (my preference) use Canonical Ubuntu 12.04 LTS official box. You must also give the box a base name.

Configuration

Next prepare an initial vagrant configuration:

This creates a configuration file named Vagrantfile, open it with your favourite editor

This is the minimal required to get vagrant to boot an Ubuntu 12.04 LTS VM and provision it with Puppet.

By default the provisioner will apply Puppet manifests declared inside manifests/default.pp So create manifests folder and default.pp manifest file

In this example we’re going to install apache and setup a VirtualHost using the apache module from sourceforge which can be installed this way:

Inside ~/vagrant/manifests/default.pp include the apache module

Boot the VM

Now we’re ready to boot the VM using vagrant up

The VM is created within a minute! Then you’ll start to see the verbose output of apache and the vhost being installed.

ssh into the box

Perform some checks:

All good!

 Other Vagrant commands

Stops the execution of the VM and saves it’s state.

Boots up the VM after a suspend.

Destroys the VM.

Re-executes the Puppet or Chef provisioner.
This is probably what you will be running the most  when you’re developing modules/cookbooks.

Shutdowns then starts the VM. If you change your path to Puppet module or manifests you will need to do a reload or destroy/up those two paths are mounted into the box using NFS.

AWS provisioner

Vagrant can also be used to provision AWS instances instead of local virtualbox VMs.

Install the plugin:

Next you will need to create a dummy box, set your AWS credentials, choose an AMI etc…follow the steps explained in the plugin Quick Start at https://github.com/mitchellh/vagrant-aws

Conclusion

Vagrant is a great tool for quickly testing deployments. Vagrant files can be customised to do many things, checkout the official documentation at http://docs.vagrantup.com/v2/ to see what all the options are.

 

 

AWS High Availability on NAT instances

If your Amazon Web Services VPC has instances in a private subnet that requires accessing the internet, you may be using a NAT instance with the VPC route table (containing a route for 0.0.0.0/0 pointing to the NAT instance). Have you considered the possibility of this creating a single point of failure?

Consider the following scenario: you have all your instances mirrored for high availability, across two availability zones. Even with a NAT instance in each availability zone, the 0.0.0.0/0 route to the NAT instance is a single point of failure.

So how to reduce this risk? If most of your outgoing traffic consists of simple http requests, such as running upgrading Operating System packages, one solution is to use proxy servers. By using Squid (or Tinyproxy) running on an instance in the public subnet in both availability zones, then adding an internal Elastic Load Balancer, you provide high availability for your outgoing traffic.

AWS NAT High Availability with Heartbeat

However, if you need NATing for production-critical applications where a proxy server isn’t enough for outgoing traffic, AWS has come up with a simple shell script that can achieve NAT high availability. It works with the two NAT instances (one in each availability zone) pinging each other. If one doesn’t respond, the active NAT instance takes over the route for 0.0.0.0/0 in the availability zone in which the NAT has failed, then attempts to reboot the unresponsive instance. It uses the ec2 api tools and IAM roles and requires each availability zone to use its own route table for the private subnets.

This is how a high level diagram would look like:

aws_nat

The steps are described in the article at http://aws.amazon.com/articles/2781451301784570

Some Improvement tips

Ensure you execute the script with bash instead of sh otherwise the conditional expressions will fail.

It is preferable to run the script as an upstart script rather than a cronjob, so you can easily stop the service when you need to perform maintenance and reboot a NAT instance. Puppet also works better with scripts running as a service so it can ensure they are always running.

Conclusion

There are many tweaks that can be done to improve the script. AWS will reportedly be launching a highly available NAT service in the future but for now this script does the job.

Puppet trick: Running apt-get update only when needed

If you’re running an environment where your private Debian apt repository is constantly changing with new packages being added or upgraded, you may want to have those packages deployed via Puppet as soon as possible.

For example if you have puppet manifests to install the latest package version available:

This will trigger the apt-get-update-private-repo exec resource on every puppet run even if nothing changed in the repository. It also marks the resource as changed in the report; when you view Puppet Dashboard, you wonder why the servers have changed every 30 mins, even though the configuration of the server hasn’t physically changed.

The solution: pre-check for changes in the Packages file

A Debian repository generates an Packages file which is downloaded everytime an apt-get update is executed. See the Debian wiki for the information on how a Debian repository works.

The Packages file change on the repository server when the repository has changed, so before an apt-get update we can check whether the Packages file on the server has been modified and compare it with the local Packages file.
Here is the modified apt-get-update exec resource:

Thanks to the onlyif the apt-get-update-private-repo command will only run if the Packages file has been downloaded and it’s contents are different to the local Packages file.

onlyif must return true for the command to run, however a cmp will return true if there are no changes hence the ! is added before cmp to invert the exit value.

In terms of network usage – it’s almost exactly the same as a normal apt-get update, but with this trick system administrators can be peace and know when a puppet run has actually changed the server.