Automated Nagios monitoring with Puppet exported resources

NagiosOne of the coolest things Puppet can do is create exported resources. In plain words it means that you can include a manifest on all your nodes which then gets customised and applied for each node thanks to the node facts.
A popular usage of exported resources is automating Nagios monitoring. Instead of manually creating a Nagios configuration with the basic checks such as load, disk usage etc… then duplicating it for each server and changing the hostname, an exported resource can do it all for you by including a single manifest.

PuppetDB configuration

Exported resources requires storing facts for the Puppet nodes so we need PuppetDB installed on the PuppetMaster.

Create /etc/puppet/puppetdb.conf

Add to /etc/puppet/puppet.conf under [main]

Create /etc/puppet/routes.yaml

Start PuppetDB and restart the PuppetMaster:

Nagios Server Configuration

Create a Nagios module with the following manifests:



Note that at the time of writing, Nagios 4 isn’t available as a package for Ubuntu 12.04 LTS so we’re assuming you built your own one which installs it under /etc/nagios4


There is a bug in Puppet when the exported resources are created, they do not have the correct permissions to allow the nagios user on the server to read them so declare a fix-permissions exec resource.


The exported resource operator <<||>> (not to be confused with the spaceship <||> operator) is the resource which will realize all @@ virtual exported resources. How it works is described further down.

Nagios NRPE nodes

For all nodes which you want to monitor with nagios, we need the Nagios NRPE server installed


Then the export manifest where the @@nagios virtual resources are declared


To include Nagios on all nodes to monitor, just add to the default node on manifests/nodes.pp

How it works

When you declare an exported virtual resource on the node, after the puppet agent run, the exported configurations are stored into PuppetDB. Then when you run the puppet agent on the Nagios server, it collects all the nodes exported resources from PuppetDB and subsequently creates the Nagios .cfg files. Beware that if you have a lot of nodes that use exported resources, it can create a long catalogue compilation time, so consider extending the run interval of the puppet agent.

Therefore all nodes must trigger a puppet run before the server can import the configurations.

Note that if you remove nagios::export class from a node, it will not remove the exported configurations in PuppetDB, the resources will still be created on export. You need to keep the virtual exported resources and set ensure to absent. Or if you’re completely removing the node, on the PuppetMaster you can deactivate it with

Exported resources can have many other uses such as managing sshkey know_hosts resources or for dynamically adding/removing load balancer members to Apache for example.

Iteration in Puppet using the future parser

People often ask me how do you iterate through an array in Puppet when you can do it easily in a template ?

Puppet uses declarative programming and is a domain specific language therefore it cannot do loops which you are probably used to doing in most languages.

Define iteration

For a while it has been possible to sort of iterate in Puppet manifests by using a define resource:

Which produces:

The name of the foo resource takes an array where each element calls the foo define.

Define resources are commonly used for apache configurations for example (see

However since every resource name must be unique, it is impossible to call a define twice, using the same values in the array.

Also if you want to iterate through an array inside a define, you’d have to call another define which can get messy.

Future Parser

Since Puppet version 3.2 experimental features are available and it includes an array iteration feature !

Important note:

The future parser is an experimental feature and is not officially supported by Puppet Labs. It is not recommended for production environments, so enable it at your own risk and ensure you have gone through the new language restrictions at
The non-capitalized variables compatibility and using variables in templates without @ prefixing (or scope.lookupvar) are two areas that need attention since there are quite a few Puppet Forge modules that still use them.

To enable the future parser on your puppetmaster, edit /etc/puppet/puppet.conf and add:

Restart the puppetmaster and check the syslog for any issues compiling catalogues for nodes. This is where you can find future parser compatibility issues.

To test array iteration with a puppet apply, it’s as simple as:

If you’re coding your Puppet modules in Geppetto, set the “Puppet target version” to “future” in the Preferences, to avoid the bad syntax highlighting.

Happy iterating !

Systems Orchestration with MCollective


marionetteWhen you have hundreds and thousands of servers, you need to be able to make quick changes to them in one go rather than ssh-ing into every server and executing repetitive commands. This is inefficient and time consuming.

Marionette Collective aka MCollective is a great tool for centralised server orchestration.

Now owned by Puppet Labs, it integrates well with Puppet, but also Chef.

What it can do

MCollective can remotely work with several system components:

  • puppet: manage Puppet agents (run a test, enable / disable, get statistics etc…)
  • package: install, uninstall a package
  • apt: upgrade packages, list number of available upgrades
  • service: start, stop, restart a service
  • nettest: check ping and telnet connectivity
  • filemgr: touch, delete files
  • process: list, kill process
  • nrpe: run nrpe commands (check_load, check_disks, check_swap)
    and more

How it works

Using a message queue, which all the MCollective agents on the servers listen to, the MCollective client (your desktop or management server) can send tasks.
The tasks can only be sent to certain agents thanks to discovery filters which can either be:

  • facts: any fact returned by Facter such as country, OS name or version, domain, ip address, mac address etc…
  • identity: the server’s hostname or fqdn
  • classes: the Puppet classes applied to the server

Filters can be combined and regular expressions can be used as well.

MCollective presentations

Watch an Introduction to Systems Orchestration with MCollective from PuppetConf 2013

Slideshares by the architect of MCollective; R.I.Pienaar

Vagrant MCollective framework
The easiest way to quickly try MCollective is to use the Vagrant MCollective framework at the bottom (just run 2 commands and it builds a Vagrant cluster !).

Installing MCollective

We’ll be installing and configuring MCollective for Ubuntu 12.04 LTS.

Setup Apt Repositories

By default MCollective works with ActiveMQ, however I’d recommend RabbitMQ over AcitveMQ.
To use the latest RabbitMQ packages, use the official RabbitMQ apt as the Ubuntu one is quite old:

We also need to use the PuppetLabs apt to use the latest MCollective packages:

Finally get the packages update:

RabbitMQ Configuration

The RabbitMQ connector uses the STOMP rubygem to connect to RabbitMQ servers.

Install rabbitmq-server:

Enable Stomp and management plugins then restart RMQ:

 Download the rabbitmqadmin script to set some settings:

Create the RMQ user, vhost, permissions and exchanges:

Add the stomp listener to the RabbitMQ config by editing /etc/rabbitmq/rabbitmq.config

Restart RabbitMQ

MCollective Agents Configuration

On any server you wish to orchestrate remotely via MCollective, you must install the mcollective-agent-* packages. Lets start with the package, service and puppet agents:

Edit the MCollective configuration on the agents at /etc/mcollective/server.cfg with the details of the RabbitMQ/Stomp server and authentication details previously set.
Remove the connector and plugin.stomp settings and replace with:

Restart MCollective

MCollective Client Configuration

On your desktop or management server, install the base MCollective and ruby-stomp packages:

Plus the client packages to communicate with the package, service and puppet agents:

Edit the MCollective client configuration at /etc/mcollective/client.cfg with the same settings as server.cfg configured on the agents:

Restart MCollective

Running MCollective

Use mco help to see the available commands. And for help on a mco command run mco  help 

The easiest way to see which servers are discoverable is to run a ping:

Get the status of a package (can can also install/uninstall/update/purge):

Get the status of the ssh service (you can also start/stop/restart):

Execute a Puppet agent run on all nodes with a concurrency of 4:

Using Filters

Before using filters you need to to know the facts and classes on a server:

Identity filter
To run mco with a server identity use:

Class filter
If you have a class apache deployed on the web servers, you can restart apache on just those servers using a class filter:

Fact filter
To update the puppet package on all Ubuntu 12.04 servers using a fact filter:


MCollective is a very useful tool which will save sys admins lots of time. It will help deploy applications and maintain servers a lot quicker.

There are many plugins that can be added to MCollective at

Be sure to checkout the official documentation for MCollective at