Systems Orchestration with MCollective

Introduction

marionetteWhen you have hundreds and thousands of servers, you need to be able to make quick changes to them in one go rather than ssh-ing into every server and executing repetitive commands. This is inefficient and time consuming.

Marionette Collective aka MCollective is a great tool for centralised server orchestration.

Now owned by Puppet Labs, it integrates well with Puppet, but also Chef.

What it can do

MCollective can remotely work with several system components:

  • puppet: manage Puppet agents (run a test, enable / disable, get statistics etc…)
  • package: install, uninstall a package
  • apt: upgrade packages, list number of available upgrades
  • service: start, stop, restart a service
  • nettest: check ping and telnet connectivity
  • filemgr: touch, delete files
  • process: list, kill process
  • nrpe: run nrpe commands (check_load, check_disks, check_swap)
    and more

How it works

Using a message queue, which all the MCollective agents on the servers listen to, the MCollective client (your desktop or management server) can send tasks.
The tasks can only be sent to certain agents thanks to discovery filters which can either be:

  • facts: any fact returned by Facter such as country, OS name or version, domain, ip address, mac address etc…
  • identity: the server’s hostname or fqdn
  • classes: the Puppet classes applied to the server

Filters can be combined and regular expressions can be used as well.

MCollective presentations

Watch an Introduction to Systems Orchestration with MCollective from PuppetConf 2013

Slideshares by the architect of MCollective; R.I.Pienaar


Vagrant MCollective framework
The easiest way to quickly try MCollective is to use the Vagrant MCollective framework at the bottom (just run 2 commands and it builds a Vagrant cluster !).
https://github.com/ripienaar/mcollective-vagrant

Installing MCollective

We’ll be installing and configuring MCollective for Ubuntu 12.04 LTS.

Setup Apt Repositories

By default MCollective works with ActiveMQ, however I’d recommend RabbitMQ over AcitveMQ.
To use the latest RabbitMQ packages, use the official RabbitMQ apt as the Ubuntu one is quite old:

We also need to use the PuppetLabs apt to use the latest MCollective packages:

Finally get the packages update:

RabbitMQ Configuration

The RabbitMQ connector uses the STOMP rubygem to connect to RabbitMQ servers.

Install rabbitmq-server:

Enable Stomp and management plugins then restart RMQ:

 Download the rabbitmqadmin script to set some settings:

Create the RMQ user, vhost, permissions and exchanges:

Add the stomp listener to the RabbitMQ config by editing /etc/rabbitmq/rabbitmq.config

Restart RabbitMQ

MCollective Agents Configuration

On any server you wish to orchestrate remotely via MCollective, you must install the mcollective-agent-* packages. Lets start with the package, service and puppet agents:

Edit the MCollective configuration on the agents at /etc/mcollective/server.cfg with the details of the RabbitMQ/Stomp server and authentication details previously set.
Remove the connector and plugin.stomp settings and replace with:

Restart MCollective

MCollective Client Configuration

On your desktop or management server, install the base MCollective and ruby-stomp packages:

Plus the client packages to communicate with the package, service and puppet agents:

Edit the MCollective client configuration at /etc/mcollective/client.cfg with the same settings as server.cfg configured on the agents:

Restart MCollective

Running MCollective

Use mco help to see the available commands. And for help on a mco command run mco  help 

The easiest way to see which servers are discoverable is to run a ping:

Get the status of a package (can can also install/uninstall/update/purge):

Get the status of the ssh service (you can also start/stop/restart):

Execute a Puppet agent run on all nodes with a concurrency of 4:

Using Filters

Before using filters you need to to know the facts and classes on a server:

Identity filter
To run mco with a server identity use:

Class filter
If you have a class apache deployed on the web servers, you can restart apache on just those servers using a class filter:

Fact filter
To update the puppet package on all Ubuntu 12.04 servers using a fact filter:

Conclusion

MCollective is a very useful tool which will save sys admins lots of time. It will help deploy applications and maintain servers a lot quicker.

There are many plugins that can be added to MCollective at http://projects.puppetlabs.com/projects/mcollective-plugins/wiki

Be sure to checkout the official documentation for MCollective at http://docs.puppetlabs.com/mcollective/deploy/install.html

One thought on “Systems Orchestration with MCollective

  1. Great writeup. Thanks!

    Just two notes:

    – on the agent configurationn you missed the mcollective package on the apt-get install line:
    sudo apt-get install mcollective mcollective-package-agent mcollective-service-agent mcollective-puppet-agent

    – on the master, where the ‘client’ is installed you don’t need to restart the mcollective service, in fact it doesn’t need to run at all, if you run:

    service mcollective stop
    update-rc.d mcollective disable

    it still works. The client.cfg file is read by the mco command, not by the mcollectived daemon.

Leave a Reply to ghomem Cancel reply

Your email address will not be published. Required fields are marked *