Introduction
When you have hundreds and thousands of servers, you need to be able to make quick changes to them in one go rather than ssh-ing into every server and executing repetitive commands. This is inefficient and time consuming.
Marionette Collective aka MCollective is a great tool for centralised server orchestration.
Now owned by Puppet Labs, it integrates well with Puppet, but also Chef.
What it can do
MCollective can remotely work with several system components:
- puppet: manage Puppet agents (run a test, enable / disable, get statistics etc…)
- package: install, uninstall a package
- apt: upgrade packages, list number of available upgrades
- service: start, stop, restart a service
- nettest: check ping and telnet connectivity
- filemgr: touch, delete files
- process: list, kill process
- nrpe: run nrpe commands (check_load, check_disks, check_swap)
and more
How it works
Using a message queue, which all the MCollective agents on the servers listen to, the MCollective client (your desktop or management server) can send tasks.
The tasks can only be sent to certain agents thanks to discovery filters which can either be:
- facts: any fact returned by Facter such as country, OS name or version, domain, ip address, mac address etc…
- identity: the server’s hostname or fqdn
- classes: the Puppet classes applied to the server
Filters can be combined and regular expressions can be used as well.
MCollective presentations
Watch an Introduction to Systems Orchestration with MCollective from PuppetConf 2013
Slideshares by the architect of MCollective; R.I.Pienaar
Vagrant MCollective framework
The easiest way to quickly try MCollective is to use the Vagrant MCollective framework at the bottom (just run 2 commands and it builds a Vagrant cluster !).
https://github.com/ripienaar/mcollective-vagrant
Installing MCollective
We’ll be installing and configuring MCollective for Ubuntu 12.04 LTS.
Setup Apt Repositories
By default MCollective works with ActiveMQ, however I’d recommend RabbitMQ over AcitveMQ.
To use the latest RabbitMQ packages, use the official RabbitMQ apt as the Ubuntu one is quite old:
|
1 2 |
sudo bash -c "echo 'deb http://www.rabbitmq.com/debian/ testing main' > /etc/apt/sources.list.d/rabbitmq.list" wget -O- http://www.rabbitmq.com/rabbitmq-signing-key-public.asc | sudo apt-key add - |
We also need to use the PuppetLabs apt to use the latest MCollective packages:
|
1 2 |
wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb sudo dpkg -i puppetlabs-release-precise.deb |
Finally get the packages update:
|
1 |
sudo apt-get update |
RabbitMQ Configuration
The RabbitMQ connector uses the STOMP rubygem to connect to RabbitMQ servers.
Install rabbitmq-server:
|
1 |
sudo apt-get install rabbitmq-server |
Enable Stomp and management plugins then restart RMQ:
|
1 2 3 |
sudo rabbitmq-plugins enable rabbitmq_stomp sudo rabbitmq-plugins enable rabbitmq_management sudo service rabbitmq-server restart |
Download the rabbitmqadmin script to set some settings:
|
1 2 |
sudo wget -P /usr/local/bin http://127.0.0.1:15672/cli/rabbitmqadmin sudo chmod u+x /usr/local/bin/rabbitmqadmin |
Create the RMQ user, vhost, permissions and exchanges:
|
1 2 3 4 |
sudo rabbitmqadmin declare user name=mcollective password=marionette tags=mco sudo rabbitmqadmin declare permission vhost=/ user=mcollective configure=.* write=.* read=.* sudo rabbitmqadmin declare exchange name=mcollective_broadcast type=topic sudo rabbitmqadmin declare exchange name=mcollective_directed type=direct |
Add the stomp listener to the RabbitMQ config by editing /etc/rabbitmq/rabbitmq.config
|
1 2 3 4 |
[ {rabbitmq_stomp, [{tcp_listeners, [{"0.0.0.0", 6163}, {"::1", 6163}]}]} ]. |
Restart RabbitMQ
|
1 |
sudo service rabbitmq-server restart |
MCollective Agents Configuration
On any server you wish to orchestrate remotely via MCollective, you must install the mcollective-agent-* packages. Lets start with the package, service and puppet agents:
|
1 |
sudo apt-get install mcollective-package-agent mcollective-service-agent mcollective-puppet-agent |
Edit the MCollective configuration on the agents at /etc/mcollective/server.cfg with the details of the RabbitMQ/Stomp server and authentication details previously set.
Remove the connector and plugin.stomp settings and replace with:
|
1 2 3 4 5 6 7 8 |
direct_addressing = 1 connector = rabbitmq plugin.rabbitmq.vhost = / plugin.rabbitmq.pool.size = 1 plugin.rabbitmq.pool.1.host = <RMQ-HOSTNAME> plugin.rabbitmq.pool.1.port = 6163 plugin.rabbitmq.pool.1.user = mcollective plugin.rabbitmq.pool.1.password = marionette |
Restart MCollective
|
1 |
sudo service mcollective restart |
MCollective Client Configuration
On your desktop or management server, install the base MCollective and ruby-stomp packages:
|
1 |
sudo apt-get install mcollective ruby-stomp mcollective-client |
Plus the client packages to communicate with the package, service and puppet agents:
|
1 |
sudo apt-get install mcollective-package-client mcollective-service-client mcollective-puppet-client |
Edit the MCollective client configuration at /etc/mcollective/client.cfg with the same settings as server.cfg configured on the agents:
|
1 2 3 4 5 6 7 8 |
direct_addressing = 1 connector = rabbitmq plugin.rabbitmq.vhost = / plugin.rabbitmq.pool.size = 1 plugin.rabbitmq.pool.1.host = <RMQ-HOSTNAME> plugin.rabbitmq.pool.1.port = 6163 plugin.rabbitmq.pool.1.user = mcollective plugin.rabbitmq.pool.1.password = marionette |
Restart MCollective
|
1 |
sudo service mcollective restart |
Running MCollective
Use mco help to see the available commands. And for help on a mco command run mco help
The easiest way to see which servers are discoverable is to run a ping:
|
1 2 3 4 5 6 7 8 9 10 |
mco ping node01.bluemalkin.net time=81.86 ms node02.bluemalkin.net time=82.71 ms node03.bluemalkin.net time=84.22 ms node04.bluemalkin.net time=85.37 ms node05.bluemalkin.net time=86.37 ms node06.bluemalkin.net time=119.15 ms ---- ping statistics ---- 6 replies max: 119.15 min: 81.86 avg: 89.95 |
Get the status of a package (can can also install/uninstall/update/purge):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
mco package puppet status * [ ===========================================> ] 6 / 6 node01.bluemalkin.net: puppet-3.3.1-1puppetlabs1. node02.bluemalkin.net: puppet-3.3.1-1puppetlabs1. node03.bluemalkin.net: puppet-3.3.1-1puppetlabs1. node04.bluemalkin.net: puppet-3.3.1-1puppetlabs1. node05.bluemalkin.net: puppet-3.3.1-1puppetlabs1. node06.bluemalkin.net: puppet-3.3.1-1puppetlabs1. Summary of Arch: No aggregate summary could be computed Summary of Ensure: 3.3.1-1puppetlabs1 = 6 Finished processing 6 / 6 hosts in 644.31 ms |
Get the status of the ssh service (you can also start/stop/restart):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
mco service status ssh * [ ==========================================> ] 6 / 6 node01.bluemalkin.net: running node02.bluemalkin.net: running node03.bluemalkin.net: running node04.bluemalkin.net: running node05.bluemalkin.net: running node06.bluemalkin.net: running Summary of Service Status: running = 6 Finished processing 6 / 6 hosts in 216.72 ms |
Execute a Puppet agent run on all nodes with a concurrency of 4:
|
1 |
mco puppet runall 4 |
Using Filters
Before using filters you need to to know the facts and classes on a server:
|
1 |
mco inventory <HOSTNAME> |
Identity filter
To run mco with a server identity use:
|
1 |
mco ping -I node01.bluemalkin.net |
Class filter
If you have a class apache deployed on the web servers, you can restart apache on just those servers using a class filter:
|
1 |
mco service apache2 restart -C apache |
Fact filter
To update the puppet package on all Ubuntu 12.04 servers using a fact filter:
|
1 |
mco package puppet update -F operatingsystemrelease=12.04 |
Conclusion
MCollective is a very useful tool which will save sys admins lots of time. It will help deploy applications and maintain servers a lot quicker.
There are many plugins that can be added to MCollective at http://projects.puppetlabs.com/projects/mcollective-plugins/wiki
Be sure to checkout the official documentation for MCollective at http://docs.puppetlabs.com/mcollective/deploy/install.html