munin-host-rename

Recently we decided upon a new host naming convention for our infrastructure at my $dayjob. So I will soon be renaming a few hosts.

I noticed that munin still doesn't provide an easy way, to rename an existing node without losing historical data. So I wrote a small shell script which does that for me and might be of use to everybody else in the same situation.

Synchronize Puppet with Git

Puppet really shines at automating infrastructures. You will notice a sudden change of working methodology, once you manage the first systems with it.

Instead of manually logging on to each single system for updating a certain part of configuration by issuing shell commands, you will stop to repeat yourself and just update a single piece of code, which describes the desired config state for all systems.

As recommended in the Puppet documentation you are well advised to keep your Puppet manifests under revision control.

I wrote a small script which will come in handy, to ease your life with keeping your repository and the manifests on the master in sync and should fit to most of the environments out there.

Once installed, you can store the manifests for each Puppet environment in its own GIT branch and every time you commit a new version to one of your branches, it will automatically sync the most recent version and inform the Puppet master process.

BTW. this could also be used to keep the manifests on multiple Puppet instances in sync.

Puppet-Sync

Puppet-sync is a Ruby based command line tool to synchronize every commit from a central GIT repository to your Puppet master instance. You should install it on your Puppet master and configure a GIT hook which calls the script over ssh.

Puppet-sync takes some parameters to specify how the environment on the master looks like. Run it with '--help' to get a list of available options. Here is an example:

puppet-sync --branch master \
            --passenger     \
            --destination /etc/puppet/environments \
            --repository ssh+git://git.ono.at/srv/git/puppet.git

By running the command above, the script connects to the git repository, fetches the manifests from the master branch and puts it into /etc/puppet/environments/production. Since we use the master branch for production, I added the logic to translate "master"-branch to "production"-environment.

Installation on the master

Instead of listing each and every shell command needed to install the environment for the script, I'd like to provide a simple manifest instead. I think this is more readable and you can either use it or figure out the appropriate shell commands. ;-)

file { "/usr/local/bin/pupept-sync":
    ensure  => present,
    source  => "file:///puppet-sync",
}

file { "/home/psync/.ssh":
    ensure  => directory,
    owner   => "psync",
    mode    => 700,
    require => User["psync"],
}

file { "/etc/puppet/environments":
    ensure  => directory,
    owner   => "psync",
    mode    => 775,
    require => User["psync"],
}

user { "psync":
    ensure     => present,
    home       => "/home/psync"
    managehome => true,
}

ssh_authorized_key { "puppet-sync-ssh-key":
    ensure  => present,
    key     => "AAAAB3.....lVBp0nPLNcs=",
    type    => "ssh-rsa"
    user    => "psync",
    require => File["${homeroot}/$name/.ssh"],
}

I have the following configuration in my puppet.conf to make puppet aware of each of the directories in /etc/environments:

...
[master]
.....
templatedir = /etc/puppet/environments/$environment/
modulepath  = /etc/puppet/environments/$environment/modules/
manifest    = /etc/puppet/environments/$environment/manifests/site.pp
manifestdir = /etc/puppet/environments/$environment/manifests

Git Hook

The only thing left is to create a Git Hook in your repository. Here is the one i use. I also created a psync user on the master, so i just need to store the private ssh key in psync's home directory.

#!/bin/sh
#
# An example hook script to prepare a packed repository for use over
# dumb transports.
#
# To enable this hook, make this file executable by "chmod +x post-update".

branch=`echo $1 | awk -F/ {'print $3'}`

sudo -u psync ssh puppet.ono.at /usr/local/bin/puppet-sync \
                                       --passenger         \
                                       --branch $branch

exec git-update-server-info

Perfect its installed and you are ready to use it. Go ahead an try to commit a new version.

Monitoring Puppet and Bacula

Rescently I blogged about my CheckMK APT plugin, capable of checking for upgradeable packages.

Meanwhile I wrote two more plugins, both adopting existing checks and migrated all plugins into one single 'checkmk' repository over at GitHub.

Bacula

This check adapts the idea of Bernhard Miklautz bacula-utils. The plugin will define four different services to be monitored on each Bacula server:

bacula.errorvols

Checks the state of available backup volumes and will report critical, if errounos volumes are found. In case of an error, it will as well return the names of these volumes.

Use the bacula-clear-errvols script to resolve these issues.

bacula.freshness

Checks whether all clients have an associated backup within the last 30 hours.

bacula.fullbackups

Checks whether all clients have an associated full-backup.

bacula.fullbackupspool

Checks whether any volumes used for full-backups come from a full-backup pool.

Puppet

The second plugin will monitor the status of the puppet agent. It integrates nicely with puppetstatus.py. This script was initially written by TMZ from the Fedora Infrastructure Team and can be used to enable or disable the puppet agent on a specific host.

You are able to disable Puppet on a server, by running the following command:

 sudo puppetstatus -d "Interesting things are about to happen."

The monitoring system will change the state of the Puppet check to WARNING and will display this message and the username of the person who was disabling the agent.

When the agent is not disabled on a host, the check will just change to WARNING after 3 and to CRITICAL after 4 hours.

I'd be glad to get some feedback on my plugins, please report any bugs by sending me an e-mail or leave a comment below.

Newer Older