caboose

Munin is pretty much the standard system monitoring software for linux machines. If you have a server on EngineYard, they don't officially support munin, so it's up to you to get it installed and working. If you use a rails monitoring plugin like NewRelic, well, it's just like that, only older, open source, and uglier. We use and rely on New Relic on Tender and Lighthouse, and it's really fucking great. But for servers with no rails apps, and to get some nice aggregated graphs, I (as chief sysadmin) like using Munin.

There are two parts to munin - the server, and the client. It's sort of confusing; there's a daemon that runs on each of your machines and accepts connections from remote (munin nodes), and there's a cron job that runs on one specific machine which periodically fetches stats from each machine and generates the graphs. I'm going to assume you have munin itself set up somewhere.

First step, you need to get some ports forwarded, because your slice isn't directly accessible to the outside world. A helpful EY tech will forward your port 4949 (munin) from each slice, to something accessible on a public IP, like 22491, 22492, etc.

Now, since we're running gentoo, you'll need to install munin using the emerge package manager. On some OSes (like fedora) you install munin-node but on Gentoo it seems you get both munin and munin-node in one package.

# emerge munin

After about 15 minutes of compiling, we're good to go. You'll want to edit the /etc/munin/munin-node.conf file and add the hostname of your main munin machine as a regex. This will (hopefully) prevent bad guys from seeing your load charts.

allow ^74\.123\.111\.100$

Munin-node works by looking in a directory full of symlinks to executables. It runs the executables and records the results periodically. So, next (still as root) we want munin-node to discover what it should be running.

# munin-node-configure --shell

You'll get a dump of about 30 results that looks like

ln -s /usr/libexec/munin/plugins/cpu /etc/munin/plugins/cpu
ln -s /usr/libexec/munin/plugins/df /etc/munin/plugins/df

You can copy/paste those to create the necessary symlinks. There shouldn't be too much configuration required.

Start up munin-node

# /etc/init.d/munin-node start

And add munin-node to the scripts to be run on startup by symlinking it to the relevant place.

# cd /etc/runlevels/default
# ln -s /etc/init.d/munin-node ./

And you're done. Now, add the new nodes to the master config file and wait until the next cron runs to update them.