PT Commons

h1. Icinga Skillshare

The monitoring software we use is "Icinga":https://www.icinga.org, which is a fork of "Nagios":http://www.nagios.com.

Our instance is at http://icinga.palantetech.com/icinga/, hosted on Ramen, the creds are in the credentials folder.

We currently have over 200 hosts, so the easiest place to look for what is happening is the "tactical overview":http://icinga.palantetech.com/cgi-bin/icinga/tac.cgi

Let's use Octavia as an example.

Here is the url for all the things being checked on octavia.

http://icinga.palantetech.com/cgi-bin/icinga/status.cgi?host=octavia

Now let's see how those are set up.

On Ramen, take a look at /etc/icinga/objects/ptc/octavia_ptc.cfg

This is the old way I was setting up checks.

I create a cfg file for each host, define the host, then define all the services for it.

Each service corresponds to a specific command definition (those live in /etc/nagios-plugins/config), which says a specific way to run a specific check script (those live in /usr/lib/nagios/plugins).

For example, the command definition for check_http is

<pre>

# 'check_http' command definition

define command{

        command_name    check_http

        command_line    /usr/lib/nagios/plugins/check_http -H '$HOSTADDRESS$' -I '$HOSTADDRESS$'

        }

</pre>

Which runs the script at /usr/lib/nagios/plugins/check_http with the -H and -I options both pointing to the $HOSTADDRESS$ variable pulled from the host definition (which for Octavia is octavia.mayfirst.org) then gives that back to Icinga which displays the results "like so":http://icinga.palantetech.com/cgi-bin/icinga/extinfo.cgi?type=2&host=octavia&service=HTTP.

Now let's look at another host, "nlg.org":http://icinga.palantetech.com/cgi-bin/icinga/status.cgi?host=nlg.org

If we look on Ramen at /etc/icinga/objects/clients/nlg/nlg.org_nlg.cfg, there are no checks defined!

That's because I've switched them over to the newer much much more efficient way.

The hosts can be put into nested hostgroups, and have services assigned to each group.

For example, the host nlg.org is in the drushbackups033 hostgroup, defined in /etc/icinga/objects/hostgroups/drushbackups.cfg.

That hostgroup is itself part of the drushbackups hostgroup, also defined in /etc/icinga/objects/hostgroups/drushbackups.cfg, the dnschecks hostgroup defined in /etc/icinga/objects/hostgroups/dns.cfg, and the domains hostgroup /etc/icinga/objects/hostgroups/domains.cfg.

Every host that is part of the domains hostgroup gets a service for expiration and blacklists, defined in /etc/icinga/objects/services/domain.cfg; if it's in dns, it gets that service, and if it's in a drushbackup### group, it gets drush backup checks for Drupal, Civi, and Default backups in number corresponding to the first, second, and third of the numbers at the end of the group (/etc/icinga/objects/services/drushbackup.cfg), as well as checking the oldest backup (still in progress).

Q&A