среда, 6 июля 2011 г.

SMS notifications in OpenNMS

We just got a lot of problems when our air conditioning system controller in data center went mad and didn't notice the failure of two air conditioners. The 60 Celsius degrees is not the best temperature for servers operations...
After dealing with this situation we decided to setup SMS-notification service at least for temperature in server room and several other parameters. We already had OpenNMS monitoring system configured, so we had to add SMS-notification to our setup.
This task was done in several steps. First of all, we created a script which would send SMS. We used Google Calendar for this purpose. One dedicated user (let's say opennms) was created for our monitoring system and all system administrators imported his calendar with the following notification settings:

  • Events reminders - By default remind me via sms 1 minutes before each event

  • New Invitations - SMS


Of course, system administrators had to register their phone numbers in Google Calendar.
When we want to send SMS notification, we create new event in opennms's calendar using gcalcli for this purpose.
We used the following script to create a new event (sleep was inserted to prevent mass event creation in case when everything is bad):

#!/bin/sh
H=$(/bin/date "+%H")
M=$(/bin/date "+%M")
sleep 1
event_text="$H:$M $@"
/usr/local/bin/gcalcli --user opennms --pw OurPassword --cals=owner quick "$event_text"


Then, we described a notification command in /usr/local/opennms/etc/notificationCommands.xml:

<command binary="true">
<name>SendSMS</name>
<execute>/root/bin/send_sms.sh</execute>
<comment>Send SMS by gcalcli</comment>
<argument streamed="false">
<switch>-subject</switch>
</argument>
<argument streamed="false">
<switch>-tm</switch>
</argument>
</command>

Argument elements describe script parameters, here we pass it notice subject and notice body (full list of possible parameters may be found here).

Later we created a destinationPath in /usr/local/opennms/etc/destinationPaths.xml:

<path name="SMS-Admins" initial-delay="0s">
<target interval="0s">
<name xmlns="">admin</name>
<autoNotify xmlns="">auto</autoNotify>
<command xmlns="">SendSMS</command>
</target>
</path>

You have to set some legal OpenNMS user or group here (in name target's sub-element).

Now you can create some notifications using OpenNMS Web UI. To control outstanding characteristics of certain parameters you should set trigger event to uei.opennms.org/threshold/highThresholdExceeded and set destinationPath for it to "SMS-Admins". You can do it by hand, adding the following entry to /usr/local/opennms/etc/notifications.xml:

<notification name="High Threshold" status="on" writeable="yes">
<uei xmlns="">uei.opennms.org/threshold/highThresholdExceeded</uei>
<description xmlns="">High threshold exceeded</description>
<!-- some filter -->
<rule xmlns="">(NODELABEL = 'our label')</rule>
<destinationPath xmlns="">SMS-Admins</destinationPath>
<text-message xmlns="">The parameter %parm[ds]% is high on node: %nodelabel%, interface:%interface%. The parameter %parm[ds]% reached a value of %parm[value]% while the threshold is %parm[threshold]%. This threshold for this alert was %parm[threshold]%.</text-message>
<subject xmlns="">Notice #%noticeid%</subject>
<numeric-message xmlns="">111-%noticeid%</numeric-message>
</notification>


The only interesting question I have now is how to do notification only for several thresholds. Just for now I deleted all unimportant (for me) thresholds. It would be cool, however, to specify instead in notification description only thresholds you are interested in...