Working With Zenoss Events

Zenoss Events are among the most powerful and useful monitoring features that Zenoss Core provides. Many different types of events are automatically generated by Zenoss when important scenarios occur for the devices being monitored.

Zenoss Event Architecture

When an event arrives at Zenoss, it is parsed, associated with an event classification and then typically (but not always), it is inserted into the status table of the events database. Events can then be viewed by users using the Event Console of the Zenoss Graphical User Interface (GUI).

The events system has the concept of active status events and historical events (two different database tables in the MySQL events database).

Understanding Event Classes

Zenoss event classes are a simple organizational structure for the different types of events that Zenoss generates and receives. This organization is useful for driving alerting and reporting. You can, for example, create an alerting rule that sends you an email or pages you when the availability of a Web site or page is affected by filtering on the /Status/Web event class^[2].

Creating Events

It is possible to create custom events using different methods available. When creating events, there are some fields that are available to provide more information about the event. The basic event fields are:

1. device
2. severity
3. summary
4. ipAddress
5. eventState
6. message
7. evid

The device field is a free-form text field that allows up to 128 characters. Zenoss accepts any value for this field, including devices that are not in the database. If the device field contains an IP address, then Zenoss queries the database for devices with a matching address. If it finds a match, it changes the device field to the found device name.

The ipAddress field is a free-form text field that allows up to 15 characters. This field is not required. If Zenoss cannot successfully locate a device based on the event's device field content, it attempts to find the device based the event ipAddress field content, if present.

~> To create an event the event must contain values for the Device, Severity, and Summary fields. If an event is missing any of these fields, then Zenoss rejects it.

Using Zensendevent

zensendevent is a Zenoss command line tool that can easily create events in Zenoss. The following are the usage options for zensendevent:

Usage: zensendevent [options] summary

Options:
  -h, --help            show this help message and exit
  -d DEVICE, --device=DEVICE
                        device from which this event is sent, default:
                        localhost
  -i IPADDRESS, --ipAddress=IPADDRESS
                        Ip from which this event was sent, default:
  -y EVENTKEY, --eventkey=EVENTKEY
                        eventKey to be used, default:
  -p COMPONENT, --component=COMPONENT
                        component from which this event is sent, default: ''
  -k EVENTCLASSKEY, --eventclasskey=EVENTCLASSKEY
                        eventClassKey for this event, default: ''
  -s SEVERITY, --severity=SEVERITY
                        severity of this event: Critical, Error, Warn, Info,
                        Debug, Clear
  -c EVENTCLASS, --eventclass=EVENTCLASS
                        event class for this event, default: ''
  --monitor=MONITOR     monitor from which this event came
  --port=PORT           xmlrpc server port, default: 8081
  --server=SERVER       xmlrpc server, default: localhost
  --auth=AUTH           xmlrpc server auth, default: admin:zenoss
  -o OTHER, --other=OTHER
                        Specify other event_field=value arguments. Can be
                        specified more than once.
  -f INPUT_FILE, --file=INPUT_FILE
                        Import events from XML file.
  -v                    Show the event data sent to Zenoss.

With that information, we can easily generate an event (the command must be used by the zenoss user):

zensendevent -s Critical -c /Status/Ping -p servermond -d CCMA05103 Testing zensendevent...

Where -p stands for component. In this case, we are specifying a custom daemon as the component.

This will effectively create the event, and we can see it in the Zenoss user interface event's list:

Using an Event Service

Inside zendmd or in our Python code we can use an Event Service to send events. For example, we could make a custom daemon send an event from within the doTask method:

# Other imports here...
from Products.ZenEvents.ZenEventClasses import Error, Critical, Clear

# ...
class MyDaemonTask(ObservableMixin):
    # ...
    def dotask(self):
        # Use the daemon's Event Service
        self._eventService.sendEvent(dict(
            summary='Event Test...',
            message='Event Test...',
            component=self._preferences.collectorName,
            eventClass='/Status/Ping',
            ipAddress=self.configId,
            device=self.configId,
            severity=Critical,
            agent=self._preferences.collectorName
            ))

Notice that we are first importing the available event severities so that we can later pass one as the severity parameter. You can take a look at the Python file where these severities are defined in $ZENHOME/Products/ZenEvents/ZenEventClasses.py.

Using the JSON API

It is also possible to manipulate events using the Zenoss JSON API. This can be done using cURL or other language wrappers around the API.

For example, we can use the Zenoss Shellscript wrapper to create an event:

source json_api.sh
zenoss_api evconsole_router EventsRouter add_event '{"summary":"Testing event...","device":"$device","component":"$component","severity":$severity,"evclasskey":"$evkey","evclass":"$evclass"}'

Replace the variables with the necessary input.

Clearing Events

Most of the events that are created in Zenoss by default have the ability to be automatically cleared. In order for an event to auto-clear or automatically clear, the event fingerprint must match for both events (the event that set the error and the one that clears it). The fingerprint is composed of the following items:

• device
• component (can be blank)
• eventKey (can be blank)
• eventClass (including zEventClearClasses from event class configuration properties)

However it is possible to manually clear events by providing the exact same information that was provided when the event was created, but using the Clear severity instead. For example:

zensendevent -s Clear -c /Status/Ping -p servermond -d CCMA05103 This will clear /Status/Ping events

References

1. Zenoss Event Management by Jane Curry
2. ZenPackers Documentation: Event Management