How Are Message (trap, syslog, etc) Events De-Duplicated On Event Manager

[rajib]

As new Message Events (http://community.zyrion.com/showthread.php?t=66) are recorded by the Message Handler component, they are subject to the advanced de-duplication logic built into Traverse. Unlike the fixed de-duplication window (http://community.zyrion.com/showthread.php?t=67) for Threshold Violation Events, a separate de-duplication window can be configured for each type of Message Event source (SNMP trap, Windows EventLog, Syslog, etc). Additionally, you have complete control over what information contained within the event is used to de-duplicate it. This is accomplished through Event Processing Rules where each rule looks for a specific text pattern (supports full regular expression) and specifies whether to accept or reject the event when a match is found. For example:


<!-- default rule for critical messages -->
<rule>
<description>Syslogd: Generic Error Event</description>
<pattern>\s+\d+\:\d+\:\d+\s+(\S+)\s+LOG_(EMERG|ALERTS|CRIT| ERR)\s+(\w+)\s+(.*)$</pattern>
<action>accept</action>
<mapping>
<field name="device_address" match="1"/>
<field name="event_severity" match="2"/>
<field name="event_facility" match="3"/>
<field name="event_text" match="4"/>
</mapping>
<transform>(${event_severity}/${event_facility}) ${event_text}</transform>
<additional-duplicate-key>${event_severity},${event_facility},${event_text}</additional-duplicate-key>
<severity>critical</severity>
<show-message>true</show-message>
<auto-clear>3600</auto-clear> <!-- 1 hr -->
</rule>


This rule is applicable to an event received by the Syslog source and looks for those with LOG_EMERG or higher severity. The <mapping> section is used to extract several key pieces of information and stores them in internal variables. These variables are later referenced to construct a new event (optional step) in <transform> section whcih is the information shown on the Event Manager. The <additional-duplicate-key> section is of importance here since it defines which pieces of the original event will be used to determine if it is a candidate to be de-duplicated into an existing event of similar characteristics. Finally the <auto-clear> section is used to automatically expire (remove) the event from Event Manager after 1 hour.

This is a generic rule defigned to catch all Syslog events of a specific severity. Additional rules can be created to focus on specific events with different criteria and parameters.

[samrog]

I've tried to add an attachment that shows the "duplicate" syslog messages.

[rajib]

Sam,

The events in question are being accepted by the Syslog source using the default/catch-all rule. As you will notice in the screenshot, the "Message Text" includes the date/time of the original event, which is part of what was sent via Syslog. In order to exclude the date/time information from the event, you will need to create a new Event Processing Rule like the one attached. The rule looks for "%ACE-3-251010" events and extracts the message test starting with that particular tag, therefore ignoring the timestamp. Now the events should be de-duplicated properly when the (reduced) message text is same.

In order to apply the new rule, you will need to extract the contents of the archive under Traverse installation directory (the XML configuration will be placed under plugin/messages directory) and restart Web Application and Message Handler components.