Merge Multiple Events Into Single Event Filter

Definition

Description

Any events matching Condition may be merged by adding FieldName from each event to the starting event.
Start merging if an event matches the starting Condition.  
End the merging if an event matches ending Condition, or after TimeInterval.
Unique sequence number is added to each merged fieldname.

 

 

Comments

This filter can be used for combining a sequence of events into a single event, for example merging multiple lines read from a text log file into a single event record.

If FieldName or the ending Condition are missing, blank or invalid, do nothing.

1. If an event matches the starting Condition, retain the starting event.

2. For every subsequent event take its FieldName and add it as a uniquely named field to the starting event (such as ev:msg1, ev:msg2, ...). Discard this merged event.

3. When the event matches the ending Condition or if the TimeInteval expires or another starting Condition is matched, release the starting event into the stream after creating the following event fields:

ev:mergeEvents.startTime = StartingTime
ev:mergeEvents.endTime = EndingTime
ev:mergeEvents.seconds= Ending Time - Starting Time
ev:mergeEvents.count= # of records merged

Example:
FieldName = "ev:msg"
Starting Condition = "ev:host containsString 'HostAB' "
Ending Condition = "ev:host containsString 'HostDE' "

Starting Event in:
ev:host = "HostAB"
ev:msg = "value 0"

Starting Event after (still held):
ev:host = "HostAB"
ev:msg = "value 0"

Event 2 in:
ev:host = "HostCD"
ev:msg = "value 1"

Starting Event after (still held):
ev:host = "HostAB"
ev:msg = "value 0"
ev:msg1 = "value 1"

Event 3 in:
ev:host = "HostDE"
ev:msg = "value 2"

Starting Event after (released):
ev:host = "HostAB"
ev:msg = "value 0"
ev:msg1 = "value 1"
ev:msg2 = "value 2"
ev:mergeEvents.startTime= May 22 2003 23:05:44
ev:mergeEvents.endTime= May 23 2003 00:06:21
ev:mergeEvents.seconds = 37
ev:mergeEvents.recordsMerged = 3

 

Example

<filter objectId="MergeMultiple" type="MergeMultipleEventsIntoSingleEventFilter">
	<parameter type="Condition">
		<negatePrimaryCondition>false</negatePrimaryCondition>
		<conditionRelation>Any</conditionRelation>
		<conditionSpec> <!—changed name from “condition” -->
			<negate>false</negate>
			<fieldName>ev:host</fieldName>
			<operator>neq</operator>
			<valueType>string</valueType>
			<value>HOST_INVALID</value>
		</conditionSpec>
	</parameter>
	<parameter type="FieldName">ev:msg</parameter>
	<parameter type="Condition">
		<negatePrimaryCondition>false</negatePrimaryCondition>
		<conditionRelation>Any</conditionRelation>
		<conditionSpec>
			<negate>false</negate>
			<fieldName>ev:host</fieldName>
			<operator>isContainedInString</operator>
			<valueType>string</valueType>
			<value>START</value>
		</conditionSpec>
	</parameter>
	<parameter type="Condition">
		<negatePrimaryCondition>false</negatePrimaryCondition>
		<conditionRelation>Any</conditionRelation>
		<conditionSpec>
			<negate>false</negate>
			<fieldName>ev:host</fieldName>
			<operator>isContainedInString</operator>
			<valueType>string</valueType>
			<value>END</value>
		</conditionSpec>
	</parameter>
	<parameter type="TimeInterval">
		<time>10</time>
		<units>min</units>
	</parameter>
</filter>