Event Collection, Processing, and Correlation Workflow

Applies to Product: USM Appliance™ LevelBlue OSSIM®

All LevelBlue USM Appliance's security monitoring and management capabilities stem from its overall ability to collect data from devices, transform the data into a common set of data fields that define events, and then process, filter, and correlate those events to identify potential threats and vulnerabilities, or real occurrences of attacks. USM Appliance also assesses the importance and priority of events by assigning risk values based on the value of the underlying assets, the source and nature of the identified threat, and the likelihood of successful attack. More detail on this overall workflow is provided in this section for the following topics:

  • Log Data Collection, Parsing, and Normalization
  • Event Processing and Filtering
  • Event Correlation, Alarms, and Notification
  • Event Visualization and Analysis

Log Data Collection, Parsing, and Normalization

Log collection is at the root of LevelBlue security management. LevelBlue USM Appliance collects logs from various sources: network devices, such as firewalls and routers, host servers and systems, and software applications running on servers. Some devices, for example, those that support the Syslog protocol, are configured to send their logs directly to the USM Appliance Sensor. For other devices, USM Appliance goes out and retrieves the logs. In both cases, data in the logs is normalized to extract and store information in common data fields that define an event: IP addresses, host names, user names, interface names, and so on. These are the events that a security analyst can analyze in USM Appliance to uncover threats and vulnerabilities, and assess an organization's risk.

Log Parsing Using Plugins

Running on a USM Appliance Sensor, an LevelBlue USM Appliance agent is configured with a collection of different log-parsing plugins, which define how to collect logs from specific devices, systems, or applications, and how to transform that log data into standardized event data fields before sending the events to the USM Appliance Server. The plugins also control other event-gathering functions on the sensor, such as intrusion detection. USM Appliance comes equipped with plugins for many commonly encountered data sources. Contact AlienVault to request a new plugin for any data source or product for which a plugin does not already exist. You can also create your own custom plugins, or customize USM Appliance’s existing plugins.

Normalization of Security Events

No matter the format of a log message, certain pieces of data (such as user names or IP and MAC addresses) are common in all of the device logs. Extracting these values out of the log message text and storing them into matching common fields is called normalization. Normalization is what allows you to perform queries across events collected from varied sources (for example, “Show all events where the source IP is”.) Although the format of the original data collected from devices may be different, similar information across devices is stored in the same field for events sent to the USM Appliance Server.

The logs are broken down into their message type, and the information from them is used to populate a standard set of fields that define an event (for example, date, sensor, plugin_id, priority, src_ip, src_port, dst_ ip, dst_port, username, userdata1).

Note: For a complete list of normalized event fields, see Review Event Details.

Event Processing and Filtering

After normalizing the data obtained from log files and other sources, the USM Appliance Sensor transmits security events to the USM Appliance Server. The USM Appliance Server also performs several additional operations on incoming events, including:

  • Parsing the event priority and reliability — Each event type is assigned a priority, which indicates how urgently the event should be investigated, and a reliability score, which assesses the chance the event is a false positive.
  • Checking asset values to calculate a risk score — The USM Appliance Server maintains an inventory of known devices on the network, with an associated asset value for each device, defining their importance to the organization. This asset value is then weighed against the event’s priority and reliability score to produce a risk value. Higher risk scores help analysts know what is most important to examine first.

    For more information on how USM Appliance calculates risk, see USM Appliance Network Security Concepts and Terminology.
  • Application of the event taxonomy — There are system and network events common across many system types, no matter the source of the event or its original data format. LevelBlue maintains a hierarchical categorization of event types (referred to as a taxonomy) to which USM Appliance can match events in policies and correlation directives.
  • Cross-checking reputation data — The USM Appliance Server checks the IP addresses specific to each event against a reputation database of Internet addresses. IP addresses that match are flagged for future reference and follow-up.

After performing these operations, and based on specified user policy and filter conditions, the USM Appliance Server will save selected or qualified events in a SIEM events database for further analysis and correlation. The events database commonly resides on the same host as the USM Appliance Server, but in large deployments, the database can be installed on a separate host for increased performance and capacity.

Event Correlation, Alarms, and Notification

Following the basic processing, analysis, and filtering that the USM Appliance Server performs, selected or qualified events are fed into the LevelBlue USM Appliance correlation engine. Using LevelBlue USM Appliance correlation, analysts can look for patterns and sequences of events across multiple devices and system types. Events may actually be processed by the correlation engine several times, as different correlation rules may take the same events as input.

Correlation directives create alarms

As events continue to feed into the correlation engine, USM Appliance generates alarms based on event conditions specified in correlation directives or rules:

  • Alarm processing starts when the conditions of a correlation directive are met.
  • Alarms may trigger on a single event matching certain conditions, or may require a specific sequence of events to trigger.
  • Alarm processing may continue over a matter of hours. Alarms that appear in the system may indicate they are still processing additional incoming events to further corroborate detection.
  • Alarms are themselves events (directive events), that can feed into other correlation directives once they are triggered, so you can create cascading levels of alarms.

In addition, when you sign up for the Open Threat Exchange® (OTX™), USM Appliance is configured to receive raw “pulse” data and indicators of compromise (IoCs), from OTX. USM Appliance correlates that data and alerts you to any related OTX pulse and IP reputation-related security events and alarms when it detects those same IoCs interacting with assets in your environment.

As soon as you log into USM Appliance, you can see from the USM Appliance dashboard which OTX indicators are active in your environment. You will receive immediate notification in the form of an event or an alarm when a malicious IP address identified in OTX communicates with any of your system assets, or when USM Appliance identifies any other IoCs seen in OTX are active in your network.

Note: For more information about how USM Appliance alarms are processed and correlated, see Alarm Management.

Event Visualization and Analysis

Events obtained from device logs, as well as those generated by the correlation engine itself, can all be searched, viewed, and reported on from the USM Appliance web UI. Two different options are available to access and view events:

  • View of security events with options to search, filter, and group events based on specific event field values. To use this option, select Analysis > Security Events (SIEM) from the web UI.
  • View of raw log events displayed with a specific time frame. To use this option, select Analysis > Raw Events from the web UI.

For more information on viewing events and performing other security management operations from the USM Appliance web UI, see Review Security Events and Review and Verify Raw Logs .