Develop New Plugins from Scratch

Applies to Product: USM Appliance™ LevelBlue OSSIM®

This section provides an overview of the process you can follow to create a new plugin, from scratch, directly editing and updating the plugin configuration (cfg) and .sql files needed to collect and normalize events from specific data sources.

Process of Creating a New Plugin

The following procedure details high level steps in the process of creating a new USM Appliance plugin.

  1. Create a <filename>.cfg file for the new plugin. You may want to make a copy of an existing .cfg file that is similar to the type of plugin you want to create, to save time.
  2. Specify a unique plugin ID (9001 and above) for the plugin and also specify the location of the log file the plugin will read from. The range of values available for user-defined plugins is 9001 to 2147483647, except for the following values, which are also reserved.

    90003, 90005, 90007, 90008, 10002, 12001, 19004, 19005, 19006, 20505

  3. Create event rules using regular expressions to match events coming in from a source log file.
  4. Create the .sql file that specifies the data written to the SIEM database for all events/rules field mapping of extracted data. You may want to make a copy of an existing .sql file that is similar to the new plugin you want to create, and change the fields to describe events defined in the new plugin.
  5. Activate the plugin.

  6. Import your .sql file to the SIEM database using the following command

    cat <plugin_name>.sql | ossim-db

  7. Test the plugin using the USM Appliance web UI to see if events are being detected

Plugin Design Best Practices

LevelBlue offers the following recommendations for developing a new or custom plugin.

  • Use a log sample as large as possible to identify events and data patterns. This helps to ensure that a sufficient number of data variations have been accounted for.
  • Extract data from the log by issuing the command, grep -v sequentially until no more data are returned.
  • Identify all of the values in the log data that may be included in an event.

  • Discard any repeated log data.

  • Look for data patterns as a way to group them into categories. One organizing principle might be, for example, having the same data field distribution.

  • USM Appliance identifies individual events using a plugin_sid, consisting of a serial number. For this reason, you may find that you need to add a translation table section to associate captured data log fields to plugin_sids. (See Creating a Plugin Configuration File for information.)

  • Estimate the frequency with which any event type would be generated from the log by counting the number of times each log line repeats.

    grep <log_line> <logFile> | wc -l

  • If you have multiple similar log lines, evaluate whether it makes sense to apply a single rule to each, thereby creating two or more unique events, or whether you should create one rule that deals with a group of similar log lines and only creates a single event.

    In the latter example, you run the risk of making a rule too complex to be effective, because of the number of matches needed.

  • Only capture the fields that will be used for correlation later on.
  • You may find that you need to create a more generic rule to capture any events that remain after specific rules have all been applied.

  • Choose a pre-check string. The plugin uses this string to search the log line before applying the rule. This acts as a filter to avoid applying the rule to a line that cannot match, improving plugin performance. For an example, see the example following step 3 in Add a New Rule to a Plugin. The fourth statement from the top shows Pre-check="Teardown", followed by the regex.

  • Order the rules starting with 0001 and finishing with 9999. Create groups numbered 0002, 0003, and so forth, leaving room for future expressions.

  • Your rules based on specific matching criteria should always be the lowest numbers and the more generic rules, the highest. This helps avoid event masking, which can occur when USM Appliance loads generic rules before specific rules.

    When a log line matches a rule, USM Appliance generates an event and does not match it to any other the rule in the queue.

  • The plugin_sids in a sql file do not need to be continuous. You may insert gaps, if needed, to make the file more maintainable. For instance, you can reserve Plugin_sids for each group of event types in the following way

    • 1000 to 1999
    • 2000 to 2999

    This method works even if you do not have 1000 different events.

  • USM Appliance supports a number of built-in functions to simplify the process of converting information extracted from logs into normalized events. In addition, you can create your own custom functions. (See Define and Use Custom Functions).

  • Note: Be careful if you add a custom function to a plugin or if you access a proprietary database. This can degrade performance if not well designed. Custom plugins might take up to five minutes to appear in the USM Appliance web interface after you add them.