Collect Amazon CloudWatch Logs

Role Availability Read-Only Investigator Analyst Manager

Amazon CloudWatch Logs monitors applications and systems using log data, aggregating and storing application logs. CloudWatch Logs is useful because you can easily configure it to process additional metadata with the log files. Visit the AWS documentation to learn more about VPC flow log collection.

Important: If you choose to enable CloudWatch Logs in your Amazon Web Services (AWS) environment, you should make sure that you are not collecting more data than you need because this service incurs AWS costs based upon usage. See the CloudWatch pricing information to plan and configure your usage.

If not already done, install and configure the Amazon CloudWatch agent to collect logs from Amazon Elastic Compute Cloud (EC2) instances. See Amazon documentation for instructions.

USM Anywhere provides some CloudWatch log collection jobs out of the box, but they are disabled by default. You can enable them under Settings > Scheduler. When enabled, these jobs monitor certain log groups and collect logs from CloudWatch every five minutes. You must configure your CloudWatch agent to use these log group names and to keep the log types the same within a given log group.

USM Anywhere Log Collection Jobs and CloudWatch Log Groups
USM Anywhere Log Collection Job Name CloudWatch Log Group Name

Default File Path

Date Format

CloudWatch - Apache-Access-Logs Apache-Access-Logs /var/log/apache2/access.log %d/%b/%Y:%H:%M:%S
CloudWatch - Linux-Audit-Logs Linux-Audit-Logs /var/log/audit/audit.log Use the default
CloudWatch - Linux-Auth-Logs Linux-Auth-Logs /var/log/auth.log %b %d %H:%M:%S
CloudWatch - Osquery-Logs OSQuery-Logs /var/log/osquery/osqueryd.results.log Use the default

If you want to collect logs from other log groups, ensure that all streams in the same group are of the same type so that USM Anywhere can use a designated data source to parse the collected raw log data. You can then set up a CloudWatch log collection job for each log group.

To create a new CloudWatch log collection job

  1. Go to Settings > Scheduler.
  2. In the left navigation menu, click Log Collection.

    Note: You can use the Sensor filter at the top of the list to review the available log collection jobs on your AWS Sensor.

  3. Click Create Log Collection Job.

    Click Create Log Collection Job to add a scheduled log collection job

    Note: If you have recently deployed a new USM Anywhere Sensor, it can take up to 20 minutes for USM Anywhere to discover the various log sources. After it discovers the logs, you must manually enable the AWS log collection jobs you want before the system collects the log data.

    The Schedule New Job dialog box opens.

    Schedule New Job Dialog Box

  1. Enter the name and description for the job.

    The description is optional, but it is a best practice to provide this information so that others can easily understand what it does.

  2. Select Sensor as the source for your new job.
  3. In the Action Type drop-down list, select Amazon Web Services.
  4. In the App Action drop-down list, select Monitor CloudWatch.

    Select the AWS sensor, the Amazon Web Service app, and the Monitor CloudWatch action

  5. Enter the Region Name, Group Name, and Stream Name information for your AWS account. Region name can be an asterisk ( * ) to monitor all regions for a given group.

  6. In Source Format, select either of the following log formats:

    • Syslog: All messages transmitted to USM Anywhere are processed with the assumption that they are syslog formatted.

      When you choose syslog as the source format, the data source selection is bypassed and USM Anywhere uses the auto-detect hints from the data sources to match the incoming messages to the correct data source.

    • Raw: Use for non-syslog formatted data.

      If you select this option, you must choose the data source that USM Anywhere will use to parse all of the streams in the group. For example, to collect Amazon Virtual Private Cloud (VPC) flow logs, select the VPC Flow Logs data source.

      Specify the region name, group name, and source format for collecting the CloudWatch logs

      Important: If a group contains streams of mixed log formats, USM Anywhere parses all of them with the data source that you chose, which produces undesired results. In this case, you need to configure CloudWatch to separate the streams into different groups so that each contains only a single log type that can be mapped to the correct data source.

  7. In the Schedule section, specify when USM Anywhere runs the job:

    1. Select the increment as Minute, Hour, Day, Week, Month, or Year.

      Warning: After a frequency change, monitor the system to check its performance. For example, you can check the system load and CPU. See USM Anywhere System Monitor for more information.

    2. Set the interval options for the increment.

      The selected increment determines the available options. For example, on a weekly increment, you can select the days of the week to run the job.

      Set the schedule for the job to run each week

      Or on a monthly increment, you can specify a date or a day of the week that occurs within the month.

      Set the schedule for the job to run each month

    3. Important: USM Anywhere restarts the schedule on the first day of the month if the option "Every x days" is selected.

    4. Set the start time.

      This is the time that the job starts at the specified interval. It uses the time zone configured for your USM Anywhere instance (the default is Coordinated Universal Time [UTC]).

  8. Click Save.

    USM Anywhere detects any enabled jobs with the same configuration and asks you to confirm before continuing. This is because having two jobs with the same configuration generates duplicate events and alarms.

The following video demonstrates how to configure AWS to capture VPC flow logs and how to configure USM Anywhere to retrieve this information to create events:

Related Video Content

To view other related training videos, click here.