USM Anywhere™

Manually Create and Configure an Export Sink for Your GCP Sensor

The export sink is what defines which logs are exported to a particular topic. You can create a single sink to export all of the logs you want your Google Cloud Platform (GCP) Sensor to receive, or create any number of individual sinks to group your exported logs by type, to maximize performance, or for any other reason that suits your specific implementation.

To create an export sink for a project or organization

  1. Log in to your GCP environment and go to the organization or project for which you want to create this sink.
  2. Go to the Exports page under Logging.
  3. Click Create Export.
  4. Enter the following information:
    • Sink Name: Enter an identifiable name for this export sink
    • Sink Service: Using the drop-down list, select Cloud Pub/Sub
    • Sink Destination: Using the drop-down list, select the topic you created for this sink

Note: If you haven't yet created a topic for this sink, you can select Create New Topic to create one from this page and immediately use it for your sink. If you do so, you must remember to go to that topic and create a subscription for it or your sensor will not receive any logs from it.

  1. Configure a filter for this sink, following the guidelines in Configuring Export Sink Filters.
  2. Click Create Export.

Important: If your sink and topic are in different GCP projects, or if you are exporting organization-level logs to a Google Cloud Pub/Sub topic in a project, you must complete some additional steps. See the following sections for detailed instructions regarding those two cases.

To create a sink that publishes to a Cloud Pub/Sub topic in a different project

Note: If you have not already granted your service account permission to this second project, first use the instructions in Preparing Your GCP Environment for Sensor Deployment to grant permission to this project now. Be sure to restart the sensor app before proceeding on to step one.

  1. Log in to your GCP environment and go to the project for which you want to create this sink.
  2. Go to the Exports page under Logging.
  3. Click Create Export.
  4. Enter the following information:
    • Sink Name: Enter an identifiable name for this export sink
    • Sink Service: Using the drop-down list, select Cloud Pub/Sub
    • Sink Destination: Using the drop-down list, select Use a Cloud Pub/Sub topic in another project

    5. When you make your selection in Sink Destination, the menu item transforms into a text field. Use that field to enter the following, substituting your relevant information where there are variables:

    pubsub.googleapis.com/projects/<project-id>/topics/<topic_name>

    Where the <project-id> you reference is the project your topic resides in.

  5. Configure a filter for this sink, following the guidelines in Configuring Export Sink Filters.
  6. Click Create Export.

To create a sink to publish from an organization to a topic in a project

Important: Unlike the previous methods, it is not possible to use the web user interface (UI) to create an export sink to publish from the organization level to a topic at the project level. Instead, use the Google Cloud Shell Editor native to your GCP environment to enter the following commands.

  1. Access the Cloud Shell editor in your GCP environment by clicking on the Activate Cloud Shell button.
    This opens a new window at the bottom of your screen, which may take a few minutes to finish loading.

  2. Use the following command to create a new sink for your organization:

    3. gcloud logging sinks create \
	<sink-name> \
	--organization=<organization-id> \
	--include-children \
	pubsub.googleapis.com/projects/<project-name>/topics/<topic-name> \
	--log-filter "logName=(\"organizations/<organization-id>/logs/cloudaudit.
		googleapis.com%2Factivity\" OR \"organizations/<organization-id>/logs/
		cloudaudit.googleapis.com%2Fdata_access\" OR \"organizations/<organization-id>
		/logs/cloudaudit.googleapis.com%2Fsystem_event\")"

    This returns the following message. Make note of the service account name (highlighted here in bold) to enter in the next step.

    Created [https://logging.googleapis.com/v2/organizations/<organization_id/
	sinks/<sink_name>].
Please remember to grant `serviceAccount:<name-of-sensor-service-account>@
	<name-of-project>.iam.gserviceaccount.com.com` the Pub/Sub Publisher 
	role on the topic.
More information about sinks can be found at https://cloud.google.com/logging/docs/
	export/configure_export

  3. Use the following command to grant the service account the permissions it requires:

    4. gcloud organizations add-iam-policy-binding <organization_id> \
	--member=<name-of-sensor-service-account>@<name-of-project>.iam.
		gserviceaccount.com> \
	--role=roles/pubsub.publisher

Configuring Export Sink Filters

The filter configured for your export sink determines which logs that sink exports to your topic.

To configure the filters for your sink

  1. Go to the export sink for which you wish to create a filter.
    You can do this either when you first created it or by opening it any time after that for editing.
  2. Click the carrot in the text box of your export filter and select Convert to advanced filter.

  3. Use the specifications described in the table below to define which filters will be exported by this sink, separating each filter specification with "OR" (as seen in the image above).

Note: Any logs included in your filter but not supported by the GCP Sensor will be discarded by the sensor.

Log Types Supported by the GCP Sensor
Log Type Filter to Capture This Log Notes
Audit Logs at the Organization Level organizations/<organization-id>/logs/cloudaudit.googleapis.com

To filter these logs further, append:

  • %2Factivity: For activity logs
  • %2Fdata_access: For data access logs
  • %2Fsystem_event: For system events

Audit Logs at the Project Level

projects/<project-id>/logs/cloudaudit.googleapis.com

To filter these logs futher, append:

  • %2Factivity: For activity logs
  • %2Fdata_access: For data access logs
  • %2Fsystem_event: For system events
VPC Flow Logs projects/<project-id>/logs/compute.googleapis.com%2Fvpc_flows
Firewall Logs projects/<project-id>/logs/compute.googleapis.com%2Ffirewall
Syslog projects/<project-id>/logs/syslog These logs are delivered via the Stackdriver logging agent
Apache Logs projects/<project-id>/logs/apache
  • -access: For access logs
  • -error: For error logs
Nginx Logs projects/<project-id>/logs/nginx
  • -access: For access logs
  • -error: For error logs