Manually Create and Configure an Export Sink for Your GCP Sensor
The export sink is what defines which logs are exported to a particular topic. You can create a single sink to export all of the logs you want your to receive, or create any number of individual sinks to group your exported logs by type, to maximize performance, or for any other reason that suits your specific implementation.
To create an export sink for a project or organization
- Log in to your GCP environment and go to the organization or project for which you want to create this sink.
- Go to the Exports page under Logging.
- Click Create Export.
- Enter the following information:
- Sink Name: Enter an identifiable name for this export sink
- Sink Service: Using the drop-down list, select Cloud Pub/Sub
- Sink Destination: Using the drop-down list, select the topic you created for this sink
Note: If you haven't yet created a topic for this sink, you can select Create New Topic to create one from this page and immediately use it for your sink. If you do so, you must remember to go to that topic and create a subscription for it or your sensor will not receive any logs from it.
- Configure a filter for this sink, following the guidelines in Configuring Export Sink Filters.
- Click Create Export.
Important: If your sink and topic are in different GCP projects, or if you are exporting organization-level logs to a Google Cloud Pub/Sub topic in a project, you must complete some additional steps. See the following sections for detailed instructions regarding those two cases.
To create a sink that publishes to a Cloud Pub/Sub topic in a different project
Note: If you have not already granted your service account permission to this second project, first use the instructions in Preparing Your GCP Environment for Sensor Deployment to grant permission to this project now. Be sure to restart the sensor app before proceeding on to step one.
- Log in to your GCP environment and go to the project for which you want to create this sink.
- Go to the Exports page under Logging.
- Click Create Export.
- Enter the following information:
- Sink Name: Enter an identifiable name for this export sink
- Sink Service: Using the drop-down list, select Cloud Pub/Sub
- Sink Destination: Using the drop-down list, select Use a Cloud Pub/Sub topic in another project
- Configure a filter for this sink, following the guidelines in Configuring Export Sink Filters.
- Click Create Export.
When you make your selection in Sink Destination, the menu item transforms into a text field. Use that field to enter the following, substituting your relevant information where there are variables:
pubsub.googleapis.com/projects/<project-id>/topics/<topic_name>
Where the <project-id> you reference is the project your topic resides in.
To create a sink to publish from an organization to a topic in a project
Important: Unlike the previous methods, it is not possible to use the web user interface (UI) to create an export sink to publish from the organization level to a topic at the project level. Instead, use the Google Cloud Shell Editor native to your GCP environment to enter the following commands.
- Access the Cloud Shell editor in your GCP environment by clicking on the Activate Cloud Shell button.
This opens a new window at the bottom of your screen, which may take a few minutes to finish loading. - Use the following command to create a new sink for your organization:
- Use the following command to grant the service account the permissions it requires:
gcloud logging sinks create \ <sink-name> \ --organization=<organization-id> \ --include-children \ pubsub.googleapis.com/projects/<project-name>/topics/<topic-name> \ --log-filter "logName=(\"organizations/<organization-id>/logs/cloudaudit. googleapis.com%2Factivity\" OR \"organizations/<organization-id>/logs/ cloudaudit.googleapis.com%2Fdata_access\" OR \"organizations/<organization-id> /logs/cloudaudit.googleapis.com%2Fsystem_event\")"
This returns the following message. Make note of the service account name (highlighted here in bold) to enter in the next step.
Created [https://logging.googleapis.com/v2/organizations/<organization_id/ sinks/<sink_name>]. Please remember to grant `serviceAccount:<name-of-sensor-service-account>@ <name-of-project>.iam.gserviceaccount.com.com` the Pub/Sub Publisher role on the topic. More information about sinks can be found at https://cloud.google.com/logging/docs/ export/configure_export
gcloud organizations add-iam-policy-binding <organization_id> \ --member=<name-of-sensor-service-account>@<name-of-project>.iam. gserviceaccount.com> \ --role=roles/pubsub.publisher
Configuring Export Sink Filters
The filter configured for your export sink determines which logs that sink exports to your topic.
To configure the filters for your sink
- Go to the export sink for which you wish to create a filter.
You can do this either when you first created it or by opening it any time after that for editing. - Click the carrot in the text box of your export filter and select Convert to advanced filter.
- Use the specifications described in the table below to define which filters will be exported by this sink, separating each filter specification with "OR" (as seen in the image above).
Note: Any logs included in your filter but not supported by the GCP Sensor will be discarded by the sensor.