📘

Choosing the right design pattern for GCP logging sinks

There is no single design for GCP logging sinks that is right for all organizations. Reach out to Netography Support if you would like further guidance in this area.

We would be happy to setup a design session to discuss your specific organization's use case and requirements and determine the best approach together, or review a proposed design before you implement it.

Using an aggregated sink for onboarding all projects in a GCP organization or folder

If you are onboarding all the projects in a GCP organization, or all the projects that are children of a folder, you can use an aggregated sink to simplify the deployment. Using an aggregated sink lets you create 1 sink for a GCP organization or folder rather than 1 sink per project.

When you create an aggregated sink following these steps, all flow logs that are enabled in all child projects (including nested folders) will be routed to the aggregated sink. This will include any new projects, VPCs, or subnetworks that get added as children, and any new flow logs that are enabled will be automatically included.

An aggregated sink at the organization or folder level is ideal if you want to onboard all enabled flow logs within an organization or folder. If you have multiple folders to onboard (that are not nested within each other), you can create 1 aggregated sink for each folder, and route each of those sinks to the same Pub/Sub topic.

Using exclusion filters to exclude project(s) or subnetwork(s)

If you want to include all the enabled flow logs by default, but exclude specific projects or subnetworks (or any other criteria you can write a filter for in GCP), you can add up to 50 exclusion filters to a sink (and each filter can be 20k characters with logical operators).

To exclude a project: logName:projects/PROJECT_ID

To exclude a subnetwork: resource.labels.subnetwork_name"=SUBNET_NAME"

For more filter examples, see GCP Logging > Sample queries.

Excluding newly enabled flow logs

If you are in an organization that uses VPC flow logs for multiple use cases, such as application troubleshooting directly in GCP, you may face the circumstance where an application team needs to enable VPC flow logs for a set of subnetworks that will generate a very high volume of logs, but you do not want to onboard these flow logs to Fusion.

The default behavior of a sink is to include all these logs by default.

In this case, you will want to ensure any GCP administrator that can enable flow logs is aware of what folders have sinks that will capture these logs by default, and that an exclusion filter needs to be added to the sink BEFORE these flow logs are enabled to avoid creating an undesired spike in flow log volume.

Manually including newly enabled flow logs

If you are onboarding a limited scope of projects or subnetworks to Fusion and want to maintain tighter control over what flow logs are onboarded, so that another GCP administrator can not inadvertently start sending flow logs to Fusion by enabling flow logs in a subnetwork they control, you may want to instead use a sink design that only includes the enabled flow logs that you specify and not any onboard any newly enabled flow logs by default.

In this case, instead of using an exclusion filter for the sink, you can use the Inclusion Filter to specify only the specific projects, subnetworks, or other criteria you want to use. The same filters shown for exclusions above can be used in the inclusion filter to include only those matching the filter.

You can end up with a very long inclusion filter if you individually include each project or subnetwork by name, and if you attempt to do this for hundreds of criteria, you will reach the 20K character limit for a filter and no longer be able to add more. Use this pattern in cases where the filter you will write and its length will not approach that limit or become too complex to manage.

Additional steps when creating an aggregated sink

To use an aggregated sink, you will need Owner access to the sink's destination, and to perform the following steps when creating the sink:

1. Select the organization or folder to onboard in the GCP project picker.
2. When creating the sink, select Include logs ingested by this folder and all child resourcesin the section Choose logs to include in sink (this option will not appear if you selected a project).
3. Add the sink's writer identity as a principal by using IAM, and then grant it the Pub/Sub Publisher role (roles/pubsub.publisher). See GCP: Route logs to supported destinations > Set destination permissions. This step may not be required in your organization.

For more information on aggregated sink configuration, see GCP: Collate and route organization- and folder-level logs to supported destinations

Follow these steps using the configuration settings below: GCP: Create a sink.