GCP Cloud DNS Logs via Pub/Sub Setup

Netography Fusion ingests Google Cloud Platform (GCP) Cloud DNS logs via a GCP Pub/Sub subscription. The steps to integrate with GCP are:

1. Prerequisite: If you have a Domain Restricted Sharing Organization Policy, add Netography to it

2. Enable Cloud DNS logs

3. Create a Pub/Sub topic

4. Create a Cloud Logging Sink Pub/Sub for the topic

5. Create a Pub/Sub Pull Subscription to the topic

6. Add Netography's GCP service account as a principal for the Pub/Sub subscription

7. In Fusion, Add GCP as a new DNS traffic source

ℹ️Cloud DNS and VPC Flow Logs use a separate Pub/Sub Subscription in Fusion

Fusion reads GCP Cloud DNS logs from a separate Pub/Sub Subscription than GCP VPC flow logs. If you have already setup GCP VPC flow logs, the easiest option is to follow the instructions below to create a new Cloud Logging Sink, Pub/Sub Topic, and Pub/Sub Pull Subscription for Cloud DNS.

It is possible to create a combined Cloud Logging Sink and Pub/Sub Topic for both VPC flow logs and Cloud DNS logs, and then use a Pub/Sub Subscription filter to deliver only logs to the correct destination, but this design incurs Pub/Sub message delivery fees for all the messages filtered out of each subscription, so it is not recommended.

👍You can onboard DNS logs for an entire GCP organization or folder by following these steps one time

You only need to create 1 GCP Pub/Sub topic, 1 GCP Cloud aggregated Logging Sink, 1 GCP Pub/Sub Subscription, and 1 Fusion GCP DNS traffic source to onboard GCP Cloud DNS logs to Fusion across networks, projects, and folders. If you need more granular control over which Cloud DNS logs should be routed to Netography, create 1 GCP Pub/Sub topic, 1 GCP Pub/Sub Subscription, and 1 Fusion GCP DNS traffic source. You can then make as many Cloud Logging Sinks as you need to route all the Cloud DNS logs to the one topic you created.

In addition to ingesting Cloud DNS logs, you may want to also ingest VPC flow logs, and enrich IP addresses in Fusion from GCP with context with the GCP Context Integration.

Prerequisites

If you have a Domain Restricted Sharing Organization Policy

If your GCP organization has an Organization Policy constraint for Domain Restricted Sharing constraints/iam.allowedPolicyMemberDomains, you must add a rule to that policy to allow Netography's GCP customer ID C04ddcbu8before adding the principal to the Pub/Sub subscription.

If you have GCP VPC flow logs in Fusion and use the same project for the Pub/Sub topic and subscription, you already have this rule if it is needed.

This constraint is the default setting for all GCP organizations created on or after May 3, 2024.

If this policy restriction exists and you do not add the rule, you will receive the following error when you save the Pub/Sub Subscription: IAM policy update failed - The ‘Domain Restricted Sharing’ organization policy (constraints/iam.allowedPolicyMemberDomains) is enforced.

For detailed instructions and options for configuration, see target="_blank">GCP: Restricting Domains

Domain Restricted Sharing Configuration

Field	Value
Custom Value	C04ddcbu8

GCP Console Steps

You must be an Organization Policy Administrator (roles/orgpolicy.policyAdmin) to perform these steps. The Organization Administrator role does NOT contain these permissions.

To update your Organization Policy to allow you to grant Netography's GCP service account access to the Pub/Sub subscription:

1. Go to the Organization Policies page in the Google Cloud console IAM & Admin section.

2. From the project picker (the box directly to the right of the Google Cloud logo at the top of your GCP console), select your GCP organization (you can choose the project you will create the Pub/Sub subscription if you prefer a more granular policy).

3. Next to where it says Filter above the list of policies, type Domain restricted sharing.

4. You should see 1 policy with that name in the list, with ID constraints/iam.allowedPolicyMemberDomains. Click ⋮ and select Edit Policy.

5. Under Policy source, select the Override parent's policy button.

6. Under Policy enforcement, select Merge with parent. 6a. Under Rules, if you see an existing rule with a ⌄, follow this step: click the ⌄. It will open a box that says Edit Rule. In that box, select the ADD VALUE button. It will create a new empty box above the button. In that box enter C04ddcbu8. 6b. Under Rules, if you see a warning that At least one rule is required in organization policies., click the ADD A RULE button below it. It will open a New Rule box. In the Policy values drop-down, select Custom. In the Policy type drop-down, select Allow. In the empty box under Custom values, enter C04ddcbu8.

7. Select the Set Policy button at bottom of the page.

GCP Setup Steps

1. Enable Cloud DNS Logs

You can skip this step if Cloud DNS logs are already enabled for VPC networks and public zones you want to monitor.

Follow these steps to enable Cloud DNS logs: GCP > Cloud DNS > Use logging and monitoring.

2. Create a Cloud Pub/Sub topic

Create a Cloud Pub/Sub topic to publish Cloud DNS logs to. If you are onboarding an individual GCP project, you can create the topic as part of creating the sink in step 3. If you are onboarding multiple projects at an organization or folder level, you can create a single topic in a designated project that you will use for centralized logging resources, and then use this one topic as the destination for a single aggregated sink, multiple individual project Cloud Logging Sinks, or a combination of the two.

To separately create the topic, follow these steps using the configuration settings below: GCP: Create a Topic

Pub/Sub Topic Configuration

Field	Value
Topic ID	Any value ( e.g. neto-dnslogs-pubsub-topic)
Add a default subscription	No
Use a schema	No
Enable ingestion	No
Enable message retention	Yes- 1 Day

Note: GCP charges for unacknowledged message retention over one day. In most circumstances, the messages will be acknowledged and removed from the topic in real time, but retention will ensure no data is lost unless the logs are not read in that period. You can adjust the retention period based on your organization's requirements.

GCP Console Steps

1. Go to the Pub/Sub Topics page in the Google Cloud console.

2. Click Create Topic.

3. Fill out the form using the above configuration values, then click Save.

3. Create a Cloud Logging Sink Pub/Sub

Create a Cloud Logging Sink with a destination of Cloud Pub/Sub topic, using the topic you created in step 2 or creating the topic in the process.

ℹ️Using an aggregated sink for onboarding all projects in a GCP organization or folder

If you are onboarding all the projects in a GCP organization, or all the projects that are children of a folder, you can use an aggregated sink to simplify the deployment. Using an aggregated sink lets you create 1 sink for a GCP organization or folder rather than 1 sink per project.

When you create an aggregated sink following these steps, all logs that are enabled in all child projects (including nested folders) will be routed to the aggregated sink. This will include any new projects that get added as children and any new enabled logs will be automatically included.

An aggregated sink at the organization or folder level is ideal for onboarding all enabled logs within an organization or folder. If you have multiple folders to onboard (not nested within each other), you can create one aggregated sink for each folder and route each of those sinks to the same Pub/Sub topic.

Choosing the correct design pattern for GCP logging sinks

There is not one design for GCP logging sinks that is right for all organizations. Contact Netography Support if you want further guidance in this area. We would be happy to set up a design session to discuss your specific organization's use case and requirements and determine the best approach together, or we could review a proposed design before you implement it.

Using exclusion filters to exclude project(s) or subnetwork(s)

If you want to include all the enabled flow logs by default but exclude specific projects or subnetworks (or any other criteria you can write a filter for in GCP), you can add up to 50 exclusion filters to a sink (and each filter can be 20k characters with logical operators).

To exclude a project: logName:projects/PROJECT_ID

For more filter examples, see GCP Logging > Sample queries.

Additional steps when creating an aggregated sink

To use an aggregated sink, you will need Owner access to the sink's destination, and to perform the following steps when creating the sink:

1. Select the organization or folder to onboard in the GCP project picker.

2. When creating the sink, select Include logs ingested by this folder and all child resources in the section Choose logs to include in sink (this option will not appear if you selected a project).

3. Add the sink's writer identity as a principal by using IAM, and then grant it the Pub/Sub Publisher role ( roles/pubsub.publisher). See GCP: Route logs to supported destinations > Set destination permissions. This step may not be required in your organization.

For more information on aggregated sink configuration, see GCP: Collate and route organization- and folder-level logs to supported destinations

Follow these steps using the configuration settings below: GCP: Create a sink.

Cloud Logging Sink Configuration

Field	Value
Sink name	Any value ( e.g. neto-dnslogs-sink)
Sink description	Any value (e.g. Netography Fusion DNS log ingest)
Sink destination service type	Cloud Pub/Sub topic
Sink destination Cloud Pub/Sub topic	Create a topic or use topic created in previous step
Inclusion filter	resource.type="dns_query"
Enable message retention	Yes- 1 Day

Inclusion Filter

The inclusion filter resource.type="dns_query" will include all Cloud DNS logs in the sink. You can add filters using inclusion or exclusion based on your desired configuration.

GCP Console Steps

1. Go to the Log Router page in the Google Cloud console.

2. Select the project (or folder or organization if using an aggregated sink) to create the sink in.

3. Click Create sink.

4. Fill out the form using the above configuration values, then click Save

4. Create a Pub/Sub Pull Subscription to the topic

Follow these steps using the configuration settings below: GCP: Create a pull subscription.

Pub/Sub Subscription Configuration

Field	Value
Subscription ID	Any value ( e.g. neto-dnslogs-sub)
Cloud Pub/Sub Topic	Topic ID from previous steps (if creating from Subscriptions page)
Delivery Type	Pull
Message retention duration	1 Day (or based on your requirements)
Retry policy	Retry after exponential backoff delay (Default min/max values)

Default values for all other fields can be used.

GCP Console Steps

1. Go to the Topics page in the Google Cloud console.

2. Click ⋮ next to the topic you created in previous step.

3. From the context menu, select Create Subscription.

4. Fill out the form using the above configuration values, then click Save.

Note: Alternatively, you can create a subscription from the Subscriptions page by entering the Topic ID from the previous step.

5. Add Netography's GCP service account as a principal to the Pub/Sub subscription

To grant Netography access to read logs from the Pub/Sub subscription, add the Netography GCP service account as a new principal in the subscription.

Follow these steps to add a principal to the subscription: GCP: Access Control for Pub/Sub > Controlling access through the Google Cloud Console

Pub/Sub Subscription Principal Configuration

Field	Value
Principal	[email protected]
Role	Pub/Sub Subscriber

GCP Console Steps

1. Go to the Subscriptions page in the Google Cloud console in the Pub/Sub section.

2. Select the subscription you created in the previous step to bring up the subscription info panel on right.

3. Select Add Principal in the info panel for the subscription.

4. Fill out the form using the above configuration values, then click Save.

Netography Fusion Setup

6. Add a new GCP DNS traffic source to Fusion

In the Fusion portal, click the gear icon to go to Settings, navigate to Traffic Sources, click Add Traffic Source, and under the DNS section, select GCP, and fill out the form using the configuration below.

GCP DNS Traffic Source Configuration

The following fields are specific to the GCP configuration.

Field	Required	Description
Project ID	yes	GCP Project ID containing the Pub/Sub subscription
Subscription ID	yes	GCP Pub/Sub Subscription ID (e.g. neto-dnslogs-sub)

Finding the Subscription ID:

• Subscription ID is the name you gave the subscription in the previous step.

• It is listed in Pub/Sub subscriptions in the GCP console in the table column Subscription ID.

• If you select a subscription by clicking the ID on that page, the Subscription detail page has the subscription ID directly below the Google Cloud logo (between the ← and Edit buttons).

• The subscription ID is the part of the subscription name after the last / (eg. if subscription name is projects/yourproject/subscriptions/neto-dnslogs-sub then subscription ID is neto-dnslogs-sub.