AWS S3 Transit Gateway Flow Logs

This document provides instructions for configuring the collection of AWS Transit Gateway Flow Logs with an S3 bucket and configure log notification with SNS and SQS using the AWS Console.

🚧

It is recommended that the s3 bucket is in the same region as the VPC. If you pointed multiple flow logs to the same bucket they will need to be differentiated by the folder prefix.

AWS Console Steps

• Create S3 bucket
• Create an IAM user
• Optional: Create the SNS Topic
• Optional: Create the SQS queue
• Update S3 bucket
• Publish to S3

Create S3 bucket

1. In the AWS Console select Services and type s3 into the search bar

2. Enter your bucket name, select your region, optionally add tags and click "create bucket"

   

3. From the S3 bucket listing check the box for the bucket you created and click Copy Bucket ARN to make note of it.

   

OPTIONAL: Create the SNS topic

1. In the AWS Console select Services and type sns into the search bar
2. Select "Standard" type.
3. Enter a name for the SNS topic.
4. Add optional tags and click "Create Topic"

OPTIONAL: Create the SQS queue

1. In the AWS Console select Services and type sqs into the search bar

2. Enter a queue name and click configure queue

3. Set Message Retention Period to 1 day.

   

4. Under Access Policy select advanced and use the following JSON. Update the SourceArn with your S3 ARN.

   JSON

   {
      "Version": "2012-10-17",
      "Id": "PushMessageToSQSPolicy",
      "Statement": [
         {
            "Sid": "allow-sns-to-send-message-to-sqs",
            "Effect": "Allow",
            "Principal": {
               "AWS": "*"
            },
            "Action": "sqs:SendMessage",
            "Resource": "*",
            "Condition": {
               "StringLike": {
                  "aws:SourceArn": "arn:aws:s3:::<bucketname>"
               }
            }
         }
      ]
   }
   

5. Select your SQS queue under the SNS Subscriptions tab click "Subscribe to Amazon SNS Topic".

   

6. Choose your SNS topic and Subscribe then click "Save".

7. Make note of your SQS URL and SQS ARN

Create Netography Policy

1. In the AWS Console select Services and type "iam" into the search bar

2. Under Access management click Policies

3. Click create policy and then the JSON tab.

4. Use the following after updating arn:aws:s3.

5. Then click review and then create

   JSON

   {
      "Version":"2012-10-17",
      "Statement":[
         {
            "Sid":"VisualEditor0",
            "Effect":"Allow",
            "Action":[
               "sqs:DeleteMessage",
               "sqs:GetQueueUrl",
               "sqs:ReceiveMessage",
               "sqs:GetQueueAttributes",
               "s3:ListBucket*",
               "s3:GetObject*",
               "s3:DeleteObject*"
            ],
            "Resource":[
               "<sqs arn>",
               "arn:aws:s3:::<bucketname>/*",
               "arn:aws:s3:::<bucketname>"
            ]
         }
      ]
   }
   

Create an IAM user

1. In the AWS Console select Services and type iam into the search bar

2. Click Add user and enter a user name

3. Check Programmatic access for the Access type and click next

4. Click Attach existing policies directly

   

5. Enter the policy name you created

6. Select it and click next

7. Fill in your tags (optional) and click next

8. Review and create user

9. Make note of Access Key ID and Secret access Key

10. From the Users table in IAM click the user you create and make note of the user ARN

Update S3 bucket

1. Select your bucket and click the properties tab

   

2. Scroll down to Event notifications and click "Create event notification"

   

3. Enter a name.

4. Check All object create events.

5. Use SQS Queue for Destination.

6. Select your SQS Queue for SQS Queue.

7. Click "Save".

   

Create Transit Gateway Flow Logs that Publish to S3

1. In the AWS Console select Services and type transit into the search bar
2. Click Transit gateways then select your Transit gateway ID link.
3. Then click the create flow log button.

Create flow log screen.

1. Select "Send to an S3 bucket" for Destination

2. Under S3 bucket ARN fill in your s3 bucket ARN.

3. Select Custom format and select all available attributes. 

4. Click the Create flow log button.

   

   
   

Netography Portal Steps

1. Navigate to Settings > Traffic Sources,
2. Click Add Traffic Source.
3. Click the AWS S3 Transit Gateway tile.

Configuration

The path to the S3 bucket ARN is constructed using the Account ID and Region fields, along with the current date, using the following structure: AWSLogs/{Account ID}/vpcflowlogs/{Region}/YYYY/MM/DD/

Example: AWSLogs/123456789012/vpcflowlogs/us-east-1/2023/06/28/

The Prefix field can be used if the flow logs are being organized in folders. e.g. setting the Prefix to folder_name would modify the above to become folder_name/AWSLogs/123456789012/vpcflowlogs/us-east-1/2023/06/28/

The following fields are specific to the AWS S3 configuration.

AWS authentication

Netography Fusion can access your AWS account using one of two different methods:

1. IAM user via an Access Key ID & Secret Access Key
2. IAM Roles using a Custom Trust Policy created by Netography.

AWS Access Key

To configure access via Access Key/Secret, select the "Key/Secret" Authentication Type. The values for the ID and Secret are accessible in the AWS IAM console.

AWS IAM Roles

You can use an IAM role in Netography Fusion to access your Cloud Flow Logs for flow ingest or account data for the AWS Context Integration. To enable this, go to the portal and retrieve the AWS Account ID and External ID from your Account Settings. Navigate to the gear button on the top right to view your Account Settings to see the Overview tab as shown below:

In AWS, you will configure permissions using the Account ID grabbed from above to create the IAM Role. When configured, AWS creates the Amazon Resource Number (ARN) for the role. For more information in configuring the permissions to the Account ID, refer to the following AWS guide:

The newly created ARN is required in order to configure IAM role access in the Netography Fusion portal.

Once the ARN has been created, the remaining steps are to toggle the Authentication Type to Role in your AWS

S3 configuration settings, input the AWS Account ID grabbed earlier from your Netography account settings, and the supply the ARN configured from AWS as shown below: