CDK CodePipeline fails to output logs when deployed to a custom VPC, how to fix?

Question

Hi everyone,
Help is very appreciated!
I'm managing a code pipeline with CDK and when I deploy it to a custom VPC with an internet gateway (public subnet) I fail to see any logs in CodeBuild. Here is my CDK Code:

```
const pipeline = new CodePipeline(this, id, {
      pipelineName: `Hubs-CDK-Pipeline-${id}`,
      selfMutation: false,
      synth: new ShellStep('Synth', {
        input: CodePipelineSource.gitHub(
          'stafflink-pty-ltd/sauron',
          id === Environment.STAGING ? 'aws' : 'main',
          {
            authentication: SecretValue.secretsManager('manavs-github-token', {
              jsonField: 'token'
            })
          }
        ),
        primaryOutputDirectory: 'cdk/cdk.out',
        commands: [
          `node -v`,
          `sudo npm i -g n --force`,
          `n lts`,
          `n prune`,
          `node -v`,
          `npm i -g yarn`,
          `yarn`,
          `yarn rw setup deploy serverless`,
          `rm -f .env`,
          `sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64`,
          `sudo chmod a+x /usr/local/bin/yq`,
          `yq eval-all -i '.provider.vpc.securityGroupIds |= ["${lambdaSG.securityGroupId}"]' api/replace.yml`,
          `yq eval-all -i '.provider.vpc.subnetIds |= ["${vpc.isolatedSubnets[0].subnetId}", "${vpc.isolatedSubnets[1].subnetId}","${vpc.isolatedSubnets[2].subnetId}"]' api/replace.yml`,
          `yq eval-all -i '.provider.iam.role.statements |= [{"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": ["${bucket.bucketArn}"]}]' api/replace.yml`,
          `yq eval-all -i '. as $item ireduce ({}; . * $item)' api/serverless.yml api/replace.yml`,
          `yq eval-all -i '. as $item ireduce ({}; . * $item)' web/serverless.yml web/replace.yml`,
          `cat api/serverless.yml`,
          `npm run ci:build`,
          `npm run ci:migrate`,
          `yarn rw deploy aws`,
          `cd cdk`,
          `npm i`,
          `npm i -g aws-cdk`,
          `cdk synth`,
          `cd ..`
        ]
      }),
codeBuildDefaults: {
        vpc,
        subnetSelection: { subnetType: SubnetType.PUBLIC },
        securityGroups: [codePipeSG],
        rolePolicy: [
          new iam.PolicyStatement({
            effect: Effect.ALLOW || undefined,
            actions: [
              'logs:CreateLogGroup',
              'logs:CreateLogStream',
              'logs:PutLogEvents'
            ],
            resources: ['*']
          }),
          new iam.PolicyStatement({
            effect: Effect.ALLOW || undefined,
            actions: [
              's3:Abort*',
              's3:DeleteObject*',
              's3:GetBucket*',
              's3:GetObject*',
              's3:List*',
              's3:PutObject',
              's3:PutObjectLegalHold',
              's3:PutObjectRetention',
              's3:PutObjectTagging',
              's3:PutObjectVersionTagging'
            ],
            resources: ['*']
          })
        ],
        buildEnvironment: {
          computeType: ComputeType.LARGE,
          buildImage: LinuxBuildImage.STANDARD_5_0,
```

Here is my security group:

```
const codePipeSG = new SecurityGroup(this, 'code-pipeline-security-group', {
      vpc,
      allowAllOutbound: true,
      securityGroupName: `hubs-codepipe-${id}`
    })
```

Here is my VPC:

```
const vpc = new Vpc(this, 'VPC', {
      cidr: id === Environment.PROD ? '10.1.0.0/16' : '10.0.0.0/16',
      natGateways: 0,
      maxAzs: 3,
      subnetConfiguration: [
        {
          name: `public-${id}-1`,
          subnetType: SubnetType.PUBLIC,
          cidrMask: 24
        },
        {
          name: `isolated-${id}-1`,
          subnetType: SubnetType.PRIVATE_ISOLATED,
          cidrMask: 28
        }
      ],
      vpcName: `hubs${id}`
    })

```

Accepted Answer

Hello,

Thanks for sharing the relevant code-snippets for your CDK Application. From this, I can see that you've configured the following:
* VPC with a subnet configuration to include `PUBLIC` (access to internet via an internet-gateway) and `PRIVATE_ISOLATED` (no access to the internet) subnets. 
* Additionally, the CodeBuild project is configured to make use of the `PUBLIC` subnet during builds.

I'm suspecting that the build-container is unable to stream logs to CloudWatch as it was not able to connect to the CloudWatch Logs service endpoint. In order to establish internet-connectivity you would need to configure your CodeBuild project to make use of subnets of the type `PRIVATE_WITH_NAT` (access to the internet via NAT).

This can be done by making the following modifications to your CDK application:

* VPC Subnet Configuration:

```
subnetConfiguration: [
    {
        name: `public-${id}-1`,
        subnetType: SubnetType.PUBLIC,
        cidrMask: 24
    },
    {
        name: `private-nat-${id}-1`,
        subnetType: SubnetType.PRIVATE_WITH_NAT,
        cidrMask: 24
    },
    {
        name: `isolated-${id}-1`,
        subnetType: SubnetType.PRIVATE_ISOLATED,
        cidrMask: 28
    }
]
```

* CodeBuild configuration would need to select the subnets of type `PRIVATE_WITH_NAT`:

```
   subnetSelection: { subnetType: SubnetType.PRIVATE_WITH_NAT },
```

Please refer to the following excerpt from the documentation [Use AWS CodeBuild with Amazon Virtual Private Cloud](https://docs.aws.amazon.com/codebuild/latest/userguide/vpc-support.html) for more details:

> You need a NAT gateway or NAT instance to use CodeBuild with your VPC so that CodeBuild can reach public endpoints (for example, to run CLI commands when running builds). You cannot use the internet gateway instead of a NAT gateway or a NAT instance because CodeBuild does not support assigning Elastic IP addresses to the network interfaces that it creates, and auto-assigning a public IP address is not supported by Amazon EC2 for any network interfaces created outside of Amazon EC2 instance launches.

CDK CodePipeline fails to output logs when deployed to a custom VPC, how to fix?

Relevant content