How do I keep my AWS CDK application within the CloudFormation resource quota?

12 minute read

I deployed an AWS Cloud Development Kit (AWS CDK) application. I want to stay within the established AWS CloudFormation resource quota.

Short description

Each CloudFormation stack can have a maximum of 500 resources. This quota includes the resources that you define and the supporting resources that AWS CDK automatically creates.

For example, if you create an AWS Lambda function, then AWS CDK creates an associated AWS Identity and Access Management (IAM) execution role. If you set the @aws-cdk/aws-lambda:useCdkManagedLogGroup feature flag to true in your cdk.json file, then AWS CDK also creates an Amazon CloudWatch Logs log group. For 50 functions in a single stack, AWS CDK creates 100–150 resources: 50 functions, 50 execution roles, and 50 log groups.

Note: By default, cdk/aws-lambda:useCdkManagedLogGroup is set to true for projects that you create with AWS CDK version v2.200.1. If the flag is false, then Amazon CloudWatch creates the log group at runtime, and the log group doesn't count against your stack's resource quota.

To stay within the resource quota, create a new AWS CDK project, and then implement the following best practices:

Split your infrastructure into multiple stacks.
Use nested stacks.
Share IAM roles across Lambda functions.
Consolidate log groups.
Monitor resource counts with aspects.
Note: The following resolutions overwrite the bin/cdk-resource-quota.ts application entry point file for each best practice. You can use a single best practice, or combine them.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Prerequisites:

Install Node.js version 18.x or later. To install the tool, see Download Node.js on the Node.js website.
Install the AWS CDK CLI v2.
Bootstrap your environment to prepare it for the AWS CDK.

Create a new AWS CDK project

Complete the following steps:

Run the following command to create a project directory that's named cdk-resource-quota on your machine:

mkdir cdk-resource-quota && cd cdk-resource-quota

Run the following command to initialize the project:

cdk init --language typescript

Note: This resolution uses TypeScript. However, you can use another language, and then modify the code as needed.
The AWS CDK CLI creates a project with the following structure:

cdk-resource-quota/
├── bin/
│  └── cdk-resource-quota.ts   # App entry point
├── lib/
│  └── cdk-resource-quota-stack.ts # Default stack
├── package.json
├── tsconfig.json
└── cdk.json

Run the following command to install the project dependencies:
```
npm install
```
Synthesize the CloudFormation template.
Note: In the output, make sure that the CloudFormation template has the CDKMetadata resource.

Split your infrastructure into multiple stacks

Complete the following steps:

To create the network stack, in the lib directory, create a file that's named network-stack.ts with the following code:

import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { Construct } from 'constructs';

export class NetworkStack extends cdk.Stack {
 public readonly vpc: ec2.Vpc;

 constructor(scope: Construct, id: string, props?: cdk.StackProps) {
  super(scope, id, props);

  this.vpc = new ec2.Vpc(this, 'MainVpc', {
   maxAzs: 2,
   natGateways: 1,
  });
 }
}

Note: The preceding stack defines the shared networking resources and exposes them for other stacks to use.

To create the application stack, in the lib directory, create a file that's named application-stack.ts with the following code:

import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';

export class ApplicationStack extends cdk.Stack {
 constructor(scope: Construct, id: string, vpc: ec2.IVpc, props?: cdk.StackProps) {
  super(scope, id, props);

  new lambda.Function(this, 'ApiHandler', {
   runtime: lambda.Runtime.NODEJS_20_X,
   handler: 'index.handler',
   code: lambda.Code.fromInline(
    'exports.handler = async () => ({ statusCode: 200 });'
   ),
   vpc: vpc,
  });
 }
}

Note: The preceding stack accepts the virtual private cloud (VPC) from the network stack through its constructor.

To update the application entry point, open the bin/cdk-resource-quota.ts file, and then replace its contents with the following code:

#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { NetworkStack } from '../lib/network-stack';
import { ApplicationStack } from '../lib/application-stack';

const env = {
 account: process.env.CDK_DEFAULT_ACCOUNT,
 region: process.env.CDK_DEFAULT_REGION,
};

const app = new cdk.App();
const networkStack = new NetworkStack(app, 'NetworkStack', { env });
const appStack = new ApplicationStack(app, 'ApplicationStack', networkStack.vpc, { env });
appStack.addDependency(networkStack);

Note: The preceding configuration connects the stacks and sets the deployment order.

Run the following command to deploy the two stacks:
```
cdk deploy --all
```
Note: AWS CDK deploys NetworkStack first, and then ApplicationStack.

Important: If you share resources between stacks with cross-stack references, then CloudFormation creates exports in the producing stack. You can't modify or delete the exports until all consuming stacks remove their references. Make sure to avoid circular dependencies and deployment deadlocks. For more information, see Resources and the AWS CDK.

Use nested stacks

Nested stacks group resources under a single deployment. The parent stack counts each nested stack as one resource, and each nested stack can contain 500 resources of its own.

To use nested stacks, complete the following steps:

To create the nested stack, in the lib directory, create a file that's named lambda-nested-stack.ts with the following code:

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';

export class LambdaNestedStack extends cdk.NestedStack {
 constructor(scope: Construct, id: string, props?: cdk.NestedStackProps) {
  super(scope, id, props);

  for (let i = 0; i < 50; i++) {
   new lambda.Function(this, `Function${i}`, {
    runtime: lambda.Runtime.NODEJS_20_X,
    handler: 'index.handler',
    code: lambda.Code.fromInline(
     'exports.handler = async () => ({ statusCode: 200 });'
    ),
   });
  }
 }
}

Note: The nested stack collects a group of related Lambda functions so that they count against their own 500-resource quota, separate from the parent stack.

To create the parent stack, in the lib directory, create a file that's named main-stack.ts with the following code:

import * as cdk from 'aws-cdk-lib';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import { Construct } from 'constructs';
import { LambdaNestedStack } from './lambda-nested-stack';

export class MainStack extends cdk.Stack {
 constructor(scope: Construct, id: string, props?: cdk.StackProps) {
  super(scope, id, props);

  new LambdaNestedStack(this, 'LambdaStack1');
  new LambdaNestedStack(this, 'LambdaStack2');

  new dynamodb.Table(this, 'DataTable', {
   partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
  });
 }
}

The parent stack includes two nested stacks and an Amazon DynamoDB table.

To update the application entry point, open the bin/cdk-resource-quota.ts file, and then replace its contents with the following code:

#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { MainStack } from '../lib/main-stack';

const env = {
 account: process.env.CDK_DEFAULT_ACCOUNT,
 region: process.env.CDK_DEFAULT_REGION,
};

const app = new cdk.App();
new MainStack(app, 'MainStack', { env });

To deploy the parent stack, run the following command:
```
cdk deploy MainStack
```
Note: The parent stack counts only two nested stack references and one Amazon DynamoDB table. Each nested stack manages its own 50 Lambda functions.

Note: A nested stack hierarchy can create, update, or delete up to 2,500 resources in a single operation across all nested stacks. Each nested stack still has the 500-resource quota. For more information, see Understand CloudFormation quotas.

Share IAM roles across Lambda functions

Important: Use the resource's Amazon Resource Name (ARN) in the IAM policy to follow the principle of least privilege. It isn't a best practice to use wildcards (*) in the resources field.

By default, AWS CDK creates a separate IAM role for each Lambda function. If multiple functions require the same permissions, then create one shared role and assign the role to all the functions to reduce your resource count.

To configure a shared IAM role, complete the following steps:

To update the default stack, open lib/cdk-resource-quota-stack.ts, and then replace its contents with the following code:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';

export class CdkResourceLimitStack extends cdk.Stack {
 constructor(scope: Construct, id: string, props?: cdk.StackProps) {
  super(scope, id, props);

  // Create a shared execution role for Lambda functions
  const sharedRole = new iam.Role(this, 'SharedLambdaRole', {
   assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
   managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName(
     'service-role/AWSLambdaBasicExecutionRole'
    ),
   ],
  });

  // Add permissions that all functions need
  sharedRole.addToPolicy(
   new iam.PolicyStatement({
    actions: ['dynamodb:GetItem', 'dynamodb:PutItem'],
    resources: [`arn:aws:dynamodb:${this.region}:${this.account}:table/example-table`],
   })
  );

  // Create 20 Lambda functions using the shared role
  for (let i = 0; i < 20; i++) {
   new lambda.Function(this, `Function${i}`, {
    runtime: lambda.Runtime.NODEJS_20_X,
    handler: 'index.handler',
    code: lambda.Code.fromInline(
     'exports.handler = async () => ({ statusCode: 200 });'
    ),
    role: sharedRole,
   });
  }
 }
}

Note: Replace example-table with your DynamoDB table name. For the iam.PolicyStatement, include only the permissions that all functions require. For functions that require additional permissions, create a dedicated IAM role.

To update the application entry point, open the bin/cdk-resource-quota.ts file, and then replace its contents with the following code:

#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { CdkResourceLimitStack } from '../lib/cdk-resource-quota-stack';

const env = {
 account: process.env.CDK_DEFAULT_ACCOUNT,
 region: process.env.CDK_DEFAULT_REGION,
};


const app = new cdk.App();
new CdkResourceLimitStack(app, 'CdkResourceLimitStack', { env });

To deploy the stack, run the following command:

cdk deploy CdkResourceLimitStack

Consolidate log groups

By default, CDK creates a log group for each Lambda function.

To reduce the resource count, complete the following steps to create a shared log group and grant your functions write access to the log group:

To update the default stack, open lib/cdk-resource-quota-stack.ts, and then replace its contents with the following code:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as logs from 'aws-cdk-lib/aws-logs';
import { Construct } from 'constructs';

export class CdkResourceLimitStack extends cdk.Stack {
 constructor(scope: Construct, id: string, props?: cdk.StackProps) {
  super(scope, id, props);

  // Create a shared log group
  const logGroup = new logs.LogGroup(this, 'SharedLogGroup', {
   logGroupName: '/aws/lambda/shared',
   retention: logs.RetentionDays.ONE_WEEK,
   removalPolicy: cdk.RemovalPolicy.DESTROY,
  });

  // Create a shared execution role
  const sharedRole = new iam.Role(this, 'SharedLambdaRole', {
   assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
   managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName(
     'service-role/AWSLambdaBasicExecutionRole'
    ),
   ],
  });

  // Create 20 Lambda functions using the shared role
  for (let i = 0; i < 20; i++) {
   const fn = new lambda.Function(this, `Function${i}`, {
    runtime: lambda.Runtime.NODEJS_20_X,
    handler: 'index.handler',
    code: lambda.Code.fromInline(
     'exports.handler = async () => ({ statusCode: 200 });',
    ),
    role: sharedRole,
    logGroup: logGroup 

   });
   logGroup.grantWrite(fn);
  }
 }
}

Important: The preceding code uses RemovalPolicy.DESTROY to delete the log group and its data when you delete the stack. For production workloads, use RemovalPolicy.RETAIN instead to keep the logs after stack deletion. The preceding code includes a shared IAM role for all Lambda functions.

To deploy the stack, run the following command:
```
cdk deploy CdkResourceLimitStack
```
Note: When multiple functions share a log group, logs from different functions appear in the same group. Use structured logging to distinguish between functions.

Monitor resource counts with aspects

Use an aspect to count resources in your stack and notify you if your configuration is near or at the quota.

Complete the following steps:

To create an aspect, in the lib directory, create a file that's named resource-counter-aspect.ts with the following code:

import * as cdk from 'aws-cdk-lib';
import { IConstruct } from 'constructs';

export class ResourceCounterAspect implements cdk.IAspect {
 private resourceCount = 0;

 public visit(node: IConstruct): void {
  if (node instanceof cdk.CfnResource) {
   this.resourceCount++;
   if (this.resourceCount > 400) {
    cdk.Annotations.of(node).addWarning(
     `Approaching resource limit: ${this.resourceCount} resources`
    );
   }
  }
 }
}

Note: The preceding aspect checks each construct in your stack during synthesis and counts the CloudFormation resources. If your resource count is larger than 400, then the aspect sends a notification.

To apply the aspect to your stack, open bin/cdk-resource-quota.ts, and then replace the entire content with the following code:

#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { CdkResourceLimitStack } from '../lib/cdk-resource-quota-stack';
import { ResourceCounterAspect } from '../lib/resource-counter-aspect';

const app = new cdk.App();
const stack = new CdkResourceLimitStack(app, 'CdkResourceLimitStack');
cdk.Aspects.of(stack).add(new ResourceCounterAspect());

To use the aspect to check whether you're at or near your resource quota, run the following command:
```
cdk synth
```

Check your resource count

After you implement the preceding best practices, run the following command to check the number of resources in your stack:

cdk synth MyStack | grep -c 'Type:'

Note: Replace MyStack with your stack name.

To check the resource count of a deployed stack, run the following describe-stack-resource AWS CLI command:

aws cloudformation describe-stack-resources \
 --stack-name MyStack \
 --query 'length(StackResources)'

Note: Replace MyStack with your stack name.

Delete the resources that you created

To reduce future costs, complete the following steps to delete the resources that you created:

Open bin/cdk-resource-quota.ts, and then replace its entire content with the following code.

#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { NetworkStack } from '../lib/network-stack';
import { ApplicationStack } from '../lib/application-stack';
import { MainStack } from '../lib/main-stack';
import { CdkResourceLimitStack } from '../lib/cdk-resource-quota-stack';
import { ResourceCounterAspect } from '../lib/resource-counter-aspect';

const env = {
 account: process.env.CDK_DEFAULT_ACCOUNT,
 region: process.env.CDK_DEFAULT_REGION,
};

const app = new cdk.App();

// Split stacks strategy
const networkStack = new NetworkStack(app, 'NetworkStack', { env });
const appStack = new ApplicationStack(app, 'ApplicationStack', networkStack.vpc, { env });
appStack.addDependency(networkStack);

// Nested stacks strategy
new MainStack(app, 'MainStack', { env });

// Shared IAM roles + consolidated log groups strategy
const cdkStack = new CdkResourceLimitStack(app, 'CdkResourceLimitStack', { env });

// Resource counter Aspect
cdk.Aspects.of(cdkStack).add(new ResourceCounterAspect());

Note: The preceding configuration makes sure that the entry point file references all the stacks.

To remove only the stacks that you defined in your application, run the following cdk destroy command:
```
cdk destroy --all
```
When prompted, confirm that you want to delete the stacks and their associated resources from your AWS account.

Related information

Best practices for developing and deploying cloud infrastructure with the AWS CDK

class NestedStack (construct)

Topics: AWS Cloud Development Kit (CDK)
Tags: AWS Cloud Development Kit (CDK)
Language: English

AWS OFFICIALUpdated a month ago

No comments

Relevant content

Reduce resources creation CloduFormation
rePost-User-2951800
asked 3 years ago
CloudFormation Resource Limit Issue Despite Using Nested Stacks
matt
asked a year ago
cdk-deploy --all keep saying no change but I have upted my source code...
RECJ
asked 2 years ago
How do I go about moving a CDK application to a new home in source control?
bimargulies
asked 9 months ago
[CDK] How do I update a stack resources in CDK without duplicating them?
rePost-User-0476594
asked 3 years ago
How do I customize my AWS CDK bootstrap and deploy the CFNToolkit CloudFormation stack?
AWS OFFICIALUpdated a month ago
How do I resolve the error "Failed to receive X resource signal(s) within the specified duration" for EC2 Windows instances in AWS CloudFormation?
AWS OFFICIALUpdated 2 years ago
How do I customize a resource property value when there is a gap between CDK higher level constructs and a CloudFormation resource?
AWS OFFICIALUpdated 3 years ago
How do I resolve the error "Failed to receive X resource signal(s) within the specified duration" in AWS CloudFormation?
AWS OFFICIALUpdated a year ago
How do I resolve the “ResourceLimitExceeded” error that I am seeing when trying to create resource AWS::SageMaker::NotebookInstance using CloudFormation?
EXPERT
Arjun
published a year ago