How do I choose the right cluster size and scaling strategy for my Amazon Redshift workload?

Short description

Amazon Redshift offers provisioned clusters and Amazon Redshift Serverless. If you use provisioned clusters, then you must select node types and cluster sizes. However, if you use Amazon Redshift Serverless, Amazon Redshift automatically scales the compute capacity.

Resolution

Choose the right node type

It's a best practice to use RA3 nodes for all new and migrated workloads. To scale compute and storage independently, RA3 nodes use high-performance SSDs for hot data and Amazon Simple Storage Service (Amazon S3) for cold data to decouple compute from storage. This managed storage approach automatically tiers cold data to Amazon S3. This reduces storage costs and maintains query performance.

Review the following information about RA3 node types:

• Use ra3.large for small workloads, development, and tests.
• Use ra3.xlplus for medium workloads.
• Use ra3.4xlarge for large online analytic processing (OLAP) and extract, transform, and load (ETL) workloads.
• Use ra3.16xlarge for high-throughput and enterprise scale workloads.

To choose a node type, consider the following sizing factors:

• Evaluate your current data size and anticipated growth over the next 12 to 24 months.
• Consider the number of concurrent users, query types, and query complexity.
• Define acceptable query response times for different workload types and your SLA targets.
• Analyze your workload patterns, such as peak usage times, batch processing windows, and seasonal variations.

Note: It's a best practice to maintain at least around 30 to 40% free disk space to accommodate table growth, maintenance operations such as VACUUM and ANALYZE, and new data ingestion. This buffer prevents performance degradation and allows for smooth cluster operations. For more information, see Vacuuming and analyzing tables automatically and Vacuuming and analyzing tables manually.

To compare your current production workload against different cluster configurations in a test environment, you can use the open source Redshift Test Drive tool. You can then validate performance, compare node types and cluster sizes, and identify the most cost-effective configuration. For more information, see redshift-test-drive on the GitHub website.

You can also conduct a proof of concept (POC) for Amazon Redshift.

Choose the appropriate resize method

Review and choose one of the resize methods for Amazon Redshift.

Elastic resize (recommended)

Elastic resize adds or removes nodes from an existing cluster and completes in minutes with minimal disruption. 

Use elastic resize for one of the following reasons:

• You're scaling within the same node type or different node type.
• Your configuration change is supported by elastic resize.
• You need minimal downtime.

To perform an elastic resize, complete the following steps:

1. Open the Amazon Redshift console.
2. In the navigation pane, choose Clusters.
3. Select your cluster.
4. Choose Actions, and then choose Resize.
5. Choose Elastic resize, and then specify the new number of nodes.
6. Choose Resize cluster to start the operation.

Classic resize with enhancements

Classic resize can take significantly longer for large datasets. For multi-node RA3 target clusters, clusters are available for reads and writes in minutes instead of hours.

Classic resize, when it targets RA3 nodes with more than 2 nodes automatically applies performance enhancements that reduce downtime.

Use classic resize when the following conditions are true:

• You want to change node types and can tolerate brief cluster unavailability during the transition. For example, you want to migrate from DC2 nodes to RA3 nodes.
• You want to switch to or from a single-node cluster.
• You need to increase the slice count.
• Elastic resize doesn't support your configuration change.

Note: For multi-node RA3 targets, classic resize with enhancements makes the cluster available for reads and writes within minutes while data redistribution continues in the background. For DC2 target, standard classic resize behavior applies.

Snapshot, restore, and resize

Use the snapshot, restore, and resize method to validate the new configuration in pre-production and minimize production downtime by preparing the new cluster in advance.

You can use the snapshot, restore, and resize method if you want to do the following**:**

• Minimize production downtime.
• Test the new configuration before cutover.
• Perform a major node type change and require near zero production downtime. 

Note: You must manually sync any writes that occur on the source cluster during the snapshot and restore window to the target cluster before cutover to avoid data loss.

For more information, see Resize Amazon Redshift from DC2 to RA3 with minimal or no downtime.

Configure concurrency scaling

Use concurrency scaling when your workload has unpredictable spikes in query volume. You can also use concurrency scale to maintain consistent query performance during peak periods or to avoid over-provisioning of your base cluster for occasional peaks.

To configure concurrency scaling, complete the following steps:

1. Open the Amazon Redshift console.
2. In the navigation pane, choose Clusters.
3. Select your cluster.
4. Choose the Workload management tab.
5. Edit your workload management configuration, and then set Concurrency Scaling mode to auto for the queues that you want to use concurrency scaling for.
6. Use the max_concurrency_scaling_clusters database parameter to define the maximum number of scaling clusters. The default value is 1.
7. Assign concurrency scaling to specific queues or user groups based on your workload requirements.

Important: Concurrency scaling provides 1 hour of free credit per day for each RA3 cluster. Amazon Redshift charges additional usage by the second. For pricing details, see Amazon Redshift pricing.

Improve performance and cost

To improve the performance and cost, take the following actions: 

• Use Amazon Redshift Serverless for intermittent workloads to pay only for the compute time consumed. For more information, see Amazon Redshift Serverless.
• Use Amazon Redshift RG instances powered by AWS Graviton for faster performance for data warehouse workloads at 30% lower price per vCPU compared to RA3 instances.
• Use Reserved Instances (RIs) to reduce compute costs compared to On-Demand pricing. For current pricing and discount details, see Purchasing Amazon Redshift reserved nodes and Amazon Redshift pricing.
• Configure workload management (WLM) queues to prioritize critical queries and prevent resource-intensive queries from affecting overall performance.
• Turn on automatic table optimization to reduce manual tuning effort and improve query performance.
• Use the Amazon Redshift console or Amazon CloudWatch to regularly monitor query performance, disk usage, and CPU utilization. Use these insights to right-size your cluster, adjust configurations, and scale up based on actual usage patterns and workload demands. 

Related information

Cluster operations

Amazon Redshift best practices for designing queries