Optimizing costs with the CloudWatch Logs Infrequent Access log class for Amazon EKS control plane logging

Introduction

As a proactive effort to help customers maximize their Amazon EKS investment, AWS introduced an exciting new feature for CloudWatch Logs, the IA log class. The IA log class is a new ingestion log class that cost-effectively manages all of your Amazon EKS control plane logging. With this class, you can host your logs natively on AWS for ad-hoc querying and after-the-fact forensic analysis. 

AWS Enterprise Support is always looking for ways to help customers save money and get the most value out of their AWS investments. Your designated Technical Account Manager (TAM) plays a crucial role in this effort, and works with you to maximize the return on your AWS investment.

Solution overview

The CloudWatch Logs IA log class offers a subset of CloudWatch Logs capabilities at a lower per GB ingestion price. This option makes it an ideal solution for log data that doesn't require real-time analysis. The IA log class includes the logs that the Kubernetes control plane generates. You can then use these logs to audit and diagnose instead of live troubleshooting. For the complete list of features and comparisons with the Standard log class, see Log classes.

Amazon EKS control plane logging

Amazon EKS integrates with CloudWatch Logs for the Kubernetes control plane. Amazon EKS provides the control plane as a managed service, and you can turn on logging without installing a CloudWatch agent. The Kubernetes control plane is a set of components that manage Kubernetes clusters and produces logs used to audit and diagnose issues. You can also deploy the CloudWatch agent to capture Amazon EKS node and container logs. To send your container logs to CloudWatch Logs, you can also use Fluent Bit. Kubernetes logging includes the following logging types:

• Control plane logging

• Node logging

• Application logging

When you turn on these logging types, CloudWatch creates a log group with multiple log streams in the Standard log class.

After you create the log group, you can't change the log class.  However, you can update the logging type for the Amazon EKS control plane to log to the IA log group. 

Solution implementation

By default, when you turn on control plane logs for your Amazon EKS clusters, AWS automatically creates CloudWatch log groups with the Standard log class. Although this provides immediate access to your logs, it might not be the most cost-effective solution for all use cases. 

To use the IA log class for your Amazon  EKS control plane logs, first create the CloudWatch log groups. If you create the log groups before the IA log class, then CloudWatch Logs stores your logs in Logs IA. Then, you can use this log configuration for after-the-fact forensic analysis.

Implementation steps

To configure Logs IA, complete the following steps:

1. If the Amazon EKS cluster has control plane logging turned on, then turn off logging.
2. Delete the log group in CloudWatch with the name /aws/eks/<Your EKS Cluster name>/cluster. Note: To back up your log group information, export the data to Amazon Simple Storage Service (Amazon S3). For more information, see Export log data to Amazon S3 using the console.
3. Create new log groups in CloudWatch, and then specify the IA log class.
4. Turn on control plane logging for your Amazon EKS cluster. Amazon EKS creates the log streams in the CloudWatch IA log group, and the messages are logged.

Through these steps, you can store your Amazon EKS control plane logs in a more cost-effective IA class from the beginning. Small optimizations such as this one can add up to significant savings over time, especially for larger deployments or multiple clusters.

Cost savings

The cost for data ingestion into a CloudWatch Logs IA class is currently up to 50% lower per GB than the Standard log class.

For example, an organization might have an Amazon EKS cluster with 500 GB of control plane logs per month. With the Standard log class, the monthly cost for ingestion is approximately $250 (500 GB x $0.5 per GB). If the organization switches to the IA log class, then the monthly cost decreases to $125 (500 GB x $0.25 per GB). This results in 50% cost savings. For more details on pricing, see the Amazon CloudWatch pricing page.

Commitment to cost optimization

This new feature for CloudWatch Logs IA is a testament to the ongoing commitment by AWS to help customers optimize their investments and get the most value out of their AWS services. The Enterprise Support team constantly explores ways to identify cost-saving opportunities and share them with customers.

The IA log class for Amazon EKS control plane logging is just one example of how AWS helps customers reduce their cloud spending. This example also helps customers consolidate all logs in one place in a cost-effective manner. 

Conclusion

The CloudWatch Logs IA class presents a valuable cost saving opportunity for Amazon EKS control plane CloudWatch Logs, particularly for after-the-fact forensic analysis. The configuration process is straightforward through the Amazon EKS console, so you can switch between the IA and Standard log classes as needed. When properly aligned with your operational needs, the CloudWatch Logs IA class can provide significant cost benefits without compromising essential logging capabilities.

With CloudWatch Logs IA class for Amazon EKS control plane logging, you can optimize your costs and maintain valuable data for ad-hoc querying and forensic analysis. This feature from AWS demonstrates the company's dedication to helping customers maximize their investments and get the most out of their AWS services. Remember to plan and create the necessary CloudWatch log groups with the IA class before you turn on control plane logging. With some upfront preparation, you can take advantage of cost savings immediately.

AWS Support engineers and TAMs can provide general guidance, best practices, troubleshooting, and operational support on AWS. To learn more about our plans and offerings, see AWS Support.

About the authors

Sudheer Sangunni

Sudheer Sangunni is a Senior TAM with Enterprise Support. With his extensive expertise in the AWS Cloud, Big Data, Sudheer plays a pivotal role in helping customers enhance their monitoring and observability capabilities within the AWS environment.

Ganesh Sambandan

Ganesh Sambandan is a Senior TAM for strategic customers at AWS, where he helps organizations implement best practices for running workloads on AWS. He's also a Technical Field Community member in the Cloud Operations domain at AWS. Based in Phoenix, Arizona, you can connect with him on LinkedIn.

Alma Mohapatra

Alma Mohapatra is an Enterprise Support Manager supporting strategic AI/ML customers running on high performance computing (HPC) environments. She leads specialized technical teams that help organizations optimize their large-scale ML workloads across distributed GPU clusters. With extensive experience in enterprise ML operations, Alma guides customers through complex performance challenges and infrastructure optimization for training LLMs. She excels at translating technical requirements into practical solutions that enhance reliability and efficiency in production AI environments. Alma is known for her collaborative approach to customer success, and works closely with TAMs to make sure that AI/ML initiatives meet critical business objectives.