AWS re:Invent 2024 - Securing Kubernetes workloads in Amazon EKS

If security concerns top your list, you're not alone. With the explosive growth of Kubernetes adoption—84% of organizations now using or evaluating it according to the 2023 CNCF survey—securing these complex environments has become increasingly critical.

At AWS re:Invent 2024, Micah Hausler, Principal Software Engineer, and George John, Senior Product Manager at AWS, tackled this challenge head-on by breaking down Kubernetes security into manageable layers. Let's explore how you can strengthen Amazon EKS security posture using their insights and the latest innovations.

Understanding the Security Landscape

Before diving into specific security controls, it's important to understand the shared responsibility model for Amazon EKS.

"For AWS and Amazon EKS, security is the highest priority," George emphasized. "When we launched Amazon EKS over six years ago, it was really in response to customer feedback that managing Kubernetes at scale is hard."

In the Amazon EKS shared responsibility model, AWS manages the security of the Kubernetes control plane, including the API servers, etcd, and scheduler. As a customer, you're responsible for securing worker nodes, configuring network policies, managing access controls, and securing the applications deployed to clusters. The following sections will explore how to address these responsibilities across three key layers: cluster configuration, infrastructure management, and application security

Cluster-Level Security Controls

Enhanced Access Management

When Amazon EKS first launched, it used the aws-auth ConfigMap to map IAM identities to Kubernetes permissions. This approach allowed you to use IAM without maintaining a separate identity provider, giving you benefits like multi-factor authentication and Amazon CloudTrail auditing. However, it required switching between different APIs and could accidentally lock users out with simple configuration errors.

To address these challenges, AWS introduced Cluster Access Management shortly after re:Invent 2023. This feature unifies APIs for simpler management, simplifies configuration with less chance for errors, and provides granular control over permissions. It even allows you to revoke the super-user permissions automatically granted to cluster creators in the older model.

"For Amazon EKS, Cluster Access Management is the future," George stated. "All the new features, all the capabilities will be built on Cluster Access Management. If you have an existing cluster that's using the older aws-auth ConfigMap, we strongly encourage you to move."

Pod Identity for AWS Resource Access

For applications running in the cluster that need to access AWS resources like S3 buckets or DynamoDB tables, Amazon EKS offers two complementary solutions.

• IAM Roles for Service Account (IRSA): IRSA works across all Kubernetes deployment models, including self-managed clusters, but requires creating an OIDC provider and updating trust policies when adding new clusters.
• Pod Identity: Pod Identity simplifies trust management by allowing a role to be assumed by the Amazon EKS service as a whole, rather than specific clusters. It supports attribute-based access control through IAM role session tags and enables reuse of policies across clusters.

"IRSA is not the older way," George clarified. "We will support both. We have no plans to deprecate IRSA. IRSA would continue to exist along with Pod Identity."

Since its launch, Pod Identity has expanded to all commercial AWS regions, China, and US GovCloud regions. AWS has also open-sourced the Pod Identity agent for more flexible deployment options and integrated it with Amazon EKS add-ons.

Monitoring and Detection

Amazon EKS integrates with several AWS services for effective security monitoring. Control plane audit logs in Amazon CloudWatch capture all API server requests, while Amazon CloudTrail records Amazon EKS API calls so you can track who's creating or modifying clusters. For network traffic analysis, Amazon Detective helps visualize and analyze VPC flow logs.

On the encryption front, Amazon EKS supports encryption for storage services like Amazon EBS, EFS, and FSx for Lustre. You can use AWS managed keys or bring your own keys. Kubernetes Secrets can also be encrypted using AWS KMS with automatic key rotation.

Infrastructure-Level Security Controls

Amazon EKS Auto Mode

One of the most exciting announcements at re:Invent 2024 was Amazon EKS Auto Mode. This feature, launched just before the session, fundamentally changes how you manage Kubernetes infrastructure.

"With Auto Mode, AWS takes responsibility for creating and deploying nodes. You only have to create pods," Micah Hausler explained. The service automatically selects optimal Amazon EC2 instance types for the workloads and creates the best nodes for specific requirements.

Amazon EKS Auto Mode continually keeps the environment secure by rotating nodes every 21 days maximum, maintaining the latest security patches. This approach significantly reduces operational overhead and shifts more of the shared responsibility to AWS.

"This really helps increase agility, and accelerate innovation because you're not having to spend time trying to keep all nodes up to date," Micah emphasized. "You really get to offload all of these operations to AWS."

Network Security Options

When it comes to securing pod networking in Amazon EKS, you have two powerful approaches available. The first is Kubernetes Network Policy, which uses the Kubernetes API to define rules about which pods can communicate with each other. These policies are enforced on the host using eBPF in the Amazon EKS VPC CNI plugin.

The second option is Security Groups for Pods. This feature applies the familiar EC2 security groups directly to Kubernetes pods through elastic network interfaces. It provides an off-host enforcement mechanism that can't be modified on the host, creating an additional layer of protection.

GuardDuty for Amazon EKS

Amazon GuardDuty offers essential security monitoring for Amazon EKS clusters through two key integrations. The first is Audit Log Protection, which analyzes Kubernetes audit logs to detect misconfigurations and known attack patterns. GuardDuty uses machine learning and rules based on observed attack patterns to identify security issues.

"We interface with the GuardDuty team a lot to say, here's new things that are coming in Kubernetes, here's new things we should look for in configuration, either misconfiguration, or attack patterns," Micah explained.

The second integration is Runtime Protection, which deploys an agent on every host to monitor for malicious activity that might not appear in audit logs. This agent-based approach can detect file system changes and other suspicious behaviors that indicate a potential security incident.

Micah recommended enabling GuardDuty at the AWS account or organization level to protect all clusters consistently. Beyond GuardDuty, he suggested using AWS Systems Manager instead of SSH for secure host access, deploying container-optimized operating systems like Bottlerocket, and verifying configurations against the CIS benchmark for Amazon EKS.

Application-Level Security Controls

Pod Security

For securing application pods, Micah recommended using Kubernetes Pod Security Standards or policy-as-code solutions like Kyverno or Open Policy Agent. He emphasized several key practices, starting with limiting (not eliminating) privileged container usage.

"When we say limit, we don't mean eliminate privileged containers," Micah clarified. He explained that some system components like the AWS VPC CNI or storage drivers legitimately need privileges, but application pods typically don't.

Another recommendation was disabling service account token mounts for pods that don't need API access. "By default, Kubernetes mounts a token into every pod so that it can talk to the API server and authenticate," Micah explained. "If you don't need it, turn it off."

He also advised restricting host path mounts and using read-only filesystems where possible. "If the pod, say again, we'll take the web traffic example, if the pod never has to write, or the application in the pod never has to write to the file system, consider switching that to a read-only file system in your container," he suggested. These measures eliminate whole categories of potential attack vectors.

Container Image Security

To secure container images, Micah strongly recommended regular vulnerability scanning. "With Amazon ECR, you can use the native Amazon Inspector integration and get scans of the and reports if there are security vulnerabilities found in the application," he explained.

For enhanced security, consider using Amazon ECR with AWS PrivateLink if you don't need internet access for pulling images. This keeps all image traffic within the VPC. Additionally, configure container images to use non-root users whenever possible, further reducing the permissions available to processes running in containers.

Cedar: Advancing Kubernetes Authorization

Cedar, an access control language and runtime evaluation engine launched by AWS in 2023, addresses fundamental limitations in Kubernetes Role-Based Access Control (RBAC) when applied to container orchestration.

"If you've ever written an RBAC policy, you've probably come into some of the limitations of this," Micah noted. "It's worked great in Kubernetes for the last, like eight years or so. But the more and more that we've heard from customers, we're seeing some of the limitations of RBAC come into play."

These limitations include the inability to create denial policies and the lack of conditional access control. Cedar solves these problems by supporting both permit and forbid policies, conditional access, and attribute-based permissions.

In a live demonstration, Micah showed how Cedar policies could permit specific users to manage ConfigMaps, require label selectors when listing resources, and enforce ownership-based controls for resource modification. When a request was denied, the system even indicated which specific policy blocked it, greatly improving troubleshooting.

Resources for Continued Learning

To help continue securing Amazon EKS environments, the speakers shared several valuable resources. The Amazon EKS Best Practices Guide provides comprehensive security recommendations. For those new to Amazon EKS, the Amazon EKS Workshop offers a guided tour of creating and configuring clusters.

Amazon EKS Blueprints provide sample configurations using Terraform and AWS CDK for getting up and running quickly. Finally, the AWS Containers Roadmap on GitHub lets you track upcoming features, provide feedback, and upvote capabilities you'd like to see.

"Our product managers use this all the time to gauge, okay, what's the most popular issue?" Micah noted about the roadmap. "You can sort by plus ones to see what's the most requested feature for Amazon EKS."

Conclusion

Securing Kubernetes workloads in Amazon EKS requires addressing security at multiple layers—cluster, infrastructure, and application. The tools and best practices shared by Hausler and John provide a comprehensive framework for enhancing your security posture.

AWS continues to innovate with features like Cluster Access Management, Pod Identity, Amazon EKS Auto Mode, and the experimental Cedar authorization system. These advancements simplify security management while providing more granular control over your Kubernetes environments.

For those interested in diving deeper, the full session recording is available on the AWS YouTube channel , featuring Micah Hausler and George John's detailed explanations and demonstrations.