Best practices for identifying and eliminating hidden cross-zone data transfer charges in highly dynamic EKS clusters?

To identify and eliminate hidden cross-zone data transfer charges in dynamic EKS clusters, you should implement a multi-layered approach combining visibility tools, architectural optimizations, and traffic management strategies.

Visibility and Monitoring

For pod-level visibility into network flows, enable Container Network Observability in Amazon EKS, which is powered by the Network Flow Monitor Agent. This provides granular insights into inter-AZ traffic patterns at the pod level. You can access the Observability Dashboard through the Amazon EKS console and use the Cluster view in the Flow table to identify inter-AZ traffic by filtering by Local AZ and Remote AZ, then sorting by data transferred. This gives you precise visibility into which workloads are generating cross-zone traffic.

You can also query Network Flow Monitor data using AWS CLI commands for programmatic analysis and automation of cost findings.

Architectural Patterns to Reduce Cross-AZ Traffic

Several strategies can help minimize inter-AZ communication:

1. Topology-Aware Routing with Traffic Distribution: Implement Kubernetes Traffic Distribution Control, which routes traffic to healthy endpoints within the target topology (same zone, same node). This prioritizes predictability and cost reduction.

2. Pod Topology Spread Constraints: To address your concern about uneven load distribution, use pod topology spread constraints to distribute pods evenly across zones or nodes. This prevents hotspots and supports balanced resource utilization. However, be aware that combining topology spread constraints with Horizontal Pod Autoscaler (HPA) can lead to uneven scaling, so plan your scaling policies accordingly.

3. Zone-Specific Deployments: For scenarios with varying traffic patterns per zone, create separate Deployments per zone. This enables independent scaling per zone, though it adds management overhead.

4. Service Mesh Implementation: Consider using a service mesh like Istio to restrict traffic to availability zones and nodes. You can use Destination Rules and the Kubernetes Service internal traffic policy to control traffic flow and reduce network-related costs.

Load Balancer Optimization

Use IP traffic mode for your load balancers to minimize data transfer costs. This allows the load balancer to proxy traffic directly to the destination pod, eliminating additional network hops and reducing cross-AZ traffic.

VPC and Network Architecture

Implement VPC endpoints for AWS services like Amazon S3 and DynamoDB to keep traffic within the AWS network and reduce internet gateway or NAT device usage. For communication between VPCs, use VPC peering or Transit Gateways for cost-effective connectivity.

Balancing Cost and High Availability

When implementing these optimizations, you have several options depending on your availability requirements:

• Leave inter-AZ traffic as is if high availability across zones is critical
• Move workloads to a single AZ if the application can tolerate zone-level failures
• Use traffic distribution controls to prefer same-zone routing while maintaining cross-zone failover capability

The key is to use Container Network Observability to identify your specific traffic patterns, then apply the appropriate combination of topology-aware routing, pod distribution strategies, and architectural optimizations that align with your availability requirements and cost objectives.
Sources
Track inter-AZ and NAT gateway traffic with EKS Container Network Observability | Containers
Cost Optimization - Networking - Amazon EKS
Optimize data transfers across AWS services | AWS re:Post

1. Size it on the bill before re-architecting

In the Cost and Usage Report, traffic between Availability Zones in one Region has the usage type {Region}-DataTransfer-Regional-Bytes, for example USE2-DataTransfer-Regional-Bytes. EC2 and network interface traffic across AZs is charged at $0.01/GB in each direction, so every GB that crosses between two of your nodes costs $0.02, and it shows up as two line items. At that rate 1,000 GB a month is $20. It's worth knowing whether you're chasing $20 or $20,000 a month before accepting any availability trade-off.

2. Attribute it to pods with VPC Flow Logs

The Amazon VPC CNI assigns each pod a private IP from your VPC, which is what makes this possible.

• Use a custom log format with pkt-srcaddr, pkt-dstaddr, bytes, flow-direction and az-id.
• Use the pkt- fields, not srcaddr/dstaddr. The flow log docs call out this exact case: the IP address of a pod in Amazon EKS "is different from the IP address of the network interface of the instance node on which the pod is running". The plain fields will attribute traffic to nodes, not pods.
• az-id is the AZ of the interface that recorded the flow, not the peer's. To tell whether a flow crossed zones, map pkt-dstaddr to its subnet. Each subnet sits entirely within one AZ, so the subnet gives you the destination AZ to compare against.
• A VPC-level flow log records every interface, so traffic between two pods is logged at both ends. Filter on flow-direction = 'egress' to count each flow once, then multiply the bytes by the $0.02/GB round trip.
• Join the pod IPs to your pod inventory to get namespace and workload. In a cluster that churns, capture that inventory over the same time window as the logs, because an IP can belong to a different pod an hour later.

To keep it cheap, publish to S3 in Parquet with Hive-compatible prefixes and hourly partitions, and query with Athena. AWS states Parquet queries are 10 to 100 times faster than plain text. Delivery to S3 still carries vended logs charges, so scope the flow log to the subnets you're investigating and turn it off when you're done.

3. On the uneven-load concern specifically

The EKS best practices guide is explicit about the trade-off between the two same-zone options:

• Topology Aware Routing "may skip hint assignments when node resources in a zone are too imbalanced to avoid excessive overload". It protects itself, at the cost of predictability.
• Traffic Distribution with trafficDistribution: PreferClose is predictable, but "endpoints within a single AZ can become overloaded if most traffic originates from that same zone". The guide's mitigation is a separate Deployment per zone, each with its own HPA, plus topology spread constraints.

So if uneven load is the risk you can't take, Topology Aware Routing's habit of standing down is the safer start. If you need predictable same-zone routing, plan for per-zone Deployments from the outset.

Sources:

• CUR usage types: https://docs.aws.amazon.com/cur/latest/userguide/cur-data-transfers-charges.html
• Flow log fields: https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html
• EKS networking cost guide: https://docs.aws.amazon.com/eks/latest/best-practices/cost-opt-networking.html

Disclosure: I work on Akal Cloud, an AWS Cost Optimization Platform. I've written up the billing side of cross-AZ transfer in more detail, including which traffic is free: https://akalcloud.ai/blog/cross-az-data-transfer-cost/

Just use Application Load BalanceR (ALB) and keep traffic in the same VPC - Cross Zone is free to/from private IPs in the same VPC to/from ALB.