How do I troubleshoot issues with the Amazon VPC CNI add-on for Amazon EKS?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

If the worker node isn't ready because the Amazon VPC CNI add-on didn't initialize, then you receive an error message that's similar to the following example:

"container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

To check for this issue, run the following command:

kubectl describe node node_name 

Note: Replace node_name with the name of the node. You can find the cni plugin no initialized error message in the output.

Take the following actions to troubleshoot the issue.

Confirm that you installed the Amazon VPC CNI add-on

If the aws-node Pods in the cluster are missing, then you receive an error message that's similar to the following example in the containerd logs:

"cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"

To resolve this issue, install the Amazon VCP CNI add-on.

Check for IAM permissions issues

To use the Amazon VPC CNI add-on, use the AmazonEKS_CNI_Policy AWS Identity and Access Management (IAM) managed policy. Or, if you use a custom policy, then make sure that it includes the following permissions:

 "ec2:AssignPrivateIpAddresses", 
 "ec2:AttachNetworkInterface", 
 "ec2:CreateNetworkInterface", 
 "ec2:DeleteNetworkInterface",
 "ec2:DescribeInstances", 
 "ec2:DescribeTags", 
 "ec2:DescribeNetworkInterfaces", 
 "ec2:DescribeInstanceTypes", 
 "ec2:DescribeSubnets", 
 "ec2:DetachNetworkInterface", 
 "ec2:ModifyNetworkInterfaceAttribute", 
 "ec2:UnassignPrivateIpAddresses"
 "ec2:CreateTags"

To identify missing IAM permissions, check the L-IPAM daemon (IPAMD) logs in the /var/log/aws-routed-eni/ipamd.log host directory. If the Amazon VPC CNI add-on doesn't have a required IAM permission, then you receive an error message that's similar to the following example:

{"level":"error","ts":"2023-11-18T01:08:34.083Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: ipamd init: failed to retrieve attached ENIs info: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::XXXXXXXXXXXXXXXXXXXX:assumed-role/rolename is not authorized to perform: ec2:DescribeNetworkInterfaces because no identity-based policy allows the ec2:DescribeNetworkInterfaces action\n\tstatus code: 403, request id: request id"}

The preceding example error message shows that the add-on requires the ec2:DescribeNetworkInterfaces permission.

If you don't have direct access to the worker node, then check the aws-node Pods on the worker node. To identify the aws-node Pod on the worker node, run the following command:

kubectl describe node node_name

Note: Replace node_name with the name of the node. Check the Non-terminated Pods section of the output for the aws-node Pod name.

Then, run the following command to view details about the aws-node Pod:

kubectl describe pod pod-name -n kube-system

Note: Replace pod-name with the aws-node Pod name.

In the output, check Event for information about missing permissions.

Example output:

Type Reason Age From Message
---- ------ ---- ---- -------
Warning MissingIAMPermissions 105s (x2 over 105s) aws-node Unauthorized operation: failed to call ec2:DescribeNetworkInterfaces due to missing permissions. Please refer https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/iam-policy.md to attach relevant policy to IAM role in the event section you can able to see the error message also you can check the AWS cloud trail event for the particular instance by filtering the cloudtrail with username and provide the instanceID there. 

Or, use AWS CloudTrail to check for specific events. For example, check the username and instance ID for the DescribeNetworkInterfaces API call to determine whether it ran.

Check for network issues

The Amazon VPC CNI add-on must reach the API server endpoint and the Amazon Elastic Compute Cloud (Amazon EC2) endpoint when it runs. If either connection fails, then the add-on can't initialize and the nodes move into the NotReady status.

To check the specific error message, review the IPAMD logs in the /var/log/aws-routed-eni/ipamd.log host directory. Make sure that the kube-proxy and coreDNS Pods can run without errors.

Check the add-on component versions

It's a best practice to update the core components to the latest version. If the add-on version doesn't match your Kubernetes cluster version, then the add-on can't initialize. Make sure that the add-on version is compatible with the Kubernetes cluster version.

Troubleshoot Amazon VPC CNI add-on errors

To identify add-on errors, use SSH to connect to the worker node where the add-on can't assign the IP address. Then, review the IPAMD logs in the /var/log/aws-routed-eni/ipamd.log file for error messages.

Failed to assign ip error

The Failed to assign ip error occurs when the Amazon VPC CNI add-on can't assign IP addresses to Pods that are scheduled to the worker nodes. To identify the error, you can also check the CloudTrail Event history page for the AssignPrivateIpAddresses event.

To determine whether the assigned subnet has no available IP addresses, run the following describe-subnets AWS CLI command:

aws ec2 describe-subnets --filters "Name=vpc-id,Values= VPCID" | jq '.Subnets[] | .SubnetId + "=" + "\(.AvailableIpAddressCount)"'

Note: Replace VPCID with your VPC ID.

The Failed to assign ip error can also occur when you set the values of the WARM_ENI_TARGET, WARM_IP_TARGET, or MINIMUM_IP_TARGET environment variables too low. Make sure that your subnet CIDR range is broad enough and has enough IP address allocations for your use case. For more information, see WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET on the GitHub website.

To check the values of the environment variables, run the following command to view your Pod details:

kubectl describe pod pod-name -n kube-system

Note: Replace pod-name with the aws-node Pod name.

You can update the environment variables in the aws-node DaemonSet:

env:
  - name: WARM_ENI_TARGET
    value: "1"
  - name: WARM_IP_TARGET
    value: "5"
  - name: MINIMUM_IP_TARGET
    value: "25"

Or, run the following command to update the environment variables:

kubectl set env ds aws-node -n kube-system WARM_ENI_TARGET=1 WARM_IP_TARGET=5 MINIMUM_IP_TARGET=25

If you use prefix delegation mode and don't use a dedicated Pod subnet range, then subnet fragmentation might cause the add-on to fail. The add-on must assign continuous /28 for IPv4 and /80 for IPv6 prefix range to the network interface. For more information about prefix delegation, see ENABLE_PREFIX_DELEGATION on the GitHub website.

You can't directly check for subnet fragmentation. Instead, check whether you still have more than 16 available IP addresses in your subnet CIDR range. If you have enough IP addresses and still frequently receive the Failed to assign ip error, then the issue is typically subnet fragmentation. To troubleshoot this issue, it's a best practice to use custom networking with prefix delegation to establish dedicated subnets for the Pods. For more information, see AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG on the GitHub website.

Failed to get pod ENI config error

If you use AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG to activate custom networking but don't create an ENIConfig resource, then you receive an error message that's similar to the following example:

"{"level":"error","ts":"","caller":"ipamd/ipamd.go:798","msg":"Failed to get pod ENI config"}"

To check whether the ENIConfig resource exists in the cluster, run the following command:

kubectl get eniconfig -o yaml

If the ENIConfig custom resource doesn't exist, then create one. For more information about custom networking, see AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG on the GitHub website.

Failed to watch error

If you don't write the YAML file in the correct format, then you might receive an error message that's similar to the following example:

"Failed to watch *v1alpha1.ENIConfig: failed to list *v1alpha1.ENIConfig:json: cannot unmarshal string into Go struct field ENIConfigSpec.items.spec.securityGroups of type []string"

This issue typically occurs when there's a mismatch between the JSON data and the Go data type definition. The preceding example error message shows that there's an issue with the securityGroups field.

To troubleshoot this issue, run the following command to check your JSON data:

kubectl describe ENIConfig

In the command's output, check the syntax based on the issue that's identified in the error message. For example, make sure that the securityGroups field in your JSON is an array of strings, not a single string.

Example output:

cat >$az_1.yaml <<EOF
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata: 
  name: us-west-2a
spec: 
  securityGroups: 
    - sg-0123456789abcdef0
  subnet: subnet-0123456789abcdef0
EOF