Why can't I run kubectl commands in Amazon EKS?

Short description

When you try to run kubectl commands, you might encounter the following issues:

• You can't run kubectl exec, kubectl logs, kubectl attach, or kubectl port-forward.
• kubectl can't reach the control plane.
• You can't install kubectl.
• You encounter authorization errors.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

You can't run kubectl exec, kubectl logs, kubectl attach, or kubectl port-forward

To run kubectl exec, kubectl logs, kubectl attach, or kubectl port-forward commands, the Amazon EKS API must establish a trusted connection with the kubelet. If authentication fails, then you receive an error similar to the following example:

"Error from server: error dialing backend: remote error: tls: internal error"

Check for network connectivity issues

Make sure that the API server can communicate with the work node on port 1025. The worker node security group must allow inbound traffic from the cluster security group on port 10250. The cluster security group must allow inbound traffic from the worker node security group on port 443. Make sure that your security groups adhere to the Amazon EKS requirements.

Check for CSR issues

If the control panel doesn't approve the certificate signing request (CSR) that the kubelet submitted, then the kubelet can't run.

To identify the CSR with issues, run the following command:

kubectl get certificatesigningrequest

To check the status of the CSR request, run the following command:

kubectl describe certificatesigningrequest csr-name -n namespace

Note: Replace csr-name with the CSR name, and namespace with the namespace name.

In the output, check for CSRs in the Pending or Approved state instead of the Approved, Issued state.

You must activate the RotateKubeletServerCertificate and ServerTLSBootstrap flags in the kubelet so that the kubelet submits and rotates the serving certificate for itself. To check if flags in the kubelet-config JSON file are set to true, run the following command from within the worker node:

cat /etc/kubernetes/kubelet/kubelet-config.json

Example output:

"serverTLSBootstrap": true
"featureGates": {
    "RotateKubeletServerCertificate": true
    }

For the kubelet to create and submit the CSR, you must attach the eks:node-bootstrapper role to the system:bootstrappers and system:nodes groups. To check if ClusterRole and ClusterRoleBinding have the eks:node:boostrapper role, run the following commands:

kubectl describe clusterrole eks:node-bootstrapper
kubectl describe clusterrolebinding eks:node-bootstrapper

Example output:

kubectl describe clusterrole eks:node-bootstrapper 
 Name: eks:node-bootstrapper
 Labels: eks.amazonaws.com/component=node
 Annotations: <none>
 PolicyRule:
 Resources Non-Resource URLs Resource Names Verbs
 --------- ----------------- -------------- -----
 certificatesigningrequests.certificates.k8s.io/selfnodeserver [] [] [create]

$ kubectl describe clusterrolebinding eks:node-bootstrapper 
 Name: eks:node-bootstrapper
 Labels: eks.amazonaws.com/component=node
 Annotations: <none>
 Role:
 Kind: ClusterRole
 Name: eks:node-bootstrapper
 Subjects:
 Kind Name Namespace
 ---- ---- ---------
 Group system:bootstrappers 
 Group system:nodes

If the eks:node-bootstrapper role is missing, then create a YAML file with the following configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks:node-bootstrapper
  labels:
    eks.amazonaws.com/component: node
rules:
  - apiGroups: ["certificates.k8s.io"]
    resources: ["certificatesigningrequests/selfnodeserver"]
    verbs: ["create"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks:node-bootstrapper
  labels:
    eks.amazonaws.com/component: node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eks:node-bootstrapper
subjects:
  - kind: Group
    name: system:bootstrappers
  - kind: Group
    name: system:nodes

Then, run the following command to update the Amazon EKS cluster:

 kubectl apply -f file-name command

Note: Replace file-name with your YAML file name.

If you use the ConfigMap authentication method, then you must also attach the instance-role that you configured in the aws-auth file to the system:bootstrappers and system:nodes groups. Add the instance-role to aws-auth in the following format:

kubectl get cm aws-auth -n kube-system -o yaml
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::12345678912:role/kubectl-cluster-nodegroup-custo-NodeInstanceRole-1KFZHWE6FCBDS
      username: system:node:{{EC2PrivateDNSName}}

Note: If you configured the Amazon EKS API as the cluster authentication mode, then check your configuration. Make sure that the worker node AWS Identity and Access Management (IAM) role maps to the system:node:{{EC2PrivateDNSName}} username. This configuration makes sure that Amazon EKS recognizes the CSR requester and automatically approves the CSR. Amazon EKS only automatically approves CSRs from the system:node:{{EC2PrivateDNSName}}.

To check requester details, run the following command:

kubectl describe csr csr-name

Note: Replace csr-name with your CSR name.

Example output:

Name:               csr-tpb4m
Labels:             <none>
Annotations:        <none>
CreationTimestamp:  Tue, 12 Dec 2023 11:18:30 +0530
Requesting User:    system:node:ip-000-00-00-000.ec2.internal
Signer:             kubernetes.io/kubelet-serving
Status:             Approved,Issued
Subject:
  Common Name:    system:node:ip-172-31-73-240.ec2.internal
  Serial Number:  
  Organization:   system:nodes
Subject Alternative Names:
         DNS Names:     ip-172-31-73-240.ec2.internal
         IP Addresses:  172.31.73.240
Events:  <none>

Make sure that the CSR requester is in the system:node:ip-abc-xx-x-xabc.ec2.internal format. If you use the same IAM role that created the cluster for worker nodes, then the requester might be kubernetes-admin instead of system:node:EC2PrivateDNSName. Amazon EKS rejects requests that aren't from system:node:EC2PrivateDNSName. If you mapped the worker node role to a custom username in aws-auth, then make sure that you correctly mapped the worker node role to system:node:EC2PrivateDNSName.

Check the kubelet logs

To check the status of the kubelet, run the following command:

systemctl status kubelet 

If the output shows that the kubelet status is Stopped, then run the following command to restart the kubelet:

sudo systemctl restart kubelet

To check the kubelet logs for the cert keyword, run the following command:

 journalctl -u kubelet | grep cert

Example error:

kubelet[8070]: I1021 18:49:21.594143 8070 log.go:184] http: TLS handshake error from 192.168.130.116:38710: no serving certificate available for the kubelet

If you don't see cert errors in the command output, then you can submit a new CSR. For more detailed logs, use the --v=4 flag. To collect worker node logs for detailed analysis, use the log-collector-script on the GitHub website.

Check the control plane logs

Prerequisite: Allow the Amazon EKS cluster to send control plane logs to Amazon CloudWatch Logs Insights.

For further insight into CSR issues, run the following command to use CloudWatch Logs Insights to check the control plane logs:

fields @timestamp, @message, @logStream, @log
| filter message like 'csr-name'
| sort @timestamp desc
| limit 10000

Note: Replace csr-name with the name of the CSR that has issues.

Kubectl can't reach the control plane

Check access to the Amazon EKS endpoint

If you activated private access for your cluster's Kubernetes API server endpoint, then you can only access the API server from the following sources:

• Your virtual private network (VPC)
• A connected network

If you run kubectl from a client that isn't part of the VPC or in a connected network, then you can't reach the cluster's API server. You receive an error similar to the following example:

"E1009 12:33:44.852680 106 memcache.go:265] couldn't get current server API group list: Get "APISERVERENDPOINT": dial tcp 11.11.111.111:443: i/o timeout"

To resolve this issue, change the cluster endpoint access to Public access. If you must have a private cluster, then create an Amazon Elastic Compute Cloud (Amazon EC2) bastion host in the cluster's VPC. Then, use the bastion host to access the cluster. You must add the bastion host's security group to the cluster's security group.

Check your host and port configurations

If you receive the The connection to the server localhost:8080 was refused error message, then check your host and port configurations. The error typically occurs when kubectl can't find the correct port and host information of the Amazon EKS API server endpoint in the kubeconfig file.

To resolve this issue, complete the following steps:

1. To update the kubeconfig file, run the following update-kubeconfig AWS CLI command:

   aws eks update-kubeconfig --region region-code --name my-cluster

   Note: Replace region-code with your AWS Region and my-cluster with your cluster name.

2. To view your current context, run the following command:

   kubectl config current-context

3. To check the IAM user or role that you configured on your environment, run the following get-caller-identity command:

   aws sts get-caller-identity

To troubleshoot IAM issues, see How do I provide cluster access to other IAM users and roles after I create a cluster in Amazon EKS?

Check control plane and worker node security group rules

If your control plane and node security group rules block access to the API server, then you receive the following error:

"The connection to the server APISERVERENDPOINT was refused - did you specify the right host or port?"

To resolve this issue, make sure that the control plane and node security groups have the required inbound and outbound rules. The API server security group must allow inbound access from the API server client on port 443.

You can't install kubectl

You must use a kubectl version that's within one minor version difference of your cluster. For example, a v1.31 client can communicate with control plane versions v1.30, v1.31, and v1.32. To avoid issues, install the latest compatible version of kubectl.

When you install kubectl, you might receive one of the following errors:

"error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

-or-

"Unable to connect to the server: getting credentials: decoding stdout: no kind "ExecCredential" is registered for version "client.authentication.k8s.io/v1alpha1" in scheme "pkg/client/auth/exec/exec.go:62""

The preceding errors typically occur when you use an earlier version of the AWS CLI to run the update-kubeconfig command. To resolve this issue, follow update the AWS CLI. For more information, see Deprecate Kubernetes client API version v1alpha1 on the GitHub website.

You encounter authorization errors when you run kubectl commands

You might encounter the following error when you run kubectl commands:

"error: You must be logged in to the server (Unauthorized)"

To resolve this issue, see How do I resolve the error "You must be logged in to the server (Unauthorized)" when I connect to the Amazon EKS API server?