AWS re:Post Knowledge Center Feedback Survey

Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.

Why can't I connect to my Amazon EKS cluster?

5 minute read

I created an Amazon Elastic Kubernetes Service (Amazon EKS) cluster that I can't connect to.

Short description

You might not be able to connect to your Amazon EKS cluster because of one of the following reasons:

You didn't create the kubeconfig file for your cluster.
You can't connect to the Amazon EKS API server endpoint.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

You didn't create the kubeconfig file

After you create your Amazon EKS cluster, configure your kubeconfig file so that you can use the kubectl command line to connect to your cluster. The kubeconfig file contains your Kubernetes cluster configuration.

Based on your operating system (OS), you can find your kubeconfig file in the following locations:

For Linux or macOS, go to $HOME/.kube/config.
For Windows, go to %USERPROFILE%\.kube\config.

To use a different file location, take one of the following actions:

Set the KUBECONFIG environment variable to point to the new location.
Use the --kubeconfig flag when you run kubectl commands to specify a custom file path.

If a kubeconfig file already exists at the default location, then Amazon EKS merges new configurations with the existing file. You can use the kubectl config command to manage your cluster contexts. For more information, see kubectl config on the Kubernetes website.

You can't connect to the Amazon EKS API server endpoint

To troubleshoot this issue, complete the following steps:

To check whether you can connect to the Amazon EKS API server URL, run the following command with increased verbosity:

kubectl get svc --v=9

The command output shows detailed connection information, including the API server URL that your connection uses. Check the IP addresses that are listed for HTTP Trace: DNS Lookup.
Example output:

I0110 16:43:36.920095   48173 loader.go:373] Config loaded from file:  /Users/abs/.kube/configI0110 16:43:36.936844   48173 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.26.0 (darwin/arm64) kubernetes/b46a3f8" 'https://S123GBNS3HJUFN467UFGH6782JHCH2891.yl4.us-east-2.eks.amazonaws.com//api/v1/namespaces/default/services?limit=500'
I0110 16:43:37.362185   48173 round_trippers.go:495] HTTP Trace: DNS Lookup for S123GBNS3HJUFN467UFGH6782JHCH2891.yl4.us-east-2.eks.amazonaws.com/ resolved to [{18.119.155.77 } {3.136.153.3 }]
I0110 16:43:37.402538   48173 round_trippers.go:510] HTTP Trace: Dial to tcp:18.119.155.77:443 succeed
I0110 16:43:37.500276   48173 round_trippers.go:553] GET https://S123GBNS3HJUFN467UFGH6782JHCH2891.yl4.us-east-2.eks.amazonaws.com//api/v1/namespaces/default/services?limit=500 200 OK in 563 milliseconds
I0110 16:43:37.500302   48173 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 40 ms TLSHandshake 44 ms ServerProcessing 52 ms Duration 563 ms
I0110 16:43:37.500308   48173 round_trippers.go:577] Response Headers:
I0110 16:43:37.500316   48173 round_trippers.go:580] Audit-Id: 37c17136-7fa7-40e9-8fe6-b24426e81564
I0110 16:43:37.500323   48173 round_trippers.go:580] Cache-Control: no-cache, private
I0110 16:43:37.500329   48173 round_trippers.go:580] Content-Type: application/json
I0110 16:43:37.500334   48173 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 508eb99e-d99b-44db-8ade-838c99fe8e9f
I0110 16:43:37.500340   48173 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: d324d3db-05ce-441b-a0ff-c31cbe8f696c
I0110 16:43:37.500345   48173 round_trippers.go:580] Date: Tue, 10 Jan 2023 21:43:37 GMT

To check whether the Amazon EKS API server is publicly accessible, run the following describe-cluster AWS CLI command:

aws eks describe-cluster --name cluster_name --region example_region --query cluster.resourcesVpcConfig

Note: Replace cluster_name with your cluster name and example_region with your AWS Region.
Example output:

{    "subnetIds": [
        "subnet-abc1",
        "subnet-abc2",
        "subnet-abc3",
        "subnet-abc4",
        "subnet-abc5",
        "subnet-abc6"
    ],
    "securityGroupIds": [
       "sg-abc7"
    ],
    "clusterSecurityGroupId": "sg-abc7",
    "vpcId": "vpc-abc9",
    "endpointPublicAccess": true,
    "endpointPrivateAccess": false,
    "publicAccessCidrs": [
        "0.0.0.0/0"
    ]
}

To allow public access, make sure that endpointPublicAccess is true. To keep access private, make sure that endpointPublicAccess is false. If endpointPublicAccess is false, then proceed to step 9. If endpointPublicAccess is true, then it's a security best practice to limit endpointPublicAccess to specific IP address ranges. Complete steps 3-8 to add your allowed IP addresses to the publicAccessCidrs allowlist.

Open the Amazon EKS console.
Choose Clusters, and then select the cluster that you want to update.
Choose the Networking tab, and then choose Manage networking.
Choose Public.
Under Advanced settings, for CIDR block, enter the public CIDR range that you want to add to the allowlist.
Important: Include the IP address of the NAT gateway that the worker nodes in your private subnets use to access the internet. Also, include the IP addresses of all NAT device networking components.
Choose Save changes.

If you configured the cluster's API server in private-only mode, then make sure that the kubectl request comes from the cluster's network. If your kubectl request is from outside of your Amazon Virtual Private Cloud (Amazon VPC), then you receive the following timeout error when you run the get svc command:

$ kubectl get svc --v=9I0110 17:15:58.889798   50514 loader.go:373] Config loaded from file:  /Users/example-user/.kube/config
I0110 17:15:58.896715   50514 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.26.0 (darwin/arm64) kubernetes/b46a3f8" 'https://S123GBNS3HJUFN467UFGH6782JHCH2891.yl4.us-east-2.eks.amazonaws.com/api/v1/namespaces/default/services?limit=500'
I0110 17:15:59.374499   50514 round_trippers.go:495] HTTP Trace: DNS Lookup for S123GBNS3HJUFN467UFGH6782JHCH2891.yl4.us-east-2.eks.amazonaws.com resolved to [{192.168.126.17 } {192.168.144.26 }]
I0110 17:16:14.285027   50514 round_trippers.go:508] HTTP Trace: Dial to tcp:192.168.126.17:443 failed: dial tcp 192.168.126.17:443: i/o timeout
I0110 17:16:29.191768   50514 round_trippers.go:508] HTTP Trace: Dial to tcp:192.168.144.26:443 failed: dial tcp 192.168.144.26:443: i/o timeout
I0110 17:16:29.196959   50514 round_trippers.go:553] GET https://S123GBNS3HJUFN467UFGH6782JHCH2891.yl4.us-east-2.eks.amazonaws.com/api/v1/namespaces/default/services?limit=500  in 30300 milliseconds
I0110 17:16:29.197724   50514 round_trippers.go:570] HTTP Statistics: DNSLookup 183 ms Dial 14906 ms TLSHandshake 0 ms Duration 30300 ms
I0110 17:16:29.197768   50514 round_trippers.go:577] Response Headers:
I0110 17:16:29.199254   50514 helpers.go:264] Connection error: Get https://S123GBNS3HJUFN467UFGH6782JHCH2891.yl4.us-east-2.eks.amazonaws.com/api/v1/namespaces/default/services?limit=500: dial tcp 192.168.126.17:443: i/o timeout
Unable to connect to the server: dial tcp 192.168.126.17:443: i/o timeout

Update the cluster security group to add the source IP address or CIDR range to your allowlist. This allows the kubectl client to connect to the Amazon EKS API server endpoint from within the cluster VPC.