How do I resolve API errors when I upgrade my cluster version in Amazon EKS?

Short description

During a cluster version upgrade, the Upgrade insights tab in the Amazon EKS console shows the status of various Kubernetes components. During the upgrade process for different components, you see the following statuses:

• An Error status shows that the cluster rejects calls that reference the specific API version after the cluster upgraded.
• A Warning status shows that there's an issue, but no immediate action is necessary. This status can happen when the Kubernetes resource is scheduled to be removed in a version that is at least 2 versions older than the current cluster version.
• A Passing status shows that the component passed all of the upgrade checks and validations. No action is needed.
• An Unknown status shows that the upgrade system can't determine the status for the component, possibly due to lack of information or an inconclusive state during the process. You might need to take action.

Before you upgrade your cluster, review components with an Error or Warning status to understand and resolve any issues.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Check the status of APIs

Use either the AWS Management Console or the AWS CLI to check the statuses of deprecated APIs in your cluster.

AWS Management Console

Complete the following steps:

1. Open the Amazon EKS console.
2. Select your cluster name.
3. In Cluster info, choose Upgrade insights.
4. Review any deprecated API warnings or error messages under Insight status.

AWS CLI

To list all upgrade insight details in the YAML file, run the following command:

 [+] aws eks list-insights --cluster-name your-cluster-name --region your-region-name --output yaml >> insight.yaml

Note: Replace your-cluster-name and your-region-name with the values from your cluster.

Save the output in your preferred file format.

To filter the insights to only show errors, run the following command:

[+] aws eks list-insights --cluster-name your-cluster-name --region your-region-name --filter categories=UPGRADE_READINESS,statuses=ERROR

Note: Replace your-cluster-name and your-region-name with the values from your cluster.

The output is saved in text format.

Resolve a deprecated API status error

If the Upgrade insights tab status shows a status that your cluster has a deprecated API, you might receive an error. For example, if a pod disruption budget (PDB) API error occurs, you might receive a message that looks like the following:

"EKS using deprecated api version "/apis/policy/v1beta1/poddisruptionbudgets" and needs to replace it with "/apis/policy/v1/poddisruptionbudgets" before upgrading EKS cluster."

To resolve the error, complete the following steps:

1. Run the following command to list the PDBs that the cluster uses:

   kubectl get pdb -A

2. Run the following command to see the output to view PDBs that use either the latest API version or a deprecated version:

   kubectl get pdb your-pdb-name -n your-namespace -o yaml

   Note: Replace your-pdb-name with your PDB and replace your-namespace with your namespace.

3. If any of your PDBs uses a deprecated API version, then update it to use the latest API version in your YAML config files. For more information, see Specifying a Disruption Budget for your Application on the Kubernetes website.

4. If you can't determine the entity that calls the deprecated API because it's a specific user agent, such as kube-apiserver, newrelic, or kube-controller-manager, then check the kube-api-server-audit log file of your EKS cluster to determine the entity.
   Note: Your EKS cluster must have Cloudwatch audit logs enabled to run following query.

5. Open the Amazon Cloudwatch Console.

6. Choose Logs Insights, and then select your log group from the menu. For example, -/aws/eks/eksworkshop/cluster.

7. Filter the time and date to show the Last request time.
   Query:

   fields @logStream, @timestamp, responseStatus.code, @message
   | filter @logStream like /^kube-apiserver-audit/
   | filter requestURI like /\/apis\/autoscaling\/v2beta2\/horizontalpodautoscalers/    ### ---> Replace this with your Deprecated API's
   | sort @timestamp desc
   | limit 1000

   Output:

   {
      "kind":"Event",
      "apiVersion":"audit.k8s.io/v1",
      "level":"Request",
      "auditID":"bd2b2b0c-8556-4468-8b35-91e2e78759a6",
      "stage":"ResponseComplete",
      "requestURI":"/apis/autoscaling/v2beta2/horizontalpodautoscalers?allowWatchBookmarks=true\u0026resourceVersion=118801277\u0026timeout=5m51s\u0026timeoutSeconds=351\u0026watch=true",
      "verb":"watch",
      "user":{
         "username":"system:serviceaccount:newrelic:newrelic-kube-state-metrics",
         "uid":"9b97a09a-cf70-4fe2-8fd8-6c87110ce672",
         "groups":[
            "system:serviceaccounts",
            "system:serviceaccounts:newrelic",
            "system:authenticated"
         ],
         "extra":{
            "authentication.kubernetes.io/pod-name":[
               "newrelic-kube-state-metrics-5bb6ccc94d-8nj9v"
            ],
            "authentication.kubernetes.io/pod-uid":[
               "5b156744-00b6-4dfb-a998-a32140409b8e"
            ]
         }
      },
      "sourceIPs":[
         "10.62.98.22"
      ],
      "userAgent":"v2.6.0",
      "objectRef":{
         "resource":"horizontalpodautoscalers",
         "apiGroup":"autoscaling",
         "apiVersion":"v2beta2"
      },
      "responseStatus":{
         "metadata":{
            
         },
         "code":200
      },
      "requestReceivedTimestamp":"2024-12-13T20:13:04.438616Z",
      "stageTimestamp":"2024-12-13T20:18:55.441744Z",
      "annotations":{
         "authorization.k8s.io/decision":"allow",
         "authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"newrelic-kube-state-metrics\" of ClusterRole \"newrelic-kube-state-metrics\" to ServiceAccount \"newrelic-kube-state-metrics/newrelic\"",
         "k8s.io/deprecated":"true",
         "k8s.io/removed-release":"1.26"
      }
   }

8. To update the deprecated API with the latest API, take one of the following actions:
   If the username shows cluster-autoscaler, then update your cluster autoscaler to latest version.
   If the username shows newrelic-kube-state-metric, then update the cluster to the newer kube-state-metrics version if installed separately. Or, if kube-state-metrics is bundled, upgrade the new relic plugin.
   Note: AWS scans the deprecated API for the past 30 days. After you update cluster-autoscaler or newrelic-kube-state-metric, it might take up to 30 days for AWS to scan the deprecated API and update the status on the Update insights tab.