Patch CoreDNS command is giving error

0

My EKS Fargate cluster throws error for CoreDNS. I tried the command that is mentioned here: https://repost.aws/questions/QUT6VOlwOXSdmvGLIcaGeerw/core-dns-pods-in-kube-system-not-working

I get this error on running the command:

$ kubectl patch deployment coredns -n kube-system --type=json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations", "value": "eks.amazonaws.com/compute-type"}]' The request is invalid: the server rejected our request due to an error in our request

Dev-TVX
asked 10 months ago1096 views
2 Answers
0
Accepted Answer

Please follow the below steps to troubleshoot your error:

Step 1. Execute kubectl describe deploy coredns -n kube-system and check the Pod template section. Does it show something like:

Pod Template:
  Labels:           eks.amazonaws.com/component=coredns
                    k8s-app=kube-dns
  Annotations:      eks.amazonaws.com/compute-type: ec2
   Service Account:  coredns

if there is annotations field in the deployment please execute the command:

kubectl patch deployment coredns \ -n kube-system \ --type json \ -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'

If there if no annotation field like below:

Pod Template:
  Labels:           eks.amazonaws.com/component=coredns
                    k8s-app=kube-dns
  Service Account:  coredns

then proceed to step 2 as executing the patch command again will result in error as we are trying to remove the annotations field from the deployment which does not exist.

Step 2 - Execute - k describe pod <your-coredns-podname> -n kube-system and look for Events field, i.e. Is it showing something like below?

Events:
  Type     Reason            Age               From               Message
  ----     ------            ----              ----               -------
  Warning  FailedScheduling  41s (x2 over 6m)  default-scheduler  no nodes available to schedule pods

Based on the error above, you will be able to find the reason of why coredns pods are not able to schedule.

If suppose the error is something similar to above, please create a fargate profile to target coredns pods like this below (replace my-cluster with your cluster name, 111122223333 with your account ID, AmazonEKSFargatePodExecutionRole with the name of your Pod execution role, and 0000000000000001, 0000000000000002, and 0000000000000003 with the IDs of your private subnets. If you don't have a Pod execution role, you must create one first. ):

aws eks create-fargate-profile \
    --fargate-profile-name coredns \
    --cluster-name my-cluster \
    --pod-execution-role-arn arn:aws:iam::111122223333:role/AmazonEKSFargatePodExecutionRole \
    --selectors namespace=kube-system,labels={k8s-app=kube-dns} \
    --subnets subnet-0000000000000001 subnet-0000000000000002 subnet-0000000000000003

If you are not able to troubleshoot based on the above, please let me know and I would be happy to help based on your comments.

Thanks, Manish

profile picture
Manish
answered 10 months ago
profile picture
EXPERT
reviewed 2 months ago
  • Hi Manish, I followed the above solution but getting a weird error. . The Deployment has the following annotation -

    Pod Template:
      Labels:           eks.amazonaws.com/component=coredns
                        k8s-app=kube-dns
      Annotations:      kubectl.kubernetes.io/restartedAt: 2023-08-31T12:14:13+05:30
      Service Account:  coredns
    

    And when i am trying to remove the annotation with the following command -

    kubectl patch deployment coredns \
        -n kube-system \
        --type json \
        -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/kubectl.kubernetes.io/restartedAt"}]'
    

    I get this weird error-

    The request is invalid: the server rejected our request due to an error in our request
    

    Could you please help ?

  • Hi Poonam,

    You do not need to remove the annotation "kubectl.kubernetes.io/restartedAt". If your coredns pods are not coming up, could you please execute the command in step 2 and paste the "Message" you get from "Events" field please?

    Thanks, Manish

0

Hi Manish,

I am also getting same error while I attempt to patch coredns -

$  kubectl patch deployment coredns -n kube-system --type json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'
The request is invalid

here are my annotation values -

$  kubectl describe deploy coredns -n kube-system | grep -i Annotations

Annotations:            deployment.kubernetes.io/revision: 2
Annotations:      kubectl.kubernetes.io/restartedAt: 2023-10-24T11:58:53Z

& here is what the current event says -

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m50s  fargate-scheduler  Pod provisioning timed out (will retry) for pod: kube-system/coredns-55f54c7b9c-wg6zx
  Warning  LoggingDisabled   4m48s  fargate-scheduler  Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
  Warning  FailedScheduling  108s   fargate-scheduler  Pod provisioning timed out (will retry) for pod: kube-system/coredns-55f54c7b9c-wg6zx
  Warning  LoggingDisabled   106s   fargate-scheduler  Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found

k/r Punit

answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions