How to monitor pod failure status on Amazon EKS Fargate-only clusters using kube-state-metrics and ADOT

12 minute read

Content level: Intermediate

In Amazon EKS Fargate-only clusters, the pod_status_failed metric from Container Insights with enhanced observability is not available. This is because enhanced observability requires the CloudWatch agent deployed as a DaemonSet, which is not supported on Fargate.

This article explains how to monitor pod failure status on Fargate-only clusters by deploying kube-state-metrics and AWS Distro for OpenTelemetry (ADOT) Collector to send kube_pod_status_phase metrics to Amazon CloudWatch.

Problem

You enabled Container Insights on your EKS Fargate cluster using ADOT, but the pod_status_failed metric listed in the Container Insights with enhanced observability metrics documentation does not appear in CloudWatch.

The ADOT-based Container Insights for Fargate only collects the following 8 metrics from kubelet's cAdvisor endpoint:

pod_cpu_utilization_over_pod_limit
pod_cpu_usage_total
pod_cpu_limit
pod_memory_utilization_over_pod_limit
pod_memory_working_set
pod_memory_limit
pod_network_rx_bytes
pod_network_tx_bytes

Pod status phase metrics (such as pod_status_failed, pod_status_running) are not included because they require the CloudWatch agent to query the Kubernetes API, which runs as a DaemonSet — unsupported on Fargate.

Resolution

Deploy kube-state-metrics as a Deployment on Fargate to collect pod status information from the Kubernetes API, then use an ADOT Collector to scrape these metrics and send them to CloudWatch.

Why the `label_matchers` filter is required

kube_pod_status_phase is emitted by kube-state-metrics once per pod per phase (Pending, Running, Succeeded, Failed, Unknown), where only the current phase has the value 1.

The phase label is not part of the CloudWatch dimension set, so without a filter all five series collapse into a single metric. A healthy Running pod then produces five datapoints per minute (SampleCount = 5) with a Maximum of 1, which is indistinguishable from a failed pod and causes alarms to fire constantly.

The label_matchers block in the configuration below keeps only the phase="Failed" series, so the metric becomes 0 for healthy pods and 1 for failed pods.

Architecture

kube-state-metrics (Deployment on Fargate)
    ↓ Exposes Prometheus metrics on port 8080
ADOT Collector (Deployment on Fargate)
    ↓ Prometheus receiver scrapes kube-state-metrics
    ↓ AWS EMF Exporter sends to CloudWatch
Amazon CloudWatch (ContainerInsights namespace)
    ↓
CloudWatch Alarm (optional)

Prerequisites

An Amazon EKS cluster with Fargate profiles
kubectl configured to access the cluster
AWS CLI configured with appropriate permissions
An OIDC provider associated with the EKS cluster (for IRSA)

Step 1: Create a Fargate profile for the monitoring namespace

aws eks create-fargate-profile \
  --cluster-name <CLUSTER_NAME> \
  --fargate-profile-name monitoring \
  --pod-execution-role-arn <FARGATE_POD_EXECUTION_ROLE_ARN> \
  --subnets <SUBNET_ID_1> <SUBNET_ID_2> \
  --selectors namespace=monitoring \
  --region <REGION>

Wait until the profile status becomes ACTIVE:

aws eks describe-fargate-profile \
  --cluster-name <CLUSTER_NAME> \
  --fargate-profile-name monitoring \
  --region <REGION> \
  --query 'fargateProfile.status'

Step 2: Create the monitoring namespace

kubectl create namespace monitoring

Step 3: Deploy kube-state-metrics

Save the following as kube-state-metrics.yaml and apply it:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - serviceaccounts
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
- apiGroups: ["authentication.k8s.io"]
  resources:
  - tokenreviews
  verbs: ["create"]
- apiGroups: ["authorization.k8s.io"]
  resources:
  - subjectaccessreviews
  verbs: ["create"]
- apiGroups: ["policy"]
  resources:
  - poddisruptionbudgets
  verbs: ["list", "watch"]
- apiGroups: ["certificates.k8s.io"]
  resources:
  - certificatesigningrequests
  verbs: ["list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources:
  - storageclasses
  - volumeattachments
  verbs: ["list", "watch"]
- apiGroups: ["admissionregistration.k8s.io"]
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs: ["list", "watch"]
- apiGroups: ["networking.k8s.io"]
  resources:
  - networkpolicies
  - ingresses
  verbs: ["list", "watch"]
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: monitoring
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-state-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.13.0
        ports:
        - name: http-metrics
          containerPort: 8080
        - name: telemetry
          containerPort: 8081
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            memory: 256Mi
        readinessProbe:
          httpGet:
            path: /
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-state-metrics
spec:
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
  - name: telemetry
    port: 8081
    targetPort: telemetry
  selector:
    app.kubernetes.io/name: kube-state-metrics

kubectl apply -f kube-state-metrics.yaml

Verify the pod is running:

kubectl get pods -n monitoring

Expected output:

NAME                                  READY   STATUS    RESTARTS   AGE
kube-state-metrics-66b44f6fbd-697sx   1/1     Running   0          2m

Step 4: Configure IAM Role for the ADOT Collector (IRSA)

Create an IAM role with the CloudWatchAgentServerPolicy managed policy. Update the trust policy to allow the monitoring:adot-ksm-collector service account to assume the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com",
          "oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:monitoring:adot-ksm-collector"
        }
      }
    }
  ]
}

Apply the trust policy:

aws iam update-assume-role-policy \
  --role-name <IAM_ROLE_NAME> \
  --policy-document file://trust-policy.json

Step 5: Deploy ADOT Collector

Save the following as adot-ksm-collector.yaml and apply it. Replace <ACCOUNT_ID>, <IAM_ROLE_NAME>, <CLUSTER_NAME>, and <REGION> with your values:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: adot-ksm-collector
  namespace: monitoring
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: adot-ksm-collector-config
  namespace: monitoring
data:
  config.yaml: |
    receivers:
      prometheus:
        config:
          global:
            scrape_interval: 60s
            scrape_timeout: 30s
          scrape_configs:
            - job_name: 'kube-state-metrics'
              static_configs:
                - targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']

    processors:
      filter/ksm:
        metrics:
          include:
            match_type: regexp
            metric_names:
              - kube_pod_status_phase

      metricstransform:
        transforms:
          - include: kube_pod_status_phase
            action: update
            operations:
              - action: update_label
                label: namespace
                new_label: Namespace
              - action: update_label
                label: pod
                new_label: PodName

      resourcedetection:
        detectors: [env]

      batch:
        timeout: 60s

    exporters:
      awsemf:
        log_group_name: '/aws/containerinsights/{ClusterName}/performance'
        log_stream_name: 'kube-state-metrics'
        namespace: 'ContainerInsights'
        region: <REGION>
        resource_to_telemetry_conversion:
          enabled: true
        dimension_rollup_option: NoDimensionRollup
        metric_declarations:
          - dimensions: [[ClusterName, Namespace, PodName], [ClusterName]]
            metric_name_selectors:
              - kube_pod_status_phase
            label_matchers:
              - label_names: [phase]
                regex: Failed

    extensions:
      health_check:

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [filter/ksm, metricstransform, resourcedetection, batch]
          exporters: [awsemf]
      extensions: [health_check]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: adot-ksm-collector
  namespace: monitoring
  labels:
    app: adot-ksm-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: adot-ksm-collector
  template:
    metadata:
      labels:
        app: adot-ksm-collector
    spec:
      serviceAccountName: adot-ksm-collector
      containers:
        - name: adot-collector
          image: public.ecr.aws/aws-observability/aws-otel-collector:v0.40.0
          command:
            - "/awscollector"
            - "--config=/conf/config.yaml"
          env:
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "ClusterName=<CLUSTER_NAME>"
            - name: EKS_CLUSTER_NAME
              value: "<CLUSTER_NAME>"
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - name: config
          configMap:
            name: adot-ksm-collector-config
            items:
              - key: config.yaml
                path: config.yaml

kubectl apply -f adot-ksm-collector.yaml

Step 6: Verify the deployment

Check that both pods are running:

kubectl get pods -n monitoring

Expected output:

NAME                                  READY   STATUS    RESTARTS   AGE
adot-ksm-collector-85db5d9979-qzbww   1/1     Running   0          2m
kube-state-metrics-66b44f6fbd-697sx   1/1     Running   0          5m

Verify ADOT Collector logs show successful scraping:

kubectl logs -l app=adot-ksm-collector -n monitoring --tail=20

Expected log output (no errors):

detected resource information  {"resource": {"ClusterName":"<CLUSTER_NAME>"}}
Scrape job added  {"jobName": "kube-state-metrics"}
Everything is ready. Begin running and processing data.

Step 7: Verify metrics in CloudWatch

After approximately 2 minutes, verify the metric appears in CloudWatch:

aws cloudwatch list-metrics \
  --namespace ContainerInsights \
  --metric-name kube_pod_status_phase \
  --region <REGION>

You should see two dimension sets for the metric:

ClusterName, Namespace, PodName — per-pod failure status (0 or 1)
ClusterName — cluster-wide, where Sum equals the number of failed pods

Step 8: Test with a failed pod

Create a pod that intentionally fails:

kubectl run test-fail --image=busybox --restart=Never -n monitoring -- /bin/sh -c "exit 1"

Wait until the pod reaches the Error status (status.phase = Failed):

kubectl get pod test-fail -n monitoring

After 3-4 minutes, query CloudWatch to confirm the failure is detected. Compare the failed pod against a healthy one to verify there are no false positives.

On Linux use $(date -u -d '6 minutes ago' '+%Y-%m-%dT%H:%M:%S') for the start time. On macOS/BSD use $(date -u -v-6m '+%Y-%m-%dT%H:%M:%S') (lowercase m for minutes).

# macOS/BSD:
START=$(date -u -v-6m '+%Y-%m-%dT%H:%M:%S')
# Linux:
# START=$(date -u -d '6 minutes ago' '+%Y-%m-%dT%H:%M:%S')

aws cloudwatch get-metric-data \
  --metric-data-queries '[
   {"Id":"failed","MetricStat":{"Metric":{"Namespace":"ContainerInsights","MetricName":"kube_pod_status_phase","Dimensions":[{"Name":"ClusterName","Value":"<CLUSTER_NAME>"},{"Name":"Namespace","Value":"monitoring"},{"Name":"PodName","Value":"test-fail"}]},"Period":60,"Stat":"Maximum"},"ReturnData":true},
   {"Id":"healthy","MetricStat":{"Metric":{"Namespace":"ContainerInsights","MetricName":"kube_pod_status_phase","Dimensions":[{"Name":"ClusterName","Value":"<CLUSTER_NAME>"},{"Name":"Namespace","Value":"monitoring"},{"Name":"PodName","Value":"<KUBE_STATE_METRICS_POD_NAME>"}]},"Period":60,"Stat":"Maximum"},"ReturnData":true},
   {"Id":"clusterwide","MetricStat":{"Metric":{"Namespace":"ContainerInsights","MetricName":"kube_pod_status_phase","Dimensions":[{"Name":"ClusterName","Value":"<CLUSTER_NAME>"}]},"Period":60,"Stat":"Sum"},"ReturnData":true}]' \
  --start-time $START \
  --end-time $(date -u '+%Y-%m-%dT%H:%M:%S') \
  --region <REGION>

Expected results:

Query	Expected value	Meaning
`failed`	`1.0`	The failed pod is detected
`healthy`	`0.0`	A running pod does not report a failure
`clusterwide`	`1.0`	One failed pod exists in the cluster

Clean up the test pod:

kubectl delete pod test-fail -n monitoring

After the pod is deleted, the cluster-wide Sum returns to 0.

Step 9 (Optional): Create a CloudWatch Alarm

Because pod names change on every deployment, alarming on the per-pod dimension set is impractical. Use the cluster-wide dimension set instead, where Sum represents the number of failed pods:

aws cloudwatch put-metric-alarm \
  --alarm-name "EKS-Fargate-Pod-Failed" \
  --namespace ContainerInsights \
  --metric-name kube_pod_status_phase \
  --dimensions Name=ClusterName,Value=<CLUSTER_NAME> \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions <SNS_TOPIC_ARN> \
  --region <REGION>

Verify the alarm state:

aws cloudwatch describe-alarms \
  --alarm-names "EKS-Fargate-Pod-Failed" \
  --region <REGION> \
  --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason}'

The alarm transitions to ALARM while a failed pod exists, and returns to OK after the failed pod is removed.

To scope the alarm to a single namespace instead, add a [ClusterName, Namespace] dimension set to metric_declarations in the ADOT configuration.

Important notes

CoreDNS on Fargate: The ADOT Collector resolves kube-state-metrics.monitoring.svc.cluster.local via DNS. On Fargate-only clusters, ensure CoreDNS is also running on Fargate by creating a Fargate profile for the kube-system namespace.
Existing ADOT Collector: If you already have an ADOT Collector deployed for Container Insights (CPU/memory metrics), keep it as-is. This solution deploys a separate collector specifically for kube-state-metrics, avoiding interference with your existing monitoring pipeline.
Detection delay: With a 60-second scrape interval and a 60-second batch timeout, a pod failure appears in CloudWatch roughly 2-3 minutes after it occurs. Lower scrape_interval and the batch timeout if you need faster detection.
Additional metrics: See the section below to also detect CrashLoopBackOff, ImagePullBackOff, and OOMKilled.
Scope: kube-state-metrics queries the Kubernetes API with a ClusterRole, so it monitors pod status across all namespaces in the cluster, regardless of which namespace it runs in.

Optional: also detect CrashLoopBackOff, ImagePullBackOff, and OOMKilled

kube_pod_status_phase only reports the pod-level Failed phase. A container stuck in CrashLoopBackOff or ImagePullBackOff keeps its pod in the Pending phase, so those conditions are not covered. Add the following metrics to detect them.

Metric	Detects	Labels
`kube_pod_container_status_waiting_reason`	`CrashLoopBackOff`, `ImagePullBackOff`, `ErrImagePull`, `CreateContainerConfigError`	`namespace`, `pod`, `container`, `reason`
`kube_pod_container_status_last_terminated_reason`	`OOMKilled`, `Error`	`namespace`, `pod`, `container`, `reason`
`kube_pod_container_status_restarts_total`	Container restarts	`namespace`, `pod`, `container`

Unlike kube_pod_status_phase, these metrics are sparse: kube-state-metrics emits a series only for the reason that is currently active, always with the value 1, and emits nothing when the container is healthy. They therefore need no label_matchers filter. Instead, expose reason as a dimension so each condition becomes its own metric stream.

Update the three relevant sections of the ADOT config.yaml:

    processors:
      filter/ksm:
        metrics:
          include:
            match_type: regexp
            metric_names:
              - kube_pod_status_phase
              - kube_pod_container_status_waiting_reason
              - kube_pod_container_status_last_terminated_reason
              - kube_pod_container_status_restarts_total

      metricstransform:
        transforms:
          - include: kube_pod_.*
            match_type: regexp
            action: update
            operations:
              - action: update_label
                label: namespace
                new_label: Namespace
              - action: update_label
                label: pod
                new_label: PodName
              - action: update_label
                label: container
                new_label: ContainerName

        metric_declarations:
          - dimensions: [[ClusterName, Namespace, PodName], [ClusterName]]
            metric_name_selectors:
              - kube_pod_status_phase
            label_matchers:
              - label_names: [phase]
                regex: Failed
          - dimensions: [[ClusterName, Namespace, PodName, ContainerName, reason], [ClusterName, reason]]
            metric_name_selectors:
              - kube_pod_container_status_waiting_reason
              - kube_pod_container_status_last_terminated_reason
          - dimensions: [[ClusterName, Namespace, PodName, ContainerName], [ClusterName]]
            metric_name_selectors:
              - kube_pod_container_status_restarts_total

Note that the metricstransform processor now uses a regular expression so the label renames apply to all kube_pod_* metrics, and that each metric group needs its own metric_declarations entry.

Verified behavior

Testing with a CrashLoopBackOff pod, an ImagePullBackOff pod, and an OOMKilled pod produced the following:

Query	Result
`kube_pod_container_status_waiting_reason`, dimensions `[ClusterName, reason=CrashLoopBackOff]`, `Sum`	`2` (two affected containers)
`kube_pod_container_status_waiting_reason`, dimensions `[ClusterName, reason=ImagePullBackOff]`, `Sum`	`1`
`kube_pod_container_status_last_terminated_reason`, dimensions `[ClusterName, reason=OOMKilled]`, `Sum`	`1`
`kube_pod_container_status_restarts_total`, healthy pod, `Maximum`	`0`

Because these metrics are absent while everything is healthy, set --treat-missing-data notBreaching on any alarm built on them.

Two caveats

Use last_terminated_reason, not terminated_reason, for OOMKilled. kube_pod_container_status_terminated_reason exists only during the brief window when a container sits in the Terminated state. In testing, it never appeared across eight consecutive scrapes of a crash-looping pod, and it never reached CloudWatch. kube_pod_container_status_last_terminated_reason persists after the container restarts and reliably reported reason="OOMKilled".

restarts_total arrives in CloudWatch as a delta, not a cumulative total. kube-state-metrics exposes it as a Prometheus counter, and the awsemf exporter converts cumulative sums to per-period deltas. A container whose raw counter read 8 reported values of 0 and 1 per minute in CloudWatch, representing restarts within each period. This is convenient for alarming on new restarts, but do not read it as a lifetime restart count.