- Newest
- Most votes
- Most comments
Yes, this is expected behavior and relates to how CloudWatch handles delayed or late-arriving data points.
When your alarm evaluated at the specific moment, it used the data available at that time. However, AWS services (including Container Insights) can publish additional data points for the same timestamp after the alarm has already made its evaluation decision. When these delayed data points arrive, the CloudWatch Metrics graph recalculates and displays the updated aggregated value, but the alarm's decision was already made based on the earlier data.
In your case, the alarm likely evaluated when it had incomplete or different data points, calculated approximately 100%, and triggered the ALARM state. Later, when additional data points arrived for the same 15:05 UTC timestamp, the graph recalculated the average to show 20%.
This is particularly common with short-lived workloads like your 5-minute ECS task, where metrics may be published with slight delays as the task completes and Container Insights processes the data.
To verify what actually triggered your alarm, the Alarm History is your source of truth—not the Metrics graph. You can check the History tab in the CloudWatch console for your alarm and review the "State update" entries. The "New state reason" field will show the exact metric value that CloudWatch used during evaluation.
To reduce these discrepancies in the future, you could:
- Match your alarm period to the metric publishing interval (Container Insights typically publishes at 1-minute intervals)
- Consider using a longer evaluation period to allow more time for all data points to arrive before evaluation
- Be aware that for batch or short-lived workloads, some timing discrepancies between alarm evaluations and graph displays are normal
The key takeaway is that the alarm evaluated correctly based on the data available when it ran—the graph simply shows a different view after late-arriving data was incorporated.
Sources
Troubleshoot inconsistent CloudWatch metric values and data gaps | AWS re:Post
Understanding CloudWatch Alarm Evaluation: Why Alarms Delay, Fire Unexpectedly, or Get Stuck | AWS re:Post
