Cloudwatch: Why is there a delay between the data breach and the alarm being triggered?

Hello,

I have a cloud watch alarm that triggers when the InvocationsPerInstance metric for our Sagemaker endpoint breaches a threshold in one minute period. The general setup:

MetricName='InvocationsPerInstance',
Namespace='AWS/SageMaker',
Statistic='Sum',
Period=60,
EvaluationPeriods=1,
DatapointsToAlarm=1

From this, I would expect that after we see a one-minute period above the threshold the alarm would be triggered. However, there seems to be a 3 minute delay between when the data breaches the threshold and when the alarm is triggered. I want the alarm to trigger as soon as there is data that breaches if possible.

Here is the newState of the alarm. You can see that the trigger is at 15:24:00 but the alarm does not get triggered until 15:27:26

"newState": { "stateValue": "ALARM", "stateReason": "Threshold Crossed: 1 out of the last 1 datapoints [29752.0 (17/01/24 15:24:00)] was greater than the threshold (28800.0) (minimum 1 datapoint for OK -> ALARM transition).", "stateReasonData": { "version": "1.0", "queryDate": "2024-01-17T15:27:26.429+0000", "startDate": "2024-01-17T15:24:00.000+0000", "statistic": "Sum", "period": 60, "recentDatapoints": [ 29752 ], "threshold": 28800, "evaluatedDatapoints": [ { "timestamp": "2024-01-17T15:24:00.000+0000", "sampleCount": 29752, "value": 29752 } ] } } }