How can I use a dead-letter queue to troubleshoot FailedInvocations for EventBridge rules?

Short description

To find the reason for FailedInvocations, associate an Amazon Simple Queue Service (Amazon SQS) DLQ with the target of an EventBridge rule. If any events fail to deliver to the target, then EventBridge sends the failed events to the DLQ.

To configure a DQL for EventBridge, complete the following steps:

1. Create an Amazon SQS standard queue.
2. Associate the Amazon SQS standard queue with the target of the EventBridge rule.
3. Send events to your EventBridge event bus that match your EventBridge rule.
4. Retrieve the failed event that EventBridge sent to the DLQ.

Resolution

Create an Amazon SQS standard queue

Complete the following steps:

1. Open the Amazon SQS console.
2. Choose Create queue.
3. For Type, choose Standard queue type.
4. Enter a Name for your queue. For example, myEventBridgeDLQ.
5. Choose Create queue.

Associate the Amazon SQS standard queue with the target of the EventBridge rule

Complete the following steps:

1. Open the Amazon EventBridge console.
2. In the navigation pane, choose Rules.
3. Select the EventBridge rule that failed to invoke your target, and then choose Edit.
4. In the navigation pane, choose Select target(s), and then chose Additional settings.
5. For Dead-letter queue, choose Select an Amazon SQS queue in the current AWS account to use as the dead-letter queue. Then, select your Amazon SQS queue in the Select an SQS queue dropdown list.
6. Choose Skip to Review and update, and then review the details.
7. Choose Update rule.

EventBridge must have permissions to send events with failed invocations to the Amazon SQS queue.

If you use the EventBridge console to associate a DLQ with the target of the EventBridge rule, then the permissions are automatically added.

If you use AWS Command Line Interface (AWS CLI), AWS SDK, or AWS CloudFormation to associate a DLQ with the target of the EventBridge rule, then you must manually apply the permissions in the resource policy. You can use the set-queue-attributes AWS CLI command to modify the policy. If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

Important: If the associated Amazon SQS queue is encrypted, then you must create a customer managed key. You must also include the following permission section in your AWS Key Management Service (AWS KMS) key policy.

{  
    "Sid": "Allow EventBridge to use the key",  
    "Effect": "Allow",  
    "Principal": {  
        "Service": "events.amazonaws.com"  
    },  
    "Action": [  
        "kms:Decrypt",  
        "kms:GenerateDataKey"  
    ],  
    "Resource": "*"  
}

For more information, see Configuring AWS KMS permissions.

Send events to your EventBridge event bus that match your EventBridge rule

To test your configuration, send events to your event bus that match your event rule.

To send events to your event bus that match your event rule, perform a change in your AWS environment for AWS service events. Or, use the PutEvents API for custom events. When the EventBridge rule initiates but fails to invoke the target, EventBridge sends all failed events to the DLQ.

Retrieve the failed event that EventBridge sent to the DLQ

Complete the following steps:

1. Open the Amazon EventBridge console.
2. In the navigation pane, choose Rules.
3. Select the EventBridge rule with the FailedInvocations Amazon CloudWatch metric data point and the configured DLQ.
4. Choose the Targets tab, and then select the configured DLQ.
5. Choose Send and receive messages. The Amazon SQS console displays the Send and receive messages page.
6. Choose Poll for messages.
   Note: The Messages section displays a list of the received messages. For each message, the list displays the message ID, sent date, size, and receive count.
7. Select one of the messages. Then, choose View details to see the event that EventBridge failed to send to the target.
8. Choose the Attributes tab to see the ERROR_CODE, ERROR_MESSAGE, RULE_ARN, and TARGET_ARN attributes.
   Note: The ERROR_CODE and ERROR_MESSAGE attributes provide the reason for the event delivery failure. For example, ERROR_CODE: NO_PERMISSIONS indicates that the target's resource-based or identity-based policy doesn't have the required permissions.

Related information

How can I troubleshoot issues with EventBridge rules?

Amazon SQS permissions