Enhancing critical workload monitoring for Incident Detection and Response with Datadog integration

Introduction

In today's cloud-first world, organizations need robust incident detection and response capabilities to maintain high availability and reliability. Incident Detection and Response provides 24/7 monitoring, detection, and response capabilities for critical workloads that run on AWS. The service combines the operational expertise of AWS with advanced tooling to help customers detect and respond to operational events quickly and effectively.

For organizations that run mission-critical workloads, integrating Incident Detection and Response with existing application performance monitoring (APM) tools, such as Datadog, creates a unified monitoring and response ecosystem. This integration allows Datadog's critical observability alerts to flow seamlessly into the Incident Detection and Response framework and provide faster incident resolution times and reduced mean time to recovery (MTTR).

Prerequisites and setup requirements

Before you implement this integration, make sure that you have the following requirements in place:

Required AWS Support plan:

You need an AWS Enterprise Support or AWS Enterprise On-Ramp Support plan to access AWS Incident Detection and Response.

AWS Identity and Access Management (IAM) permissions for Datadog integration:

For information on IAM permissions that you need for Datadog integration, see Datadog AWS IAM Permissions documentation.

Required Datadog permissions:

• Datadog admin role or a custom role with AWS integration configuration permissions.

• Valid Datadog API key with appropriate permissions.

• Access to create and configure Datadog monitors and notification rules.

Initial integration setup can take 15-30 minutes, and testing and validation can take 15-30 minutes. However, your setup time can vary depending on your familiarity with AWS services and existing Datadog configuration complexity.

Integration architecture

The integration between Incident Detection and Response and Datadog uses Amazon EventBridge and AWS Lambda to process and route monitoring alerts. You can configure Datadog to automatically send alert notifications to EventBridge when a monitor alert condition is met, as seen in the following image:

The architecture demonstrates the event flow from Datadog monitoring through to AWS Incident Detection and Response:

1. Datadog monitors your applications and infrastructure and generates alerts based on your defined monitor thresholds and conditions.

2. A partner event source is a connection in EventBridge that allows Datadog to send events to your AWS account.

3. An EventBridge partner event bus serves as the event bus that receives alerts from Datadog through the partner event source.

4. EventBridge rules route relevant alerts to a Lambda function for transformation.

5. Lambda transforms the Datadog alert payload into the format that Incident Detection and Response requires.

6. A custom EventBridge event bus receives the transformed events for Incident Detection and Response consumption.

7. Incident Detection and Response receives these alerts and initiates the appropriate response procedures according to your engagement model.

Integration configuration

Configure Datadog and integrate with AWS

To create an EventBridge integration in Datadog, you must have either the managed Datadog administrator role or a custom role with AWS integration configuration permissions. Before you configure the EventBridge integration, set up the main AWS integration in Datadog for the AWS account where you provision the EventBridge event bus. To do this, complete the following steps:

1. Open the Datadog console as an administrator.

2. Choose Integrations and search for Amazon Web Services.

3. Choose the Amazon Web Services integration tile. Then, choose Add AWS Account(s).

4. Use the guided setup to choose CloudFormation as the method. Then, include the following parameters:
   For AWS Region, select the AWS Region where you provision the CloudFormation stack.
   For Datadog API Key, create or select the appropriate Datadog API key for the AWS integration.
   For Datadog Application, create the Datadog application key automatically for the AWS integration.
   For Send AWS Logs to Datadog, select No.
   For Detect security issues, select No.

5. Choose Launch CloudFormation Template.
   You are then redirected to the CloudFormation console to deploy the required AWS resources for integrating Datadog and AWS.

Note: Be sure that the AWS integration is fully configured and active before you configure the EventBridge integration. This step is required to make sure that Datadog has the necessary permissions to create partner event sources in your AWS account.

Configure Datadog EventBridge integration

1. In the Datadog console, choose Integrations and search for Amazon EventBridge.

2. Choose the Amazon EventBridge integration tile.

3. On the EventBridge integration page, choose Configuration to set up a new EventBridge connection.

4. In the configuration dialog box, under your AWS account ID, choose Add Event Source. Then, include the following parameters:
   For Event Source Name, enter a name for the EventBridge event bus where Datadog sends the alerts.
   For AWS Region, select the Region for EventBridge integration.
   Select Create New Event Bus. This allows Datadog to create the required EventBridge event bus automatically.

5. Choose Save. 

Note: After you complete the setup, Datadog creates a partner event source in EventBridge in the selected AWS account and Region. The event source name follows the format: aws.partner/datadog.com/<event-source-name>

Validate the Datadog EventBridge partner event source

After you configure the Datadog EventBridge integration, make sure that the Datadog integration role provisions the required partner event source automatically in EventBridge. This setup is required for Datadog to deliver the alert notification to EventBridge. Complete the following steps:

1. Open the EventBridge console in the account and Region where you created the EventBridge integration in Datadog.

2. In the navigation pane, under Integration, choose Partner event sources.
   You can see the Datadog integration under Partner event sources with the Active status, as shown in the following screenshot:

Configure Datadog monitors to send an alert notification to EventBridge

To configure your existing Datadog Monitors to send alert notifications to the EventBridge integration, create a notification rule:

1. In the Datadog console, choose Monitors. Then, choose Manage Monitors.

2. Select the monitors that correspond to your critical application alerts.

3. Choose Edit.

4. In the Configure notifications & automations section, choose Create a notification rule.

5. In the New Monitor Notification Rule configuration wizard, complete the following parameters:
   For Match notifications with specific tags, select the appropriate tag for the target Datadog monitor.
   For Choose routing conditions and recipients, under Recipient, select the Datadog EventBridge event bus with the format @awseventbridge-<integration-name>.
   For Name Rule, enter a name for the monitor notification rule.
   Choose Create Rule.

Create a custom EventBridge event bus to ingest transformed events

Incident Detection and Response uses a custom event bus to ingest transformed events that Datadog sends. To create a custom event bus that can ingest the transformed events, complete the following steps:

1. Open the EventBridge console.

2. In the navigation pane, under Buses, choose Event buses.

3. Under Custom event bus, choose Create event bus. Then, enter a name and description for your event bus.

4. Choose Create to create the event bus.
   You can find this event bus on the Event buses page.

Create a Lambda function for Datadog payload transformation

Complete the following steps to create a Lambda function to transform the events from the Datadog partner event bus. The Lambda function transforms the Datadog alert payload and sends the events to the custom event bus that Incident Detection and Response uses:

1. Open the Lambda console.

2. In the navigation pane, choose Functions.

3. Choose Create function.

4. On the Create function page, choose Author from scratch.

5. For Function name, enter a name for your function. For example, Datadog-AWSIncidentDetectionResponse-LambdaFunction.

6. For Runtime, select the appropriate Lambda runtime.
   Note: The Lambda function code in this guide was tested for Python 3.13.

7. Choose Create function.

Your next step is to set up this function to transform events between the partner event bus and the custom event bus that you created. Incident Detection and Response requires the following JSON key-value pair in the alert payload. The Lambda function transforms the alert to include the required JSON key-value pair:

{
  "detail-type": "ams.monitoring/generic-apm",
  "source": "GenericAPMEvent",
  "detail": {
    "incident-detection-response-identifier": "Your alarm name from your APM"
  }
}

1. In the Lambda console, choose Functions, and then select the function that you created.

2. On the Code tab, copy and paste the following code to replace the default Lambda function. Note: Replace Datadog-AWSIncidentDetectionResponse-EventBus with the custom event bus that you created for AWS Incident Detection and Response. The code extracts the monitor name from the Datadog alert payload to use as the incident identifier.

import logging
import json
import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Change the EventBusName to the custom event bus name you created previously or use your default event bus which is called 'default'.
# Example 'Datadog-AWSIncidentDetectionResponse-EventBus'
EventBusName = "Datadog-AWSIncidentDetectionResponse-EventBus" 

def lambda_handler(event, context):
    # Set the event["detail"]["incident-detection-response-identifier"] value to the name of your alert that is coming from your APM. Each APM is different and each unique alert will have a different name. 
    # Replace the dictionary path, event["detail"]["meta"]["monitor"]["name"], with the path to your alert name based on your APM payload. 
    # This example is for finding the alert name for Datadog.
    event["detail"]["incident-detection-response-identifier"] = event["detail"]["meta"]["monitor"]["name"] 
    logger.info(f"We got: {json.dumps(event, indent=2)}")
    
    client = boto3.client('events')
    response = client.put_events(
     Entries=[
              {
               'Detail': json.dumps(event["detail"], indent=2),
               'DetailType': 'ams.monitoring/generic-apm', # Do not modify. This DetailType value is required.
               'Source': 'GenericAPMEvent', # Do not modify. This Source value is required.
               'EventBusName': EventBusName # Do not modify. This variable is set at the top of this code as a global variable. Change the variable value for your eventbus name at the top of this code.    
             }
     ]
    )
    print(response['Entries'])

3. Choose Deploy.

To configure the Lambda function to send events to the custom event bus, you must grant the PutEvents permission to the Lambda execution role. Complete these steps:

1. In the Lambda console, under Functions, select the function that you created.

2. On the Configuration tab, choose Permissions.

3. Select the execution role, and then choose Edit.

4. Under Existing role, select the hyperlinked role name to open the execution role in the IAM console.

5. In the IAM console, under Permissions policies, select the existing policy name for the function. In the navigation pane, choose Policies.

6. Choose Edit.

7. On the Policy editor page, choose Add new statement.
   The policy editor adds a new blank statement.

8. Copy and paste the following policy to replace the blank statement. For Resource, enter the Amazon Resource Name (ARN) of the custom event bus that you created.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSIncidentDetectionResponseEventBus",
            "Effect": "Allow",
            "Action": "events:PutEvents",
            "Resource": "arn:aws:events:{region}:{accountId}:event-bus/Datadog-AWSIncidentDetectionResponse-EventBus"
        }
    ]
}

9. Confirm that the required permission is added to the role.

10. Choose Set this new version as the default, and then choose Save changes.

Create an EventBridge rule to route Datadog alerts

Create an EventBridge rule to send the events from the Datadog partner event bus to the Lambda function for transformation:

1. Open the EventBridge console.

2. In the navigation pane, choose Rules.

3. For Event bus, select the Datadog partner event bus. Example: aws.partner/datadog.com/<eventsource-id>

4. Choose Create Rule. Then, complete the following parameters:
   For Name, enter a name for your rule.
   For Description, enter a description for your rule.

5. Choose Next.

6. For Event source, select AWS events or EventBridge partner events. 

7. For Event pattern, select Use pattern form.

8. For Event source, choose EventBridge partners.

9. For Partner, select Datadog.

10. For Event type, select All Events.

11. Choose Next.

12. In the Select target(s) page, for Target types, select AWS service.

13. For Select a target, select Lambda function.

14. For Function, select the function that you created.

15. Choose Next.

Test the integration

Complete the following steps to verify that the integration works:

1. To trigger a test alert in the Datadog console, modify the monitor threshold or use the Test Notification feature available under the configured Datadog monitor.

2. Check the Lambda function’s Amazon CloudWatch Logs to verify that the function received and processed the Datadog alert.

3. Check the CloudWatch metrics for the event bus. Verify that the events are being delivered to the custom event bus that you created.

If your integration is successful, then Datadog alerts appear in the Lambda function logs with the full event payload. Also, EventBridge publishes the events to the custom event bus with the required field incident-detection-response-identifier.

Troubleshoot common issues

Datadog partner event source doesn’t appear in EventBridge

Symptoms:

• Partner event source isn’t visible in the EventBridge console.

• The status of the Datadog partner event source shows as Pending instead of Active.

Possible causes:

• The Datadog IAM role doesn’t have the required permissions.

• You didn’t fully configure AWS integration in Datadog.

• You selected the incorrect Region during setup.

Resolution steps:

1. Verify that the Datadog AWS integration is active and properly configured.

2. Check that the IAM role that Datadog created has the necessary EventBridge permissions.

3. Make sure that you’re checking the correct Region where you configured the integration.

4. Wait a few minutes for the partner event source to appear after configuration.

5. If the issue persists, then recreate the EventBridge integration in Datadog.

Lambda function doesn’t receive the events

Symptoms:

• No logs appear in CloudWatch for the Lambda function.

• EventBridge rule shows no invocations.

Possible causes:

• You didn’t configure the EventBridge rule properly.

• The event pattern matching is incorrect.

• The Lambda function doesn’t have the required permissions.

Resolution steps:

1. Verify that the EventBridge rule targets the correct Lambda function.

2. Make sure that you turned on the rule and the rule is active.

3. Make sure that you configured the correct trigger for the Lambda function.

4. Test the EventBridge rule with the AWS Command Line Interface (AWS CLI) or console.

5. Review CloudWatch metrics for the EventBridge rule to confirm that EventBridge processes the events.

Datadog monitors don’t send notifications to EventBridge

Symptoms:

• Datadog triggers the monitor alerts, but no events appear in EventBridge.

Possible causes:

• You configured the notification rule incorrectly.

• Make sure that the EventBridge integration name is accurate.

• The monitor tags don't match notification rule criteria.

Resolution steps:

1. Verify that the notification rule recipient format looks like the following: @awseventbridge-<integration-name>

2. Check that monitor tags match the notification rule criteria.

3. Test the notification with Datadog's Test Notification feature.

4. Review Datadog's integration logs for any error messages.

5. Make sure that the EventBridge integration is active and properly connected.

For additional support, see AWS Support or contact your Technical Account Manager (TAM).

Conclusion

You can integrate Incident Detection and Response with Datadog to create a proactive monitoring and response system for your critical workloads on AWS. This integration uses your existing Datadog monitoring investment while adding AWS expertise in incident response. Your configuration that routes Datadog alerts through EventBridge and Lambda automatically escalates critical alerts to the AWS Incident Detection and Response team for rapid resolution.

To get started with AWS Incident Detection and Response, submit the Contact us form or contact your AWS account manager.

---

About the author

Jason Wei-Lun Hsia

Jason Wei-Lun Hsia is a Senior Specialist Solutions Architect at AWS based in Sydney. He specializes in cloud operations and security. Jason helps customers use AWS capabilities to design and operate secure scalable solutions to deliver measurable business outcomes. He’s passionate about building effective solutions to manage cloud environments at scale.