How can I receive Amazon SNS notifications when my AWS Glue job changes states?

4 minute read
0

I want to receive a notice when an AWS Glue extract, transform, and load (ETL) job succeeds, fails, times out, or stops.

Short description

To receive a notice when your AWS Glue job changes status, complete the following steps:

  1. Create an Amazon Simple Notification Service (Amazon SNS) topic. The SNS topic sends messages to subscribing endpoints or clients.
  2. Create an Amazon EventBridge event rule for AWS Glue job state changes that you want to monitor.

Resolution

Note: When your job changes status, AWS Glue produces events. For more information, see AWS Glue job run statuses.

Create and subscribe to an Amazon SNS topic

1.    Open the Amazon SNS console.

2.    In the navigation pane, choose Topics.

3.    Choose Create topic.

4.    For Type, choose Standard.

5.    Enter a Topic name. The Display name field is optional.

6.    Choose Create topic.

7.    In the navigation pane, choose Subscriptions.

8.    Choose Create subscription.

9.     For Details, complete the following fields:

        For Topic ARN, choose the ARN of the topic that you created.

        For Protocol, choose Email.

        For Endpoint, enter the email address that you want to send notifications to.

10.    Choose Create subscription.

In the subscription confirmation email that you receive, choose Confirm subscription. Note the SNS topic that you created. Use this topic when you create the EventBridge rule.

Create an EventBridge event rule

1.    Open the EventBridge console.

2.    In the navigation pane, choose Rules, and then choose Create rule.

3.    Enter a Name for your rule. Keep the default Event bus and Rule type settings, and then choose Next.

4.    For Creation method, choose Custom pattern (JSON editor).

5.    In the Event pattern box, enter the following example event pattern to capture AWS Glue state change events:

{  
    "detail-type": [  
        "Glue Job State Change",  
        "Glue Job Run Status"  
    ],  
    "source": [  
        "aws.glue"  
    ],  
    "detail": {  
        "state": [  
            "FAILED",  
            "SUCCEEDED",  
            "TIMEOUT",  
            "STOPPED",  
            "RUNNING",  
            "STARTING",  
            "STOPPING"  
        ]  
    }  
}

Tip: If you don't want to receive a notice for specific status changes, then remove the corresponding parameter, such as RUNNING, from the event pattern.

6.    Choose Next.

7.    For Target types, choose AWS service.

8.    For Select a target, choose SNS topic.

9.    For Topic, choose the name of the SNS topic that you created.

10.    Choose Next

11.    (Optional) Enter tags for the rule, and then choose Next.

12.    Choose Create rule.

For more information, see Creating event patterns and Automating AWS Glue with Amazon CloudWatch Events.

Test your configuration

To test the event rule and SNS topic, run an AWS Glue job. After the job changes to the state that you specify in the event rule, confirm that you receive an email notification.

Important: If the rule isn't running, then make sure that the event pattern is correct.

Example

You can capture Failed Job Run events and send the event to an SNS topic. The following is an example event for a failed Glue job:

{  
    "version": "0",  
    "id": "abcdef01-1234-5678-9abc-def012345678",  
    "detail-type": "Glue Job State Change",  
    "source": "aws.glue",  
    "account": "123456789012",  
    "time": "2017-09-07T06:02:03Z",  
    "region": "us-west-2",  
    "resources": [],  
    "detail": {  
        "jobName": "MyJob",  
        "severity": "ERROR",  
        "state": "FAILED",  
        "jobRunId": "jr_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",  
        "message": "JobName:MyJob and JobRunId:jr_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef failed to run with exception Role arn:aws:iam::123456789012:role/Glue_Role should be given assume role permissions for Glue Service."  
    }  
}

To parse all failed jobs for AWS Glue, complete the preceding steps.

Related information

How can I use an AWS Lambda function to receive SNS alerts when an AWS Glue job fails a retry?

How can I automatically start an AWS Glue job when a crawler run completes?

Events from AWS services

AWS OFFICIAL
AWS OFFICIALUpdated 9 months ago
3 Comments

Hey, I only have access to using a specific event bus created for my project and not the 'default' one. Would this cause a problem? Because I am not receiving email notifications

sg03
replied 6 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 6 months ago
profile picture
replied 4 months ago