How do I receive Amazon SNS notifications when my AWS Glue job changes states?

Short description

To receive a notice when your AWS Glue job changes states, complete the following steps:

1. Create an Amazon Simple Notification Service (Amazon SNS) topic. The SNS topic sends messages to subscribed endpoints or clients.
2. Create an Amazon EventBridge event rule for the AWS Glue job state changes that you want to monitor.

Resolution

Note: When your job changes states, AWS Glue produces events. For more information, see AWS Glue job run statuses.

Create and subscribe to an Amazon SNS topic

Complete the following steps:

1. Open the Amazon SNS console.
2. In the navigation pane, choose Topics.
3. Choose Create topic.
4. For Type, choose Standard.
5. For Topic name, enter a name. The Display name field is optional.
6. Choose Create topic.
7. In the navigation pane, choose Subscriptions.
8. Choose Create subscription.
9. For Details, complete the following fields:
   For Topic ARN, choose the ARN of the topic that you created.
   For Protocol, choose Email.
   For Endpoint, enter the email address that you want to send notifications to.
10. Choose Create subscription.

In the subscription confirmation email that you receive, choose Confirm subscription. Note the SNS topic that you created. Use this topic when you create the EventBridge rule.

Create an EventBridge event rule

Complete the following steps:

1. Open the EventBridge console.
2. In the navigation pane, choose Rules, and then choose Create rule.
3. For Name, enter a name for your rule. Keep the default Event bus and Rule type settings, and then choose Next.
   Note: Only the default bus can receive events from AWS services, but the default bus can forward events to other buses.
4. For Creation method, choose Custom pattern (JSON editor).
5. In the Event pattern box, enter the following example event pattern to capture AWS Glue state change events:

   {   "detail-type": [    
           "Glue Job State Change",    
           "Glue Job Run Status"    
       ],    
       "source": [    
           "aws.glue"    
       ],    
       "detail": {    
           "state": [    
               "FAILED",    
               "SUCCEEDED",    
               "TIMEOUT",    
               "STOPPED",    
               "RUNNING",    
               "STARTING",    
               "STOPPING"    
           ]    
       }    
   }

   Tip: If you don't want to receive notices for a specific state change, then remove the corresponding parameter, such as SUCCEEDED, from the event pattern.
6. Choose Next.
7. For Target types, choose AWS service.
8. For Select a target, choose SNS topic.
9. For Topic, choose the name of your SNS topic.
10. Choose Next.
11. (Optional) Enter tags for the rule, and then choose Next.
12. Choose Create rule.

For more information, see Creating event patterns and Automating AWS Glue with Amazon CloudWatch Events.

Test your configuration

To test the event rule and SNS topic, run an AWS Glue job. After the job changes to a state that you specified in the event rule, confirm that you receive an email notification.

Important: If the rule isn't running, then make sure that the event pattern is correct.

Example

You can capture Failed Job Run events and send the events to an SNS topic. The following is an example event for a failed Glue job:

{   "version": "0",    
    "id": "abcdef01-1234-5678-9abc-def012345678",    
    "detail-type": "Glue Job State Change",    
    "source": "aws.glue",    
    "account": "123456789012",    
    "time": "2017-09-07T06:02:03Z",    
    "region": "us-west-2",    
    "resources": [],    
    "detail": {    
        "jobName": "MyJob",    
        "severity": "ERROR",    
        "state": "FAILED",    
        "jobRunId": "jr_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",    
        "message": "JobName:MyJob and JobRunId:jr_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef failed to run with exception Role arn:aws:iam::123456789012:role/Glue_Role should be given assume role permissions for Glue Service."    
    }    
}

To parse all failed jobs for AWS Glue, complete the preceding steps.

Related information

How can I use a Lambda function to receive SNS alerts when an AWS Glue job fails a retry?

How can I automatically start an AWS Glue job when a crawler run completes?

Events from AWS services