How to create a AWS Glue Connector for data sources in VMware Cloud on AWS

In this article, I will show you how to create AWS Glue Connections that will provide connectivity between data sources (eg. databases) in VMware Cloud on AWS and AWS Glue.

This becomes useful when you want to use AWS Glue with VMware Cloud on AWS as a data source. AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.

Prerequisites

• VMware Cloud on AWS Software Defined Data Centre
• Data Source that can accept JDBC connections
• AWS Account

Method

1. Open the AWS Console, open the AWS Glue Service
2. . Select Connections from the AWS Glue menu

3. Select Create connection
4. Fill in the Connection properties (only select "Require SSL" if you have configured SSL on the data source)

5. Fill in the Connection access properties. (If you are using AWS Secrets Manager you can can select the secret from the drop down, else you can use username / password)
6. JDBC Driver Class and JDBC Driver S3 Path is optional.

7. Fill in the Network options, this says optional but it is essential if you are using VMware Cloud on AWS
8. Select the correct VPC and Subnet (these will need to have connectivity / be able to route to your VMware Cloud on AWS data source)
9. For the Security Group see step 10

10. To enable AWS Glue to communicate between its components, either select or create a security group with a self-referencing inbound rule for all TCP ports.
11. Add a self-referencing inbound rule to allow AWS Glue components to communicate. Specifically, add or confirm that there is a rule of Type All TCP, Protocol is TCP, Port Range includes all ports, and whose Source is the same security group name as the Group ID

12. Add a rule to for outbound traffic also. Either open outbound traffic to all ports, or create a self-referencing rule of Type All TCP, Protocol is TCP, Port Range includes all ports, and whose Source is the same security group name as the Group ID
13. Select Create connection
14. One the connection is created, we need to test it for connectivity
15. Select the Connection you have just created, select the Actions drop down and select Test connection

16. Select an IAM role, I have used the AWSGlueServiceRole, select Confirm

17. This should take 1 - 2 mins to test connectivity

Now that you have your AWS Glue Connection created, you can start to create AWS Glue Crawlers and AWS Glue Jobs.