AWS Glue Visual Studio - Redshift Target Node

0

Hey all, I have been trying to perform a simple S3 to redshift data push using an S3 source node and a Amazon Redshift target node. I have been getting errors such as 'Failed to connect to IP Address', 'py4j.protocol.Py4JJavaError: An error occurred while calling o101.getSink. : com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

Have checked permissions, policies, etc with AWS Admin. Any tips to fix this? Note- I am attempting this in a glue job which also has several other nodes for transforms, etc.

sg03
asked 5 months ago341 views
4 Answers
0
Accepted Answer

In my case, I had 2 connections on my glue job and I only required the JDBC connection to redshift. I had an additional network connection which wasn't required since everything was on the same network

sg03
answered 5 months ago
0

Notice that Glue is trying to use the MySQL driver driver, which will use a different port and connectivity.
Doublecheck you are using the right connection and that is well defined (or point the target directly to Redshift without a connection/table)

profile pictureAWS
EXPERT
answered 5 months ago
  • Hey Gonzalo thanks very much for your response. Currently, I have picked the appropriate connection to my database in redshift, and also picked the schema and the table to ingest into in redshift. What do you mean by point to redshift without the connection/table? Where would the data end up going then?

0

Double check your Glue connection was set up for Redshift and not generic JDBC (as was pointed out, your connection thinks it is MySQL). Also as a reminder, when using Glue with Redshift we also strongly recommend using Glue 4.0 as the newest connectors are exponentially better than the old.

For running queries before or after a data load I recommend using the redshift_connector python library on pypi via --additional-python-modules.

AWS
Zach
answered 5 months ago
0

'Communications link failure' typically indicates that the Glue job is unable to reach the source and/or destination target.

Check the vpc setup, like the subnets and the security groups, refer the below documentation for reference

[+] Redshift connections - Set up Amazon VPC - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-redshift-home.html#aws-glue-programming-etl-redshift-config-vpc

AWS
SUPPORT ENGINEER
answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions