Using boto3 client redshift-data APIs in AWS Glue python shell job gives ConnectTimeoutError error

Question

I am getting "ConnectTimeoutError: Connect timeout on endpoint URL: "https://redshift-data.us-east-1.amazonaws.com/" in AWS Glue python shell job when I am using boto3 clients redshift-data APIs. Below are the boto3 APIs I use in python script.

```
client = boto3.client('redshift-data')

response = client.execute_statement(
                    ClusterIdentifier=redshift_cluster,
                    Database=redshift_db,
                    DbUser=redshift_db_user,
                    Sql=sql,
                    StatementName=stmt_name)

response = client.list_statements(
            MaxResults=2,
            NextToken='',
            RoleLevel=True,
            StatementName=stmt_name,
            Status='ALL'
        )
```
I am not sure why boto3 client is trying to access "https://redshift-data.us-east-1.amazonaws.com/" endpoint. If I run this script local machine (after setting aws secrets in environment vars), it runs successfully. The issue appears only with Glue job.

The IAM role of Glue job has permissions AWSGlueServiceRole, AmazonRedshiftFullAccess, AmazonRedshiftAllCommandsFullAccess and AmazonRedshiftDataFullAccess.

Any body has idea about this?

Answer

According to this [AWS Documentation](https://docs.aws.amazon.com/general/latest/gr/redshift-service.html), we can understand that whenever you try to connect to Redshift programmatically then it will inherently make use of the endpoint depending upon your region.

1) Please do ensure that a connection is attached to your Glue job such that it is able to reach the endpoint. You can add a network connection to your Glue job mentioning the VPC and subnet. Please do ensure that the Glue job has access to reach the redshift endpoint through the subnet mentioned. Please do attach a private subnet with NAT gateway to the Glue job.

2) Make sure that security group attached to Glue job has a self referencing inbound rule.

3) Make sure that the security group of redshift cluster is allowing inbound traffic from the security group of Glue job. If it is not, then add an inbound rule to the redshift cluster's security group.

Please refer this [article](https://aws.amazon.com/premiumsupport/knowledge-center/connection-timeout-glue-redshift-rds/) for more details.

Using boto3 client redshift-data APIs in AWS Glue python shell job gives ConnectTimeoutError error

Relevant content