Glue Pyspark create table in Redshift

Question

I'm following documentation from :
1. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-redshift-readwrite.html
2. https://github.com/spark-redshift-community/spark-redshift

My code:
![Enter image description here](/media/postImages/original/IMEbE35B_QRJSB3km3UK5VcQ)

Logs:
![Enter image description here](/media/postImages/original/IMvorRP4RaSIa2_4CYIGmOww)

I am getting these timeout messages until job reaches it's timeout threshold and fails. 
Is that IP from log my internal Redshift Serverless address?

Am I missing something?

I would appreciate any help.

Answer

Timeouts could be due to multiple reasons. The most probably reason in my opinion the security group rules not allowing the EMR clusters to reach the Redshift clusters. Check the security group and network ACL rules for the resource you are trying to access. Make sure that the rules allow inbound and outbound traffic for the appropriate protocols and port ranges.

To get the IP address of a Redshift serverless cluster, you can use the DescribeClusters action of the Amazon Redshift API.

Here is an example of how to use the DescribeClusters action to get the IP address of a Redshift serverless cluster using the AWS CLI:

```
aws redshift describe-clusters --cluster-identifier my-serverless-cluster --query 'Clusters[*].Endpoint.Address'
```
Let me know if you still face any issues.

Glue Pyspark create table in Redshift

相關內容