Glue Pyspark create table in Redshift
I'm following documentation from :
- https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-redshift-readwrite.html
- https://github.com/spark-redshift-community/spark-redshift
My code:
Logs:
I am getting these timeout messages until job reaches it's timeout threshold and fails. Is that IP from log my internal Redshift Serverless address?
Am I missing something?
I would appreciate any help.
- 최신
- 최다 투표
- 가장 많은 댓글
Timeouts could be due to multiple reasons. The most probably reason in my opinion the security group rules not allowing the EMR clusters to reach the Redshift clusters. Check the security group and network ACL rules for the resource you are trying to access. Make sure that the rules allow inbound and outbound traffic for the appropriate protocols and port ranges.
To get the IP address of a Redshift serverless cluster, you can use the DescribeClusters action of the Amazon Redshift API.
Here is an example of how to use the DescribeClusters action to get the IP address of a Redshift serverless cluster using the AWS CLI:
aws redshift describe-clusters --cluster-identifier my-serverless-cluster --query 'Clusters[*].Endpoint.Address'
Let me know if you still face any issues.
관련 콘텐츠
- 질문됨 6달 전
AWS 공식업데이트됨 2년 전
Timeouts are one of the symptoms for permissions. Check if the IAM user/role running the Glue Job has access to Redshift or not.