Glue Pyspark create table in Redshift
I'm following documentation from :
- https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-redshift-readwrite.html
- https://github.com/spark-redshift-community/spark-redshift
My code:
Logs:
I am getting these timeout messages until job reaches it's timeout threshold and fails. Is that IP from log my internal Redshift Serverless address?
Am I missing something?
I would appreciate any help.
- Mais recentes
- Mais votos
- Mais comentários
Timeouts could be due to multiple reasons. The most probably reason in my opinion the security group rules not allowing the EMR clusters to reach the Redshift clusters. Check the security group and network ACL rules for the resource you are trying to access. Make sure that the rules allow inbound and outbound traffic for the appropriate protocols and port ranges.
To get the IP address of a Redshift serverless cluster, you can use the DescribeClusters action of the Amazon Redshift API.
Here is an example of how to use the DescribeClusters action to get the IP address of a Redshift serverless cluster using the AWS CLI:
aws redshift describe-clusters --cluster-identifier my-serverless-cluster --query 'Clusters[*].Endpoint.Address'
Let me know if you still face any issues.
Conteúdo relevante
AWS OFICIALAtualizada há 2 anos- AWS OFICIALAtualizada há 3 anos
Timeouts are one of the symptoms for permissions. Check if the IAM user/role running the Glue Job has access to Redshift or not.