- Newest
- Most votes
- Most comments
Hello.
Is the subnet where RDS is located a private subnet?
In that case, try setting up an S3 gateway VPC endpoint.
It is possible that there is no route to access S3 after connecting to RDS, resulting in an error.
https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html
Hi Pavel,
Could you check if the IAM role assumed by the Glue ETL job includes permissions to your “b20240516” S3 bucket?
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::b20240516*/*"
]
}
]
}
The only way I managed to reproduce your error was because of lack of permissions.
If you have that policy already, please could you also attach the logs for the scenario "source is S3 and the destination is S3"?
Thank you, Monica_A and Riku_Kobayashi for trying to help. Obviously the problem was that Endpoint is for Service Name = "com.amazonaws.us-east-1.s3" (and there were no other suitable values for choice) but all my S3 buckets were NOT in "us-east-1". After I replaced S3 bucket to another one that is located in "us-east-1" the job run completed successfully. Anyway all I want to say - I hate AWS Glue :-), the simplest things, that should take couple of minutes, take days and additionally you need to fight with different Glue bugs. I would be so happy to work with Azure Data Factory, but unfortunately in current project I need to work with Glue.
Relevant content
- asked a year ago
- asked 2 years ago
- Accepted Answerasked 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
RDS belongs to VPC consisting of two subnets, both of them are Public.
Glue does not have a public IP even if connected to a VPC, so it cannot access S3 by default. So we need to set up an S3 VPC endpoint to be able to access S3. https://docs.aws.amazon.com/glue/latest/dg/start-connecting.html
The Endpoint does exist: Service Name = "com.amazonaws.us-east-1.s3", with Route Table including both public Subnets of the VPC. Endpoint Status is "Available", Type is "Gateway". Route Table "Main" sign = No.