- 最新
- 投票最多
- 评论最多
The most likely cause is a misconfiguration in the S3 path, IAM role permissions, or S3 bucket policy. Double check these settings to make sure that your AWS Glue job has the necessary access to write to the S3 bucket.
Here are examples of how each of these should be configured for your AWS Glue job to write to an S3 bucket successfully:
S3 Path:
output_path = "s3://your-bucket-name/your-folder-name/"
IAM Role Permissions:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}
```
S3 Bucket Policy:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::account-id:role/your-glue-role"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::your-bucket-name/*"
}
]
}
```
Hi, thanks for your update, I have my glue-role and has the policy : amazonfullacessS3policy which is: { "Effect": "Allow", "Action": [ "s3:", "s3-object-lambda:" ], "Resource": "*" }
so basically should have all access according to above. The bucket is public "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::007800227665:role/Glue-conexion-XXXX" }, "Action": "", "Resource": "arn:aws:s3:::demo-dwh/" }
so I think it fullfils what you suggested right?
Hi Erick,
Could you specify what the function you're using to perform the write to S3? Are you using a Glue sink function or are you calling a Spark function directly? Also, what transformation steps are you performing? It's possible the error you're seeing is not due to having inadequate permissions. This error is also sometimes seen with converting Glue DynamicFrames into Spark DataFrames and vice versa.
My job has 2 boxes, one the oracle datasource and the other is the the data target S3 bucket , that is all, Perhaps as you mentioned something is missing about spark and dynamicframes, I dont´t really know.
I suspect the issue is that you are trying to write at the root of the bucket, which even if possible is not a good practice. Try something like: 3://bucket_name/my_data/
相关内容
- AWS 官方已更新 3 年前
- AWS 官方已更新 7 个月前
I did change bucket and got now Error Category: UNCLASSIFIED_ERROR; An error occurred while calling o108.pyWriteDynamicFrame. demo-data-raw-migracion-dwh.s3.amazonaws.com