Not able to read from raw bucket if I try to access the files within subfolders

0

Here is my IAM role permissions while reading the buckets : "sig": "AllowPermissions", "Effect": "ALLOW" "Action": [ "§3:Getobject" w3:ListBUcket" "§3:Listobjects", "§3:ListobjectsV2" 1. "Resource": [ "Fn: :Sub": "${bucket1}/" "Fn: :SUb": "${bucket2}/" }. "Fn::Sub": "${bucket3} /*" ] I have a method which calls while reading the data from raw bucket using spark reader . def listAllFilesFromSubFolders (spark: Sparksession, s3path: String) : Array[string] = { val hadoopConf = spark. sparkContext. hadoopConfiguration vat filesystem = Filesystem.get (new java.net. URI (spath), hadoopconf) def recursive (path: Path): Array [Path] = { val status = fileSystem. listStatus (path) I val files - status. filter (I_.isDirectory) .map (getPath) val directoryFiles - status. filter (.isDirectory) .map (-getPath) directoryFiles. flatMap(recursive) ++ files val allFiles = recursive (new Path (spath)) allFiles.filter(-getName.startswith("global")).map(_-tostring) } Facing exception while reading above method with this exception : java.io.IOException:com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Copy Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: ; S3 Extended Request ID: /EH99x0PwpoQwyACxw8wv3dFVdPEz2GgqmrMOMf2yvhoI6QL4hzYSw=; Proxy: null), S3 Extended Request ID:j/EH99x0PwpoQCwyACxw8w3dFVdPEz2GgqmrMOMf2yvhOI6QL4hzY5w= at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.list(Jets3tNativeFilesystemStore.java:303) at com.amazon.ws.emr.hadoop.fs.s3n.sNativeFilesystem.liststatus(sNativeFileSystem.java:665) at com.amazon.ws.emr.hadoop.fs.s3n.sNativeFileSystem.listStatus(sNativefileSystem.java:636) at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.listStatus(EmrFilesystem.java:473) at com.ics.utils.FilePathGenerator$.recursive$1(FilePathGenerator.scala:34) at com.ics.utils.FilePathGenerator$.listAllFilesFromSubFolders(FilePathGenerator.scala:40) at $.main(.scala:111) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method. java: 498) at com.amazonaws.services.glue.SparkProcessLauncherPlugin.invoke(ProcessLauncher.scala:48)

  • Hi, please update you question and use quote format </> to make your IAM policy readable by re:Post community.

已提问 1 年前331 查看次数
1 回答
1

Hello,

From the error looks like a copy operation is being performed which is throwing the access denied error. Can you check if you have the below permission in your IAM policy on the required bucket because Copy operation uses below permissions.

s3:GetObject s3:PutObject S3:GetObjectTagging S3:PutObjectTagging

Also, If the object is encrypted using an AWS KMS key, then confirm that your IAM identity/role has the correct permissions to the key. If your IAM identity/role and AWS KMS key belong to the same account, then confirm that your key policy grants the required AWS KMS permissions. Moreover, there should not be explicit deny at bucket policy, please check for that as well.

For most common causes of Access Denied (403 Forbidden) errors in Amazon S3 please visit this link -
https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html

Hope this helps.

AWS
已回答 1 年前
profile pictureAWS
专家
已审核 1 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则