Questions tagged with AWS Lake Formation

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

If I commit an update_table_objects transactions that remove files from the table, is there any garbage collection that will clean them up from S3 later? The closest thing I can find is this article: https://docs.aws.amazon.com/lake-formation/latest/dg/data-compaction.html. However, it seems to only relate to cleaning up objects if a transaction is cancelled.
0
answers
0
votes
36
views
tjtoll
asked 9 days ago
While running Glue I see these arguments passed to job: { 'job_bookmark_option': 'job-bookmark-disable', 'job_bookmark_from': None, 'job_bookmark_to': None, 'JOB_ID': 'j_c8afc16edb1420c2fb878249843e27280db60efcd37b4f6c7c469c4a55a1b5bd', 'JOB_RUN_ID': 'jr_d74caf9a56f744d09ac4d7fd076caa3d8da3cbc5d58f925ea532dc3c7dfcdf32', 'SECURITY_CONFIGURATION': None, 'encryption_type': None, 'enable_data_lineage': None, 'RedshiftTempDir': 's3://aws-glue-assets-myaccount-us-east-1/temporary/', 'TempDir': 's3://aws-glue-assets-myaccount-us-east-1/temporary/', 'JOB_NAME': 'my-job', } I spotted parameter called "enable_data_lineage" . For the next run I set this parameter to True, like this: "--enable-data-lineage true" in Job parameters section. After this my job startup time jumped from 7 seconds to 3 min and 10 seconds. I went to logs to check what's going on and I spotted error messages like this: 2022-12-30 12:27:54,873 WARN [Thread-12] lineage.LineagePersistence$ (LineagePersistence.scala:isCatalogLineageSettingEnabled(99)): Exception occurred while getting catalog lineage settings, lineage for this job run will be disabled com.amazonaws.services.lakeformation.model.InternalServiceException: Received an unexpected Content-Type: text/html', expected one of [application/json]. HTTP status code:503 (Service: AWSLakeFormation; Status Code: 500; Error Code: InternalServiceException; Request ID: ffe50f30-ec28-4648-814a-e267be0453da; Proxy: null) I tried to search for documentation, but no luck.. How to properly set-up this feature? Is there any examples?
1
answers
0
votes
31
views
asked a month ago
i am trying to get give lake formation tag to user /role ,like how we give the same to the data catalog
1
answers
0
votes
24
views
Krishna
asked a month ago
I have manually created an lake formation tag key :classification with tag value :non pii and associated to tag to table columns,here i want use glue. Job using detetect pii and custome code (boto 3 library)to over write the same lake formation tag key : classification with tag value : pii , pls clarify the same by using glue job can we over write detect pii with glue .
0
answers
0
votes
10
views
Krishna
asked a month ago
I am trying to create standard glue external tables with terraform, replacing a number of lake formation governed tables. The original lake governed tables had no table specific permissions granted and where dropped from the control panel. Any attempt to create a standard external table from any means (cli, boto3, glue console, lake formation console, terraform) I receive a `AlreadyExistsException` on that table. When calling the table from the cli or with boto3 it can't find the corresponding table (EntityNotFound or similar) If in either terraform or the control panel the table type is changed to governed, the table is created successfully (so with all the same settings as before i.e. region, path, classification etc). We would like to be able to create these tables as standard external tables, but seem completely unable to with no idea whether this is a bug of some sort (likely in lake formation?) or we're missing something? Any help is appreciated
0
answers
0
votes
19
views
Stephen
asked a month ago
I ran basic sql in Athena to view the catalog table which was created by the glue-crawler (crawler job ended successfully and created the "metadata" catalog table in the "hw-db" db) : `SELECT * FROM "AwsDataCatalog"."hw-db"."metadata" limit 10;` and got the following error: ``` HIVE_UNKNOWN_ERROR: com.amazonaws.services.lakeformation.model.InvalidInputException: Unsupported vendor for Glue supported principal: arn:aws:iam::{...}:root (Service: AWSLakeFormation; Status Code: 400; Error Code: InvalidInputException; Request ID: {...}; Proxy: null) This query ran against the "hw-db" database, unless qualified by the query. ``` any ideas?...
1
answers
0
votes
61
views
Erez
asked 2 months ago
I'm working on deploying to LakeFormation via Terraform. Specifically, granting data location access to a lambda role. I'm getting an error when the role/user I'm deploying with in Terraform isn't an admin on LakeFormation (I haven't tried playing around w/ granular policies on the caller yet). Has anyone come across the same issue and what was the resolution? The caller is a service user which is used by other groups across the org, so I would ideally like to avoid elevating any more of its permissions. Configuration : ``` resource "aws_lakeformation_permissions" "datalake-permissions" { principal = aws_iam_role.lambda-role.arn permissions = ["DATA_LOCATION_ACCESS"] data_location { arn = data.aws_s3_bucket.datalake-bucket.arn } } ``` This is the error : `error creating Lake Formation Permissions (input: { Permissions: ["DATA_LOCATION_ACCESS"], Principal: { DataLakePrincipalIdentifier: "arn:aws:iam::{account_id}:role/lambda_role" }, Resource: { DataLocation: { ResourceArn: "arn:aws:s3:::{my-bucket}" } } }): AccessDeniedException: Resource does not exist or requester is not authorized to access requested permissions.` Also made sure the bucket exists and isn't an issue.
1
answers
0
votes
52
views
asked 2 months ago
Hello, I'm current redeploying a CI/CD pipeline from a Legacy Terraform to Terraform on Cloud. The following error first appeared on the newly migrated pipelines: HIVE_UNKNOWN_ERROR: com.amazonaws.services.lakeformation.model.InvalidInputException: Unable to assume role. Please verify Lake Formation has access to role arn:aws:iam::561######914:role/aws-reserved/sso.amazonaws.com/us-west-2/AWSReservedSSO_AdministratorAccess_0bb#####78e (Service: AWSLakeFormation; Status Code: 400; Error Code: InvalidInputException; Request ID: 73d56a83-6796-4cbe-befb-3e0b4e736773; Proxy: null) After trying to grant permissions manually we oscillated between propagating this error to all databases on the project to retrieving this error to only a few databases. We tried to grant permission through the *Data lake permissions*, with LF-Tags and also with the Databases. But without success. Any idea on what to do?
1
answers
0
votes
53
views
asked 2 months ago
Hi Team, i was trying recently launched security lake, I followed the steps mentioned in the following link, ``` https://docs.aws.amazon.com/security-lake/latest/userguide/getting-started.html#enable-service ``` I am currently using this service for only one account and only one region. i have created admin IAM user and through that I am working. i have created a role "AmazonSecurityLakeMetaStoreManager" and attached policy and sts as mentioned in ``` https://docs.aws.amazon.com/security-lake/latest/userguide/manage-regions.html#iam-role-partitions ``` also i have added this role as Data lake administrator in lake formation ![Image showing role added as admin](/media/postImages/original/IMi2-gQZ4LT4auOQGc5Qy5Bw) i am not getting what actual permission i am missing to give ![image showing when trying to enable security lake](/media/postImages/original/IMT5modkeTQfCHikCssOCx-g) i am getting this error when i am trying to enable security lake through AWS console Any help is appreciable.. Thanks in advance.
1
answers
0
votes
47
views
Shriw
asked 2 months ago
I can't solve this error. I have given sagemaker IAM permission within the lake formation for this table, I have errors also using awswrangles lib within sagemaker notebook what am I doing wrong? Studio: ![Enter image description here](/media/postImages/original/IMB_zL50UyRjKvNJYnta9ZKQ) Notebook: ![Enter image description here](/media/postImages/original/IM9pyYOkXNQ5urTECMAL6row) ![Enter image description here](/media/postImages/original/IM6ihq2P_ZQJ-myFQKB3Uijw)
1
answers
0
votes
34
views
asked 2 months ago
Hello all, i am using AWS account with 12 month free tier. I have crated a IAM user with administrative Access and loged in with that user. After that i create an IAM role with AmazonS3 full access and GlueServiceRole policy. In data lake formation i crated a database with this new role. Now i am creating a glue crawler which will crawl S3 bucket and will store the schema of csv file in the Lake formation database. Everything works well but when i create crawler it gives error saying account <my account id> is denied access. I also used role for this crawler with upper mentioned policies. The image of error is also attached. I will be very thankful for your help. Thank you.![Here is the image of error](/media/postImages/original/IMEJEFTIF0TX69I_gRn8gCyg)
1
answers
0
votes
48
views
asked 2 months ago
I encounter the error when I try to create a new Athena dataset on QuickSight (click on [Validate connection] button); * Error Code: ACCESS_DENIED_TO_RESULT_STAGING_AREA * Error Message: [Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. Access denied when writing to location: s3://<Athena-query-result-bucket-name>/xxxx [Execution ID: xxxxx] Environment: - there're 2 s3 buckets; one is "s3://<Source-data-bucket-name>" for query target, and another one is "s3://<Athena-query-result-bucket-name>" for Athena query workgroup. - Using Lake Formation, and both buckets are registered as datalake location. - After create QuickSight account, I allow access to 2 s3 buckets and Athena via QuickSight account management - On Lake Formation console, I did grant; - QuickSight execution role (aws-quicksight-service-role-v0) access to 2 s3 buckets - QuickSight Group ARN (arn:aws:quicksight:region:accountId:group/default/groupName) access to the raw data access bucket What else should I do to make enable to create Athena data set on QuickSight console? Thank you for your help.
0
answers
0
votes
45
views
AWS
asked 2 months ago