Questions tagged with Data Lakes

Hello, I'm looking to push Salesforce data to a Data Lake. This data lake table needs to hold different versions of the record. I experimented with AppFlow, but I couldn't really get the control I...

Hi folks, Curious to know if it is possible to expose an AWS-hosted Snowflake data lake/warehouse directly using an API Gateway? I found documentation on Snowflake's site which mentions the use of a...

Hi, Is there a way to perform time travel with Athena queries on Apache Hudi tables in a way similar to the one described here [Implement a cdc based upsert in a data lake using Apache Iceberg and...

I am having trouble connecting MWAA to snowflake. I used the MWAA UI to automatically create a VPC, security group, and IAM for my airflow environment. I can not get snowflake to show as an option in...

I am building an ETL pipeline using primarily state machines, Athena, and the Glue catalog. In general things work in the following way: 1. A table, partitioned by "version", exists in the Glue...

I created the tutorial from this...

I'm a beginner in tasks of date engineer. I have a taks to create a data lakehouse and i'm tryding undestand how to do it using these tools: DMS, S3, Glue and Hudi. I already created a simple data...

Hello! I am looking for an equivalent to this solution that MIcrosoft has flaunted called IDP intelligent data platform, it is governance + operations + analytics in one. they flaunt synapse with aml...

I'm investigating and deploying https://docs.aws.amazon.com/solutions/latest/data-lake-solution/welcome.html Looking at the GitHub repo https://github.com/aws-solutions/aws-data-lake-solution it looks...

From time to time I have a csv file coming with single row and it breaks the Glue Crawler because of the at least 2 row requirement to be classified as a CSV. Is there a way I can provide a custom CSV...

We have incoming file with the fixed length field length (.dat). For example: |2|123 |AWS |0505 |3 When Glue Crawler crawles the file, it ignores all the int/long values that have trailing...

Hello, due to the following Step by Step Guide provided by the official AWS Athena user-guide (Link at the End of the question), it should be possible to connect Tableau Desktop to Athena and Lake...