Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hi All,
We have multiple sql servers installed on windows on prem environment, now plan is migrate those to AWS RDS.. We are planning to us below method.
1) Native import/Export using S3 and the...
1
answers
0
votes
533
views
asked 2 years agolg...
I'm a beginner in tasks of date engineer.
I have a taks to create a data lakehouse and i'm tryding undestand how to do it using these tools: DMS, S3, Glue and Hudi.
I already created a simple data...
1
answers
0
votes
666
views
asked 2 years agolg...
Hello! I am looking for an equivalent to this solution that MIcrosoft has flaunted called IDP intelligent data platform, it is governance + operations + analytics in one. they flaunt synapse with aml...
0
answers
0
votes
141
views
asked 2 years agolg...
I am trying to build a data pipeline for a data engineering project With the help of S3, Glue, Athena, etc., I am stuck when setting up glue crawler for indexing over data. Even I set up the role...
1
answers
2
votes
1834
views
asked 2 years agolg...
hi all,
I have a Dag that I schedule to run every quarterly using: **'schedule_interval='@quarterly'**, but for some reason it runs every day.
I don't see the run in managed airflow UI but the...
1
answers
0
votes
652
views
asked 2 years agolg...
My aws account got hacked and my lightsail instace deleted- is there any recover option for that instance -please help
Now we got our account back but the instance is no more .And i don't have any...
1
answers
0
votes
640
views
asked 2 years agolg...
Using AWS Pydeequ in databricks I am performing Data Quality checks. When I run this below mentioned code it provide only metrics results as my output (like Check_level, check_status, constraint,...
0
answers
0
votes
170
views
asked 2 years agolg...
As a publisher of data set on AWS Data Exchange, we would like the data asset downloaded by customers on their server from AWS Data Exchange to be technically un-usable when they terminate the...
1
answers
0
votes
303
views
asked 2 years agolg...
Would like to know what would be the way to migrate per user data currently hosted at on-prem vdi to AWS workspaces? Will this be a case to case migration since the on-prem VDI data might be only...
3
answers
0
votes
509
views
asked 2 years agolg...
I am setting up a new data lake and have been tasked with creating the master data tables in the data bricks delta lake component. I'm trying to do this in a use-case agnostic way (or as agnostic as...
1
answers
0
votes
406
views
asked 2 years agolg...
How can DMS do more complex mapping that is not simply 1-1 from source to target?
For example, if you need to check for data across multiple tables for some business logic and based on those results,...
2
answers
0
votes
799
views
asked 2 years agolg...
AWS DataPipeline Is Failed for EMrClusterBackup with Error Resource Not Healthy:- JobFlow retiredlg...
AWS DataPipeline Is Failed for EMrClusterBackup with Error Resource Not Healthy:- JobFlow retired
1
answers
0
votes
669
views
asked 2 years agolg...