Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hi team,
Our organization is embarking on a project to design a shared data model where we can efficiently store and manage a large payload.
Multiple lines of business will be both **updating this...
1
answers
0
votes
300
views
asked a month agolg...
I have a MySQL database in AWS RDS. I want to import bulk data in to table in database. which contains thousands of rows.
I need to do it from my website with also deployed in AWS. Web is developed...
1
answers
0
votes
216
views
asked 5 months agolg...
I am building a data pipeline to Load data into Redshift from an S3 data lake.
Data are stored in Parquet format on S3 and I would like to load them into the respective Redshift tables using an AWS...
1
answers
0
votes
363
views
asked 5 months agolg...
I want to add Confluent Cloud Apache Kafka as a Data source in AWS ETL job to read data stream from Kafka topic.
I created a cluster, topic, AWS SQS source connector and AWS S3 sink connector in...
1
answers
0
votes
310
views
asked 5 months agolg...
Don't see any data when Migrating existing and changes data from MYSQL to S3 using DMS and Kinesis.lg...
I am using DMS and Kinesis DataStream and Delivery stream to migrate existing and changes from MYSQL to S3 bucket. But I don't see the data is coming in S3. I specified the schema name and table name...
0
answers
0
votes
171
views
asked 5 months agolg...
Everyday a new emr cluster span up and terminated after completing the step job. Checking the cloudtrail, seems a Data Pipeline created it. I am not sure how to get more details like who created, what...
2
answers
1
votes
236
views
asked 5 months agolg...
I HAVE MULTIPLE CSVS ABOUT A SINGLE PATIENT AND I WOULD LIKE TO KNOW HOW DO I COMBINE ALL THE CSVS BECAUSE ALL THE COLUMNS INSIDE THE CSVS MAKE UP AN ALL THE INFORMATION FOR ONE PATIENT. THE CSV'S ARE...
1
answers
0
votes
380
views
asked 7 months agolg...
I am getting following error in my data pipeline
`We currently do not have sufficient t1.micro capacity in the Availability Zone you requested (ap-southeast-2a). Our system will be working on...
2
answers
0
votes
257
views
asked 8 months agolg...
Im trying to delete all the data pipeline that shows up in
```
aws datapipeline list-pipelines
```
in one go. How do I do that using aws-cli?
1
answers
0
votes
288
views
asked 8 months agolg...
I want to get the notification on Failed DataPipelines through event bridge only when they fail. I used the following event pattern but this didn't work. It can be done through the data pipeline SNS...
2
answers
0
votes
305
views
asked 8 months agolg...
We have one API gateway that receive data 24x7 of gzipped meter data and the data come in concurrently(some times 5000 posts per second, sometimes not much), we are sure the compressed data won't...
1
answers
1
votes
353
views
asked 9 months agolg...
I'm trying to set up an AWS Data Pipeline so I can clone large huggingface repo's to S3.
I'm encountering issue's when creating the permissions policy to use with a role for my data pipeline. [I'm...
2
answers
0
votes
296
views
asked 10 months agolg...