Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
In AWS Glue jobs, within the Targets node, I am unable to see the data types such as struct, array or map while changing the schema. Does AWS Glue not support these data types?
1
answers
0
votes
171
views
asked 20 days agolg...
I have the the data of the access logs of the users in opensearch index, this data track the information of the users like at what time users access the door and got the arrival, but one user can...
0
answers
0
votes
277
views
asked 21 days agolg...
Hi team,
Our organization is embarking on a project to design a shared data model where we can efficiently store and manage a large payload.
Multiple lines of business will be both **updating this...
1
answers
0
votes
330
views
asked 2 months agolg...
I have a MySQL database in AWS RDS. I want to import bulk data in to table in database. which contains thousands of rows.
I need to do it from my website with also deployed in AWS. Web is developed...
1
answers
0
votes
240
views
asked 6 months agolg...
I am building a data pipeline to Load data into Redshift from an S3 data lake.
Data are stored in Parquet format on S3 and I would like to load them into the respective Redshift tables using an AWS...
1
answers
0
votes
431
views
asked 6 months agolg...
I want to add Confluent Cloud Apache Kafka as a Data source in AWS ETL job to read data stream from Kafka topic.
I created a cluster, topic, AWS SQS source connector and AWS S3 sink connector in...
1
answers
0
votes
343
views
asked 6 months agolg...
Don't see any data when Migrating existing and changes data from MYSQL to S3 using DMS and Kinesis.lg...
I am using DMS and Kinesis DataStream and Delivery stream to migrate existing and changes from MYSQL to S3 bucket. But I don't see the data is coming in S3. I specified the schema name and table name...
0
answers
0
votes
186
views
asked 6 months agolg...
Everyday a new emr cluster span up and terminated after completing the step job. Checking the cloudtrail, seems a Data Pipeline created it. I am not sure how to get more details like who created, what...
2
answers
1
votes
263
views
asked 6 months agolg...
I HAVE MULTIPLE CSVS ABOUT A SINGLE PATIENT AND I WOULD LIKE TO KNOW HOW DO I COMBINE ALL THE CSVS BECAUSE ALL THE COLUMNS INSIDE THE CSVS MAKE UP AN ALL THE INFORMATION FOR ONE PATIENT. THE CSV'S ARE...
1
answers
0
votes
414
views
asked 8 months agolg...
I am getting following error in my data pipeline
`We currently do not have sufficient t1.micro capacity in the Availability Zone you requested (ap-southeast-2a). Our system will be working on...
2
answers
0
votes
277
views
asked 8 months agolg...
Im trying to delete all the data pipeline that shows up in
```
aws datapipeline list-pipelines
```
in one go. How do I do that using aws-cli?
1
answers
0
votes
317
views
asked 9 months agolg...
I want to get the notification on Failed DataPipelines through event bridge only when they fail. I used the following event pattern but this didn't work. It can be done through the data pipeline SNS...
2
answers
0
votes
327
views
asked 9 months agolg...