Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Questions tagged with AWS Data Pipeline

AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.

Content language: English

Filter questions
Select tags to filter
Sort by
Sort by most recent
Filter Questions by:

Browse through the questions and answers listed below or filter and sort to narrow down your results.

86 results

AWS Glue S3 to Redshift ETL job - ignored `preactions` and `...

I am building a data pipeline to Load data into Redshift from an S3 data lake. Data are stored in Parquet format on S3 and I would like to load them into the respective Redshift tables using an AWS ...
AWS Data PipelineExtract Transform & Load DataAmazon RedshiftAWS Glue
1
answers
1
votes
1.8K
views
asked 2 years ago

AWS Glue Connection with Customer managed Apache Kafka Data ...

I want to add Confluent Cloud Apache Kafka as a Data source in AWS ETL job to read data stream from Kafka topic. I created a cluster, topic, AWS SQS source connector and AWS S3 sink connector in Con...
AWS Data PipelineAWS GlueAWS DataSyncAmazon Managed Streaming for Apache Kafka (Amazon MSK)Extract Transform & Load Data
1
answers
0
votes
991
views
asked 2 years ago

Don't see any data when Migrating existing and changes data ...

I am using DMS and Kinesis DataStream and Delivery stream to migrate existing and changes from MYSQL to S3 bucket. But I don't see the data is coming in S3. I specified the schema name and table name...
Amazon KinesisMySQLAWS Database Migration ServiceAWS Data Pipeline
0
answers
0
votes
243
views
asked 2 years ago

Need more guidance on how to check the data pipeline objects

Everyday a new emr cluster span up and terminated after completing the step job. Checking the cloudtrail, seems a Data Pipeline created it. I am not sure how to get more details like who created, what...
Accepted AnswerAmazon EMRAWS Data Pipeline
2
answers
1
votes
542
views
asked 2 years ago

HOW DO I COMBINE MULTIPLE CSVS INTO ONE

I HAVE MULTIPLE CSVS ABOUT A SINGLE PATIENT AND I WOULD LIKE TO KNOW HOW DO I COMBINE ALL THE CSVS BECAUSE ALL THE COLUMNS INSIDE THE CSVS MAKE UP AN ALL THE INFORMATION FOR ONE PATIENT. THE CSV'S ARE...
Extract Transform & Load DataAWS Data PipelineAWS Glue
1
answers
0
votes
760
views
asked 2 years ago

Unable to create EC2 Resource

I am getting following error in my data pipeline `We currently do not have sufficient t1.micro capacity in the Availability Zone you requested (ap-southeast-2a). Our system will be working on provisi...
AWS Data Pipeline
2
answers
0
votes
454
views
asked 2 years ago

delete all data pipeline using aws cli

Im trying to delete all the data pipeline that shows up in ``` aws datapipeline list-pipelines ``` in one go. How do I do that using aws-cli?
AWS Data PipelineAWS Command Line Interface
1
answers
0
votes
572
views
asked 2 years ago

Event Bridge Notification for the Failed Datapipelines

I want to get the notification on Failed DataPipelines through event bridge only when they fail. I used the following event pattern but this didn't work. It can be done through the data pipeline SNS s...
AWS Data PipelineAmazon EventBridge
2
answers
0
votes
618
views
asked 2 years ago

How to build a data pipeline with one api gateway?

We have one API gateway that receive data 24x7 of gzipped meter data and the data come in concurrently(some times 5000 posts per second, sometimes not much), we are sure the compressed data won't excc...
Amazon API GatewayAmazon KinesisAWS Data PipelineAWS LambdaAmazon Timestream
1
answers
1
votes
635
views
asked 2 years ago

Errors given when specifying permissions for a data pipeline

I'm trying to set up an AWS Data Pipeline so I can clone large huggingface repo's to S3. I'm encountering issue's when creating the permissions policy to use with a role for my data pipeline. [I'm a...
Accepted AnswerAWS Identity and Access ManagementAWS Data PipelineIAM PoliciesSecurity, Identity, & Compliance
2
answers
0
votes
587
views
asked 3 years ago

http.client.RemoteDisconnected at the end of a SageMaker tra...

Hello. I'm running SageMaker training jobs through a library called ZenML. The library is just there as an abstraction layer, so that when I return the artifacts gets automatically saved to S3. The li...
Amazon S3 GlacierMachine Learning & AIAmazon SageMakerAWS Data Pipeline
0
answers
0
votes
211
views
asked 3 years ago

How can I test connection to legacy DB using GlueContext of ...

Hi, I'm running a data pipeline from legacy DB(oracle) to Redshift using AWS Glue. I want to test the connection to the legacy DB before executing ETL without test query in working python script. as-...
AWS GlueAWS Data PipelineExtract Transform & Load Data
1
answers
0
votes
446
views
asked 3 years ago
  • 1
  • 2
  • 3
  • 4
  • 5
  • •••
  • 8
  • Page size
    12 / page