Questions tagged with AWS Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hi There,
I'm sourcing data from dynamo db and residing into s3 bucket using AWS data pipeline. I have been running this pipeline once in a week to get up-to date the records from dynamo db table....
**Transfer data from RDS MySQL to S3 bucket
**
I set up a data pipeline to transfer data from RDS Mysql to S3 Bucket, where before thet I set up the RDS mysql database with username and password....
asked 2 years ago
Hi,
I have a use case where I am fetching data on certain items (unique itemID) multiple times a day (identified by day_BatchTime) and storing them on a dyanmoDB. My composite primary key consists...
Hello
I am working on serverless application, and i was looking for something handle the frontend part and I found Honeycode since its native and its codeless.
So is it possible to my Honeycode app...
Hi,
I have a problem in that I make heavy use of EMRs, and I orchestrate their use with Data Pipeline - multiple daily runs are automated and EMRs are launched and terminated on conclusion.
However,...
I'm trying to create a data pipeline to export Dynamodb data to S3, but after following the online guide to the letter, the DataPipelineDefaultResourceRole isn't in the dropdown referred to above, the...
## Main problem
I understand that is no need to add Auto Scaling to an EMR cluster launched by Data Pipeline. Instead, we can specify the **capacity up-front** and it will be used for the duration of...
Currently I use the [CreateExportTask](https://docs.aws.amazon.com/ko_kr/AmazonCloudWatchLogs/latest/APIReference/API_CreateExportTask.html) API to backup my log data.
The problem is, exported data...
I'm confused about how staging of an S3Datanode is billed when done as part of a ShellCommandActivity with the 'stage' property set to true (i.e. I do not have CSV data and am not using a...
I have a few questions regarding data preparation for Forecast.
I have a dataset with about 3,000 item_id's, the data is recorded on weekdays only (no row for weekends/holidays), and the forecast...
I have an etl pipeline that loads json data from a source bucket, runs an etl job with bookmarking enabled, and writes as parquet to a target bucket.
I'd like to ensure that the target bucket never...
I'm working on a step function state machine and can create lambdas in python and node to update an existing item in ddb. However, I can't seem to find any examples with service integrations AND...