Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
I'm trying to create a data pipeline to export Dynamodb data to S3, but after following the online guide to the letter, the DataPipelineDefaultResourceRole isn't in the dropdown referred to above, the...
1
answers
1
votes
558
views
asked 2 years agolg...
## Main problem
I understand that is no need to add Auto Scaling to an EMR cluster launched by Data Pipeline. Instead, we can specify the **capacity up-front** and it will be used for the duration of...
0
answers
0
votes
145
views
asked 2 years agolg...
Currently I use the [CreateExportTask](https://docs.aws.amazon.com/ko_kr/AmazonCloudWatchLogs/latest/APIReference/API_CreateExportTask.html) API to backup my log data.
The problem is, exported data...
1
answers
1
votes
1562
views
asked 2 years agolg...
I'm confused about how staging of an S3Datanode is billed when done as part of a ShellCommandActivity with the 'stage' property set to true (i.e. I do not have CSV data and am not using a...
0
answers
0
votes
76
views
asked 2 years agolg...
I have a few questions regarding data preparation for Forecast.
I have a dataset with about 3,000 item_id's, the data is recorded on weekdays only (no row for weekends/holidays), and the forecast...
0
answers
0
votes
107
views
asked 2 years agolg...
What's the best way to filter out duplicated records in a Glue ETL Job with bookmarking enabled?lg...
I have an etl pipeline that loads json data from a source bucket, runs an etl job with bookmarking enabled, and writes as parquet to a target bucket.
I'd like to ensure that the target bucket never...
1
answers
0
votes
5857
views
asked 2 years agolg...
I'm working on a step function state machine and can create lambdas in python and node to update an existing item in ddb. However, I can't seem to find any examples with service integrations AND...
1
answers
0
votes
645
views
asked 2 years agolg...
On the AWS EMR console, we are seeing AWS EMR 6.5.0 version being available.
However, EMR Documentation doesn't have any specific information on 6.5.0.
When will the documentation be updated based on...
1
answers
0
votes
415
views
asked 2 years agolg...
Hello,
Where can I find more details on AWS' approach around data models? This would include industry-specific data models AWS is fully invested in.
1
answers
0
votes
250
views
asked 2 years agolg...
We currently run into the Problem, that some data analytic workloads exceed the 15 minutes timeout of a lambda.
It is a multistep process with steps that are parallelizable and some that are not....
3
answers
0
votes
2681
views
asked 2 years agolg...
What are the key points to choose one of the following:
* Data pipeline,
* Step function
* Amazon Managed Workflows for Apache Airflow
1
answers
0
votes
2485
views
I have a Data Pipeline which reads CSV files from an S3 bucket and copies the data into an RDS database.
I specify the bucket/folder name and it goes through each CSV file in the bucket/folder and...
2
answers
0
votes
326
views
asked 2 years agolg...