Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hello. I'm running SageMaker training jobs through a library called ZenML. The library is just there as an abstraction layer, so that when I return the artifacts gets automatically saved to S3. The...
0
answers
0
votes
142
views
asked 10 months agolg...
Hi, I'm running a data pipeline from legacy DB(oracle) to Redshift using AWS Glue.
I want to test the connection to the legacy DB before executing ETL without test query in working python...
1
answers
0
votes
267
views
asked a year agolg...
Hi Team,
I have set up an **AWS DataPipeline** to run my EMR jobs on `On-Demand` instances. However, I now want to switch to using `Spot` Instances to reduce costs. I have configured the...
2
answers
0
votes
325
views
asked a year agolg...
Hello team,
As per our client requirement we need to migrate data from quickbook to aws rds, for that we need to use appflow service and use connector for quickbook. Is anyone able to connect via...
0
answers
0
votes
125
views
asked a year agolg...
I have to convert a pipeline to step function and I have this step which loads a dynamo db table with data from S3. I am not sure how to convert this to step function. Below is the step. Any idea how...
1
answers
0
votes
363
views
asked a year agolg...
Good afternoon.
I would like to know SE when we send a snapshoot of mysql 5.7 rds allocated storage size of 8tb.
When I restore from S3 to a new RDS instance, it can only consume the instance that has...
1
answers
0
votes
293
views
asked a year agolg...
Hi there,
We are migrating from datapipeline to step function, and on datapipeline I am able to forward the log messages for individual steps to a log file, but this doesn't work on step function....
1
answers
0
votes
422
views
asked a year agolg...
Hi All
I am trying to create a batch segment job and I have created the segment solution and while creating the batch job, I entered the link to input file on S3 and also added link to output folder...
1
answers
0
votes
271
views
asked a year agolg...
I want to read data from "Bentley Projectwise " and "MS Form" . Can anyone please suggest what are the best ways to connect and read data from Bentley Projectwise and MSForms?
1
answers
0
votes
239
views
asked a year agolg...
Hi,
I have an EmrActivity step on the data pipeline that backups the dynamoDB and stores it in S3. I would like to use this functionality on a step function or something other than Data Pipeline. If...
1
answers
0
votes
278
views
asked a year agolg...
We have Microsoft SQL DB and we are taking backup of data in CSV format from MSSQL db and uploading to S3 bucket. I tried to query data using Athena and AWS API Gateway to build API. Athena looks do...
1
answers
0
votes
248
views
asked a year agolg...
I'm attempting to use AWS Data Pipeline to move a CSV file from my computer to AWS Data Lake as a parquet file. I'm unable to finad the exact template to select to migrate from my local...
0
answers
0
votes
79
views
asked a year agolg...