Questions tagged with AWS Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hi there,
We are migrating from datapipeline to step function, and on datapipeline I am able to forward the log messages for individual steps to a log file, but this doesn't work on step function....
Hi All
I am trying to create a batch segment job and I have created the segment solution and while creating the batch job, I entered the link to input file on S3 and also added link to output folder...
I want to read data from "Bentley Projectwise " and "MS Form" . Can anyone please suggest what are the best ways to connect and read data from Bentley Projectwise and MSForms?
Hi,
I have an EmrActivity step on the data pipeline that backups the dynamoDB and stores it in S3. I would like to use this functionality on a step function or something other than Data Pipeline. If...
We have Microsoft SQL DB and we are taking backup of data in CSV format from MSSQL db and uploading to S3 bucket. I tried to query data using Athena and AWS API Gateway to build API. Athena looks do...
I'm attempting to use AWS Data Pipeline to move a CSV file from my computer to AWS Data Lake as a parquet file. I'm unable to finad the exact template to select to migrate from my local...
Hello, when we see the AWS data pipeline console we see a message that AWS is planning to remove access to console by 04/03/2023. So we are checking how can we use the data pipelines from AWS CLI...
I am transforming my table by adding new columns using SQL Query Transform in AWS Glue Job studio.
SQL...
I want to run multiple python scripts to run one after another These are connected, what is the best way to do it ( want to use Ec2 Instance)
I am looking for a way to share my Terraform-based modern data stack solution, which I have created using various AWS services, with other companies privately via AWS. However, in the future, we might...
Where has the option of configuring request and response mapping template gone when creating a resolver for your data source? The way it was done in VTL?