Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.
All Content tagged with AWS Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
Content language: English
Filter content
Select tags to filter
Sort by
Sort by most recent
90 results
I'm trying to create an Iceberg table on S3 using Athena SQL console, and it is working fine if the table is not partitioned. But I want to create a partitioned table.
1. On using the create table sy...
I'm working on building a real-time data pipeline on AWS to process data from IoT devices. Here's the architecture I'm aiming for:
1. AWS IoT Core receives data from edge devices.
2. Data is routed to...
I am working with AWS Athena and QuickSight. I have a dataset with boolean fields, and I need to ensure that these fields remain intact as true and false instead of being converted into integers (0 an...
I was trying to build a Data pipeline with No-Code ETL using AWS Glue Studio, and I kept getting the error message "Node is Misconfigured", when I add the join node using Visual ETL (see attached). Un...
Hi!
I have some heavy endpoints in my backend right now to procesate and generate that. The flow is the next one.
I get a request from the frontend side in my main microservice, I take those paramet...
asked 2 years ago
We have been incurring cost for aws data pipeline in **us-east-1** region but there is no data pipeline in that region in our account.
We used this method to check datapipeline. https://docs.aws.amazo...
We are planning to migrate our databases to Aurora, and we would like to know what the impact in our pipeline would be. We are currently running a dms task with Full load, ongoing replication that mig...
Hi,
Am new to aws services. Unfortunately my project manager gave me a task to do the below, can any on please give me a steps and clear understandings.
**Deploy a three tier web application using C...
asked 2 years ago
Hi Team,
I am trying to run SQL statements in Redshift by triggering using EventBridge on S3 file arrival. I am able to run SQLs using Data API but how ever I want to pass the event details to redshi...
In AWS Glue jobs, within the Targets node, I am unable to see the data types such as struct, array or map while changing the schema. Does AWS Glue not support these data types?
I have the the data of the access logs of the users in opensearch index, this data track the information of the users like at what time users access the door and got the arrival, but one user can swip...
Hi team,
Our organization is embarking on a project to design a shared data model where we can efficiently store and manage a large payload.
Multiple lines of business will be both **updating this ...