Automating data updates on Quicksight

0

Hello,

I have multiple different CSV files that I receive via email, on a day to day basis.

I would like to build a pipeline to ensure that any datasets that I receive through emails, can be picked up by Quicksight automatically and all the updates are done without much intervention.

As a result I thought of the following:

  1. Update all the CSVs (from email) I get each day, into S3 Buckets.

  2. Point the S3 buckets to AWS Athena.

  3. Quicksight will use Athena to do automatic visual updates.

For this to work, I will have to ensure that the data manipulation is done outside AWS, and then manually update CSVs inside S3 buckets on my own, each day.

Another alternative idea is the following:

  1. Place all data into S3 Buckets.

  2. Use Glue data brew to manipulate data.

  3. Store it back to S3 buckets.

  4. Use Quicksight to visualise data (pointing it back to S3 buckets).

Can someone please advise of a better method than this ? I am looking for more automation and easier methods of manipulating data for Quicksight to update each day. My data columns don't change, but new date fields and values are added each day.

Dinesh
已提問 2 年前檢視次數 687 次
1 個回答
0

Hi, @Dinesh

I understand that you want to automate the following three things.

  1. Importing data into the lake

  2. Processing of intermediate data

  3. Drawing

  4. is not mentioned in detail, so ignore it.

  5. can be realized by using the automatic update function of SPICE.
    Before doing 1), you need to sort out where all your data comes from and how you can get it. In many cases it can be automated with Glue or Lambda / Step Functions.

profile picture
專家
iwasa
已回答 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南