Automating data updates on Quicksight

0

Hello,

I have multiple different CSV files that I receive via email, on a day to day basis.

I would like to build a pipeline to ensure that any datasets that I receive through emails, can be picked up by Quicksight automatically and all the updates are done without much intervention.

As a result I thought of the following:

  1. Update all the CSVs (from email) I get each day, into S3 Buckets.

  2. Point the S3 buckets to AWS Athena.

  3. Quicksight will use Athena to do automatic visual updates.

For this to work, I will have to ensure that the data manipulation is done outside AWS, and then manually update CSVs inside S3 buckets on my own, each day.

Another alternative idea is the following:

  1. Place all data into S3 Buckets.

  2. Use Glue data brew to manipulate data.

  3. Store it back to S3 buckets.

  4. Use Quicksight to visualise data (pointing it back to S3 buckets).

Can someone please advise of a better method than this ? I am looking for more automation and easier methods of manipulating data for Quicksight to update each day. My data columns don't change, but new date fields and values are added each day.

Dinesh
gefragt vor 2 Jahren687 Aufrufe
1 Antwort
0

Hi, @Dinesh

I understand that you want to automate the following three things.

  1. Importing data into the lake

  2. Processing of intermediate data

  3. Drawing

  4. is not mentioned in detail, so ignore it.

  5. can be realized by using the automatic update function of SPICE.
    Before doing 1), you need to sort out where all your data comes from and how you can get it. In many cases it can be automated with Glue or Lambda / Step Functions.

profile picture
EXPERTE
iwasa
beantwortet vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen