Cannot find AWS tool to ingest data from Source (Miro)

Question

Hi! I have been searching and playing around with services and cannot seem to find what I need.

I am using the following architecture to guide me in building out my end-to-end solution: https://aws.amazon.com/blogs/architecture/text-analytics-on-aws-implementing-a-data-lake-architecture-with-opensearch/

For my company, we have a lot of third party applications that we need to grab data from and place into the S3 bucket. However, I cannot find a service that seems to fit my needs for a handful of applications. My first one in particular is Miro.

I have played with AppFabric, but this makes assumptions on set up and also seems to be focused more on logs and security. That is not what I need here. I have gone through the other services listed in the diagram listed above, and cannot find what I need. I know there is a plethora of services...so I may be looking over this for sure! AWS Glue seems to be more of an ETL, but also does not have a native connector to Miro.

But when using Miro as my example...how can I get all the data I want (metadata, boards, image files, etc) to my S3 bucket.

Thank you so much! I am new to a lot of this!!! Also, for my other applications...is there a guide in helping what service may fit what type of product.

tsimp · Answer

I was thinking lambdas may be the way to go...but I wasn't 100% sure if I wasn't missing a service that would help out! The list of services grows often and cool stuff flies under my radar all the time!

THANK YOU SO MUCH!

Sasidhar · Answer

To extract data from Miro (or any other app) and push that data into an AWS S3 bucket, you generally need to leverage APIs provided by the application. In the case of Miro, they do offer a REST API that allows you to access boards, widgets, and other data.

https://developers.miro.com/reference/api-reference

Here's a simplified flow of how you could do this:

Use the Miro API to extract data.
Process the data if needed (transform, clean, etc).
Upload the processed data to your AWS S3 bucket.

To automate this, you might use AWS Lambda, a service that lets you run your code without provisioning or managing servers. Lambda can be triggered by various events, and it can call the Miro APIs, process the data, and then store it in S3.

Cannot find AWS tool to ingest data from Source (Miro)

Relevant content