Best way to run a web scraper from my S3 website and save the returned information to a database

0

I currently have a website hosted on S3 with a working RDS database connection. I would like to run a dynamic web scraper from this site that activates based on user input. Ideally, the user will click a button which will activate a search for a specific product across a specified list of retail sites. This data will be displayed on the site and also saved to the database. It doesn't matter if the data goes directly to the site or to the database. I would like to use an existing web scraping tool such as Scrapy. What is the best way to handle this?

1 個回答
2

One potential way to approach this, would be to set up the web scraper as a Lambda function, and expose the lambda to the website frontend via API gateway. The lambda can then write any output to s3 or to your RDS database.

AWS
bendutt
已回答 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南