Best way to run a web scraper from my S3 website and save the returned information to a database

0

I currently have a website hosted on S3 with a working RDS database connection. I would like to run a dynamic web scraper from this site that activates based on user input. Ideally, the user will click a button which will activate a search for a specific product across a specified list of retail sites. This data will be displayed on the site and also saved to the database. It doesn't matter if the data goes directly to the site or to the database. I would like to use an existing web scraping tool such as Scrapy. What is the best way to handle this?

1 Respuesta
2

One potential way to approach this, would be to set up the web scraper as a Lambda function, and expose the lambda to the website frontend via API gateway. The lambda can then write any output to s3 or to your RDS database.

AWS
bendutt
respondido hace 2 años

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas