Running Spark jobs on ECS

Is running Spark job on ECS (Fargate) recommended ? K8s has support for running spark drivers .. is there something similar for ECS ? I understand there are other services (Glue, EMR ) for Spark on AWS, but would like to understand the feasibility / complexity for running Spark on ECS Fargate.

主題

無伺服器容器

標籤

AWS Fargate Amazon Elastic Container Service

語言

English

JackSparrow

已提問 8 個月前檢視次數 1246 次

1 個回答

最新
最多得票
最多評論

Hello.

Running Apache Spark on ECS (especially Fargate) is not a standard approach and isn't as streamlined as using EMR or Kubernetes. ECS is primarily designed for container orchestration of microservices, but with some effort, you can run Spark jobs on it.(check this article https://datachef.co/blog/run-spark-applications-on-aws-fargate/)

Regards, Andrii

專家

Andrii S

已回答 8 個月前

專家

Oleksii Bebych

已審閱 10 天前

JackSparrow
8 個月前
Thanks for the response. I understand it's not as streamlined as EMR or k8s. Having said that, I would specifically like to understand any known limitation / consideration to run Spark on ECS .

Running Spark jobs on ECS

相關內容