Running Spark jobs on ECS

Is running Spark job on ECS (Fargate) recommended ? K8s has support for running spark drivers .. is there something similar for ECS ? I understand there are other services (Glue, EMR ) for Spark on AWS, but would like to understand the feasibility / complexity for running Spark on ECS Fargate.

主题

无服务器容器

标签

AWS Fargate 亚马逊弹性容器服务 (ECS)

语言

English

JackSparrow

已提问 8 个月前1241 查看次数

1 回答

最新
投票最多
评论最多

Hello.

Running Apache Spark on ECS (especially Fargate) is not a standard approach and isn't as streamlined as using EMR or Kubernetes. ECS is primarily designed for container orchestration of microservices, but with some effort, you can run Spark jobs on it.(check this article https://datachef.co/blog/run-spark-applications-on-aws-fargate/)

Regards, Andrii

专家

Andrii S

已回答 8 个月前

专家

Oleksii Bebych

已审核 9 天前

JackSparrow
8 个月前
Thanks for the response. I understand it's not as streamlined as EMR or k8s. Having said that, I would specifically like to understand any known limitation / consideration to run Spark on ECS .

Running Spark jobs on ECS

相关内容