Usando AWS re:Post, accetti AWS re:Post Termini di utilizzo
AWS re:Postre:Post

Running Spark jobs on ECS

0

Is running Spark job on ECS (Fargate) recommended ? K8s has support for running spark drivers .. is there something similar for ECS ? I understand there are other services (Glue, EMR ) for Spark on AWS, but would like to understand the feasibility / complexity for running Spark on ECS Fargate.

Argomenti
ServerlessContainer
Tag
AWS FargateServizio container Amazon Elastic
Lingua
English
JackSparrow
posta 8 mesi fa1246 visualizzazioni
1 Risposta
1

Hello.

Running Apache Spark on ECS (especially Fargate) is not a standard approach and isn't as streamlined as using EMR or Kubernetes. ECS is primarily designed for container orchestration of microservices, but with some effort, you can run Spark jobs on it.(check this article https://datachef.co/blog/run-spark-applications-on-aws-fargate/)

Regards, Andrii

profile picture
ESPERTO
Andrii S
con risposta 8 mesi fa
profile picture
ESPERTO
Oleksii Bebych
verificato 9 giorni fa
  • JackSparrow
    8 mesi fa

    Thanks for the response. I understand it's not as streamlined as EMR or k8s. Having said that, I would specifically like to understand any known limitation / consideration to run Spark on ECS .

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande

Contenuto pertinente