Manual submitted step jobs failing

Question

We have a airflow setup runs the EMR jobs daily basis. I noticed an odd behavior that when I resubmit job for calculating the adhoc reports, spark application failed with below error, arguments seems to be leading some issue at AWS end as the same job worked before.

" No main class set in JAR; Please specify one with --class "

Accepted Answer

Hello,

Sorry for the inconvenience caused. This is an ongoing issue in EMR Step console for spark-application type as the order of the arguments changed when submitting. Internal team is being worked on fixing this issue and you will be able to submit the jobs as it was before once fixed. Meanwhile, to unblock from this situation,  you can do workaround with below,

1.  Submit the job through command-runner and pass the spark-submit as argument,

a. Choose "Customized Jar" in Add step

b. Add "command-runner.jar" in the Jar-Location

c. In the Argument field, place the entire spark-submit command (example - spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi s3:///samplejob/spark-examples.jar 10)

2.  Use CLI method,

```
aws emr add-steps --cluster-id  --steps Type=Spark,Name="SparkPi",Args=[--deploy-mode,cluster,--class,org.apache.spark.examples.SparkPi,s3:///samplejob/spark-examples.jar,10],ActionOnFailure=CONTINUE

```

Manual submitted step jobs failing

Contenuto pertinente