Launching the Spark history server and viewing the Spark UI using Docker

0

Hi,

Have followed the below documentation to set up the Spark History server to see Spark UI Logs. Am able to run the container but not able to access the URL http://localhost:18080 .

docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=s3a://<bucket>/logs/ -Dspark.hadoop.fs.s3a.access.key=<ACCESS KEY> -Dspark.hadoop.fs.s3a.secret.key=<Secret Key>" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"

Where do I check whether the history server has been started and what is the Error info?

AWS
asked 4 months ago227 views
1 Answer
0

Hello.

Can you check the container status with the following command?
If it has started without any problems, "running" should be displayed.

docker ps -a

You can check the container log with the following command.
Are there any errors shown in the container logs?

docker logs -f container_id
profile picture
EXPERT
answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions