Can't access spark UI even if I use the ip address reported in the console

Question

I cannot access the web UI of my containerized spark cluster even if I copy and paste the following Ip address:

Stopped Spark web UI at http://987c8b219d32:4040

The cluster that I've built is taken from this tutorial Spark Cluster Tutorial

Answer 1

The log specifies that the web UI is stopped. You can't access the Web UI after the spark application have stopped as it is explained in the documentation:

Note that this information is only available for the duration of the application by default.

To enable accessing the web UI after the application have stopped you need to:

set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage.

Please check the documentation here.

Also note, that in the tutorial, the default web UI ports are changed to:

SPARK_MASTER_WEBUI_PORT=8080 \
SPARK_WORKER_WEBUI_PORT=8080 \

and that they are exposed outside the container in the docker-compose file to the ports 9090, 9091 and 9092:

  - "9090:8080"
  - "9091:8080"
  - "9092:8080"

Which means that you can access the Web UI using: http://localhost:9090