What is SPARK_DRIVER_BIND_ADDRESS in the docker entrypoint.sh for the spark image

Question

Most entrypoint.sh files that are used to run the spark docker image contains a variable called SPARK_DRIVER_BIND_ADDRESS, for example this entrypoint.sh

I'm not able to find what this property is used for, and how to get it's value ?

Answer 1

You can find on the official Spark documentation what it is, quote:

Hostname or IP address where to bind listening sockets. This config overrides the SPARK_LOCAL_IP environment variable (see below).

It also allows a different address from the local one to be advertised to executors or external systems. This is useful, for example, when running containers with bridged networking. For this to properly work, the different ports used by the driver (RPC, block manager and UI) need to be forwarded from the container's host.

https://spark.apache.org/docs/latest/configuration.html

So I suppose you have to pass an environment variable something like SPARK_DRIVER_BIND_ADDRESS=0.0.0.0 into a container to get it started without errors.