I am working in Databricks with Sparkling Water 3.40.0.4; I have a total driver memory of 512 GB and six workers with 64 GB each. When I call
hc = H2OContext.getOrCreate()
the internal H2O cluster is created across six workers, but the total cluster memory size is roughly 60 GB. I can create a normal, non-sparkling water H2O cluster and pass the max_mem_size and min_mem_size arguments to the init() method, which will return a much larger sized cluster, but I can't seem to find how to do that on Databricks.
h2o.init(max_mem_size="200g")
That returns roughly 200 GB of memory for the cluster.
I created a local Spark installation and changed the spark.driver.memory property, and changing this resulted in a larger sparkling water cluster size, but explicitly setting that property in Data Bricks has no change on the sparkling water cluster there.
Is there a configuration I can pass to Spark or an internal H2O cluster to set a larger memory size on Databricks?