Error in creating H2OContext in databricks using pysparkling

Question

I am using spark version 2.4.4 and h2o-pysparkling-2.4 on the databricks and running following code

h2oConf = H2OConf().set('spark.sql.autoBroadcastJoinThreshold', '-1')
hc = H2OContext.getOrCreate(conf=h2oConf)

Sometimes it is working well but sometimes it is giving me following error

Py4JJavaError: An error occurred while calling o2542.getOrCreate.
: org.apache.spark.SparkException: Exception thrown in awaitResult:

Please suggest me the way to resolve this issue

Answer 1

• Make sure you have "Enable Autoscaling" unchecked. You need fixed number of nodes for Sparkling Water.
• Try using a non-ML runtime.
• Also, check if you have any other packages that could conflict with pysparkling.

You can follow the standard setup here.