Databricks Connect 13.3: Could not find connection parameters to start a Spark remote session

136 Views Asked by At

I use databricks connect v11 and try to use it now for v13. So I do the follwing:

pip install -U "databricks-connect==13.3.3"

Then I run   

       echo "y
       https://adb-xxx.7.azuredatabricks.net
       xxxx-2
       yyyy
       zzz
       15001" | databricks-connect configure

  Then I run  python3 -m pytest whereby one file uses SparkSession.builder.getOrCreate() which results in the following error:

RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.

EDIT: I also run:

export SPARK_REMOTE="sc://adb-xxxx.7.azuredatabricks.net:15001/;token=xxx-2;x-databricks-clus
ter-id=xxx"

which didn't work either.

In the home directory a valid .databricks-connect file is existing

Obviously, some config step is missing which was not necessary for v11

0

There are 0 best solutions below