How to remove libraries from Azure Databricks Cluster

Question

I am using terraform to configure databricks cluster. I was defining libraries to install and made a mistake in the terraform script by giving comma separated libraries.

in main.tf

library {
  pypi {
    package = "fbprophet==0.6,pandas"
  }
}

the installation faild, but there is no way to remove the faild installation from the library list.

when I try to do anything with terraform (apply/destroy) i get the following error:

$ terraform apply
module.azurerm-databricks-instancecluster.databricks_cluster.cluster: Refreshing state... [id=0517-053850-sniff405]
╷
│ Error: library_pypi[fbprophet==0.6,pandas] failed: java.lang.RuntimeException: ManagedLibraryInstallFailed: org.apache.spark.SparkException: Process List(/databricks/python/bin/pip, install, fbprophet==0.6,pandas, --disable-pip-version-check) exited with code 1. ERROR: Invalid requiremrocess List(/databricent: 'fbprophet==0.6,pandas'                                                                                                                    pandas'
│  for library:PythonPyPiPkgId(fbprophet,Some(0.6,pandas),None,List()),isSharedLibrary=false

any idea on how to resolve this issue?

Answer 1

Steps to remove libraries from Azure Databricks Cluster:

Step1: Select the libraries which you want to remove.

Step2: Click on Uninstall and confirm

Step3: Restart the cluster to remove the libraries

When you uninstall a library from a cluster, the library is removed only when you restart the cluster. Until you restart the cluster, the status of the uninstalled library appears as Uninstall pending restart.