deploy databricks workspace with powershell fails to deploy more than one worskapce

Question

Im trying to deploy the databrick workspace with yml pipeline that should run powershell script.

this is my function:

function Create-Databricks {
    param (
        [string] $rgName,
        [string] $dbName
    )
    
    az extension add --name databricks
    az databricks workspace create --resource-group $rgName --name $dbName --location 
    westeurope --output none

}

and this is my for each loop that has to create multiple resoures groups and their own databricks workspaces

foreach ($key in $iniContent["trainees"].Keys) {
# Get the trainee name
$traineeName = $iniContent["trainees"][$key]

# Set Azure context
# az account set --subscription "721fd46a-a810-4202-aa25-8b58b0410e1b"

$names = $traineeName -split ' '
$lastname = ($traineeName -split ' ')[1].Substring(0, [Math]::Min(5, 
$traineeName.IndexOf(" "))).ToLower()
# RG name
$resourceGroupName = "rg-academy-databricks-$lastname"
# SA name
$storageName = "stacademy$lastname"
# DB name
$databricksName = "dbs-academy-course-$lastname-001"
$accessconnectorName = "dbscon-academy-course-$lastname-001"
....

# Check if Databricks workspace exists
$dbExists = Get-AzDataBricksWorkspace -ResourceGroupName $resourceGroupName -Name 
$databricksName -ErrorAction SilentlyContinue
if (-not $dbExists) {
    Write-Output "Databricks workspace $databricksName does not exist. Creating..."
    # Call function to create Databricks workspace if it doesn't exist
    Create-Databricks -rgName $resourceGroupName -dbName $databricksName -tags $tags
}
else {
    Write-Output "Databricks workspace $databricksName already exists. Skipping 
creation."
}

}

It is creating successfully the first interation and deployes resource group, storage account, and databricks workspace. But when it goes to the second iteration, it does eveything except databricks workspace and throws this error

WARNING: Default enabled including preview versions for extension installation now. Disabled in May 2024. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only.

WARNING: Extension 'databricks' 0.10.2 is already installed.

ERROR: The command failed with an unexpected error. Here is the traceback:

ERROR: deadlock detected by _ModuleLock('msrest.exceptions') at 2811310367824

Traceback (most recent call last):

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 664, in execute

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 731, in _run_jobs_serially

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 712, in _run_job

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 1035, in call

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 108, in result

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 130, in wait

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 83, in _start

File "C:\Program Files\Common Files\AzureCliExtensionDirectory\databricks\azext_databricks\aaz\latest\databricks\workspace_create.py", line 206, in _execute_operations

self.pre_operations()

File "C:\Program Files\Common Files\AzureCliExtensionDirectory\databricks\azext_databricks\custom.py", line 38, in pre_operations

from msrestazure.tools import is_valid_resource_id, resource_id

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msrestazure/init.py", line 28, in

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msrestazure/azure_configuration.py", line 34, in

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msrest/init.py", line 28, in

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msrest/configuration.py", line 38, in

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msrest/universal_http/init.py", line 53, in

File "", line 1173, in _find_and_load

File "", line 171, in enter

File "", line 116, in acquire

_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('msrest.exceptions') at 2811310367824

To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Answer 1

Deploy data bricks workspace with PowerShell fails to deploy more than one workspace:

Check below to avoid conflicts with your PowerShell script:

• Move az extension add command from inside of the function to outside to install it only once. Otherwise, it will install and make a conflict every time the loop executes.

• And also check the extension release and update it if required using az extension update --name databricks.

• Use try catch blocks to catch exceptions under function block as it helps to understand the error or warning more clearly.

• Also check the pipeline configuration is upgraded and compatible with the CLI version you are using.

I tried your script in my environment by applying different use cases and was successful as shown.

az extension add --name databricks
function Create-Databricks {
    $rgName = "xxx"
    $dbName = "newdb2j" 
    az databricks workspace create --resource-group $rgName --name $dbName --location westeurope

}

$dbExists = Get-AzDataBricksWorkspace -ResourceGroupName $rgName -Name $dbName -ErrorAction SilentlyContinue
if (-not $dbExists) {
    Write-Output "Databricks workspace $databricksName does not exist. Creating..."
    Create-Databricks -rgName $rgName -dbName $dbName
}
else {
    Write-Output "Databricks workspace $databricksName already exists. Skipping 
creation."
}