Randomly hanging PowerShell with cmdlet Connect-AzAccount and Get-AzDataFactoryV2PipelineRun

168 Views Asked by At

I'm using the following PowerShell script to print ADF status in the scheduling tool ActiveBatch, but the script randomly hangs, when I investigated, the PowerShell process is active in the background. For example if the script is triggered at 1am it will print status up to 2am then nothing will get printed in the ActiveBatch console but the triggered PowerShell is active when I checked with Get-Process command.

I'm using invoke-command to run Get-AzDataFactoryV2PipelineRun, this is triggering threads in the process, at the time script hang it had a user request thread followed by an event pair low but these two wait reasons are there every time the loop initiates. I'm thinking of running the script directly instead of using invoke-command, could this make any difference?

Following is the script which hangs

[Int32] $PollSeconds = 30
$PipelineRunId = "111111111111111111"

function Invoke-RetryCommand {
    Param(
        [Parameter(Mandatory = $True)] [scriptblock] $ScriptBlock,
        [Int32] $Maximum = 3,
        [Int32] $Delay = 10
    )
    $Attempt = 0

    While ($True) {
        $Attempt++

        Try {
            return Invoke-Command -ScriptBlock $ScriptBlock               
        }
        Catch {
            If ($Attempt -eq $Maximum) {
                Throw
            }
            Write-Error $error[0] -ErrorAction Continue
            Start-Sleep -Seconds ($Delay * $Attempt)
        }
    }
}


$IsRunning = $True

While ($IsRunning) {
    # Logging into Azure
    Connect-AzAccount -Credential $Credential -Tenant $TenantId -Subscription $SubscriptionId -ServicePrincipal -Scope Process
    $ProcessContext = Get-AzContext

    # Get Azure Data Factory pipeline run status
    $Result = Invoke-RetryCommand -ScriptBlock {
    Get-AzDataFactoryV2PipelineRun -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineRunId $PipelineRunId -DefaultProfile $ProcessContext -ErrorAction Stop
    }
    
    # Check Azure Data Factory Pipeline run status and act accordingly
    Switch ($Result.Status) {
        "Queued" {
            Write-Host $(Get-Date -format "HH:mm:ss") :$Status
            Start-Sleep -Seconds $PollSeconds
        }
        "InProgress" {
            Write-Host $(Get-Date -format "HH:mm:ss") :$Status
            Start-Sleep -Seconds $PollSeconds
        }
        "Canceling" {
            Write-Host $(Get-Date -format "HH:mm:ss") :$Status
            Start-Sleep -Seconds $PollSeconds
        }
        "Failed" {
            Write-Host $(Get-Date -format "HH:mm:ss") :$Status
            $FailureMessage = $Result.Message.ToString()
            $ErrorMessage = "`r`n`r`n`r`n ############ADF FAILURE############ `r`n " +
            "ADF Pipeline Name = $PipelineName `r`n " +
            "GUID = $PipelineRunId `r`n " +
            "ADF Failure Message: $FailureMessage `r`n " +
            "################################## `r`n `r`n `r`n"

            Throw $ErrorMessage
        }
        "Cancelled" {
            Write-Host $(Get-Date -format "HH:mm:ss") :$Status
            Throw "CANCELLED - ADF Pipeline Name = $PipelineName, GUID = $PipelineRunId"
        }
        "Succeeded" {
            Write-Host $(Get-Date -format "HH:mm:ss") :$Status
            Write-Host "SUCCEEDED - ADF Pipeline Name = $PipelineName, GUID = $PipelineRunId"
            $Result | Format-List
            $IsRunning = $False
            Break
        }
        Default {
        Write-Host $(Get-Date -format "HH:mm:ss") :$Status
        }
    }
}
0

There are 0 best solutions below