I'm using the following PowerShell script to print ADF status in the scheduling tool ActiveBatch, but the script randomly hangs, when I investigated, the PowerShell process is active in the background. For example if the script is triggered at 1am it will print status up to 2am then nothing will get printed in the ActiveBatch console but the triggered PowerShell is active when I checked with Get-Process command.
I'm using invoke-command to run Get-AzDataFactoryV2PipelineRun, this is triggering threads in the process, at the time script hang it had a user request thread followed by an event pair low but these two wait reasons are there every time the loop initiates. I'm thinking of running the script directly instead of using invoke-command, could this make any difference?
Following is the script which hangs
[Int32] $PollSeconds = 30
$PipelineRunId = "111111111111111111"
function Invoke-RetryCommand {
Param(
[Parameter(Mandatory = $True)] [scriptblock] $ScriptBlock,
[Int32] $Maximum = 3,
[Int32] $Delay = 10
)
$Attempt = 0
While ($True) {
$Attempt++
Try {
return Invoke-Command -ScriptBlock $ScriptBlock
}
Catch {
If ($Attempt -eq $Maximum) {
Throw
}
Write-Error $error[0] -ErrorAction Continue
Start-Sleep -Seconds ($Delay * $Attempt)
}
}
}
$IsRunning = $True
While ($IsRunning) {
# Logging into Azure
Connect-AzAccount -Credential $Credential -Tenant $TenantId -Subscription $SubscriptionId -ServicePrincipal -Scope Process
$ProcessContext = Get-AzContext
# Get Azure Data Factory pipeline run status
$Result = Invoke-RetryCommand -ScriptBlock {
Get-AzDataFactoryV2PipelineRun -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineRunId $PipelineRunId -DefaultProfile $ProcessContext -ErrorAction Stop
}
# Check Azure Data Factory Pipeline run status and act accordingly
Switch ($Result.Status) {
"Queued" {
Write-Host $(Get-Date -format "HH:mm:ss") :$Status
Start-Sleep -Seconds $PollSeconds
}
"InProgress" {
Write-Host $(Get-Date -format "HH:mm:ss") :$Status
Start-Sleep -Seconds $PollSeconds
}
"Canceling" {
Write-Host $(Get-Date -format "HH:mm:ss") :$Status
Start-Sleep -Seconds $PollSeconds
}
"Failed" {
Write-Host $(Get-Date -format "HH:mm:ss") :$Status
$FailureMessage = $Result.Message.ToString()
$ErrorMessage = "`r`n`r`n`r`n ############ADF FAILURE############ `r`n " +
"ADF Pipeline Name = $PipelineName `r`n " +
"GUID = $PipelineRunId `r`n " +
"ADF Failure Message: $FailureMessage `r`n " +
"################################## `r`n `r`n `r`n"
Throw $ErrorMessage
}
"Cancelled" {
Write-Host $(Get-Date -format "HH:mm:ss") :$Status
Throw "CANCELLED - ADF Pipeline Name = $PipelineName, GUID = $PipelineRunId"
}
"Succeeded" {
Write-Host $(Get-Date -format "HH:mm:ss") :$Status
Write-Host "SUCCEEDED - ADF Pipeline Name = $PipelineName, GUID = $PipelineRunId"
$Result | Format-List
$IsRunning = $False
Break
}
Default {
Write-Host $(Get-Date -format "HH:mm:ss") :$Status
}
}
}