Using powershell jobs to track powershell winRM remoting NO winRM timeout

Question

I have got a powershell script working, the job runs a list of servers, remotely connects to the servers and runs a simple script to get the hostname from the server. The issue is that it does get stuck on some occasions. I read up on https://4sysops.com/archives/invoke-command-compensating-for-slow-responding-computers/. I came across the use of jobs where a timeout can be specified.

Here is my complete script and at the moment, it currently hangs on some hosts.

workflow Connect-windows-Servers
{
param(
[string[]] 
$server_list
)


foreach -parallel -throttlelimit 50  ($server_name in $server_list){
write-output "Inside Workflow but outside inline $server_name $(Get-Date)"
    InLineScript {
    $output_status = "" | Select-Object -Property server_name, isError, result 
    $in_server_name = $using:server_name

    write-output "inline  = $in_server_name"
    $username = "my_usersname"
    $password = "my_password"
    $secstr = New-Object -TypeName System.Security.SecureString
    $password.ToCharArray() | foreach-object {$secstr.AppendChar($_)}
    $credentials = New-Object -typename System.Management.Automation.PSCredential -argumentlist $username,$secstr

    $result = Invoke-Command -ComputerName $in_server_name -ScriptBlock {   [System.Net.Dns]::GetHostName() } -ErrorVariable ErrVar  -Credential $credentials
    
    write-output "Error value is $ErrVar"

    if ($ErrVar)
    {
        $output_status.server_name = $in_server_name
        $output_status.isError = $true
        $output_status.result = $ErrVar
        $deployment_output += New-Object -TypeName psobject -Property @{server_name = $in_server_name; isError = $true ; result = $ErrVar } 


        $sql_query = "insert into my_table (servername,isfailed, result) select '$in_server_name','$true','$ErrVar'"
        write-output $sql_query

        Invoke-Sqlcmd -ServerInstance "db_server" -Query $sql_query -Database  "db_name"     

    }     
    else
    {

        $output_status.server_name = $in_server_name
        $output_status.isError = $false
        $output_status.result = $result
        $deployment_output += New-Object -TypeName psobject -Property @{server_name = $in_server_name; isError = $false ; result = $result } 

        $sql_query = "insert into my_table (servername,isfailed, result) select '$in_server_name','$false','$result'"
        write-output $sql_query

        Invoke-Sqlcmd -ServerInstance "db_server" -Query $sql_query -Database  "db_name"       

        }
     }
  }
    }

$list = @('server1',server2)

Connect-windows-Servers -server_list $list

What is the issue.

The issue here is that the script above hangs on some hosts, and when that happens the whole process is stuck. No errors at all from winRM, this makes automation very difficult, the code also needs to be able to run in parallel as there are over 2,000 hosts.

What have I done ?

I introduced powershell jobs, however the issue with this is that I cant seem to be able to get the value of $results, what I want to know is hosts that were successful connection wise, get the value of the host name back and for hosts that timed out, I want to be able to identify them and most importantly use workflows or any other means to run this in parallel, please note that I am on powershell v5.

If the section below is changed to

$result = Invoke-Command -ComputerName $in_server_name -ScriptBlock {   [System.Net.Dns]::GetHostName() } -ErrorVariable ErrVar  -Credential $credentials
$result | wait-job timeout (1*60) | Remove-Job

What then happens is that I loose track of the status of the particular server as $result no longer contains the hostname and I have no way to work out which jobs timed out.

Another solution that could work using my original code snippet is if I can get winrM to gracefully timeout and if there is a way for me to know when this happens so that I can log it for the particular host.

Please ignore the hardcoded username/password, its not good practice. As I am running the script several times, its easier this way then attending to a prompt several times.

Answer 1

I would approach the code using New-PSSession with a Try / Catch and a List where you can store the possible errors.

The downside is that the loop to establish the connection is linear and it can take time.

$servers = 'host1', 'host2', 'host3'
$failed  = [Collections.Generic.List[object]]::new()
$session = foreach($server in $servers) {
    try {
        New-PSSession $server -Credential $cred -ErrorAction Stop
    }
    catch {
        $failed.Add([pscustomobject]@{
            Server    = $server
            Exception = $_.ToString()
        })
    }
}

# Highly unlikely this could fail if you could establish a PSSession
if($session) {
    Invoke-Command -SessionName $session -ScriptBlock { [System.Net.Dns]::GetHostName() }
    Remove-PSSession $session
}

if($failed) {
    # Has all records of failed connections with the corresponding error messages
    $failed
}

If you wanted something faster but dirtier you can use redirection:

$servers = 'host1', 'host2', 'host3'
$result  = Invoke-Command -ComputerName $servers -ScriptBlock {
    [System.Net.Dns]::GetHostName()
} -Credential $credential -ErrorAction Continue 2>&1

$failed, $success = $result.where({ $_ -is [System.Management.Automation.ErrorRecord]}, 'Split')
$failed  # => has the possible errors
$success # => has the remote results