Piping to Where-Object and Foreach.Object not working in module delayed loaded in $Profile

Question

Preamble: this is not about "fixing the code", as I already fixed it. This is about "understanding what went wrong, so to avoid similar mistakes in future"

Situation:
Powershell 7.4.1.

I use THIS piece of code(which I got from some website I cannot recall) in my $Profile to delay-load modules and scripts. The live one has more code but not relevant: I've tested this is where the trouble is.

Specifically, I use it to load a Module I wrote for personal use. (I know I don't actually need to load modules in my $Profile script as long as they are in my $env:PSModulePath; I'm sure there was a reason I did it but honestly cannot remember what.)

True contents of the original module do not matter as the Minimum Reproducible Example is:

scirpt in $profile

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [Text.Encoding]::UTF8
# https://seeminglyscience.github.io/powershell/2017/09/30/invocation-operators-states-and-scopes
$GlobalState = [psmoduleinfo]::new($false)
$GlobalState.SessionState = $ExecutionContext.SessionState

$Job = Start-ThreadJob -Name TestJob -ArgumentList $GlobalState -ScriptBlock {
    $GlobalState = $args[0]
    . $GlobalState {

        # We always need to wait so that Get-Command itself is available
        do {
            Start-Sleep -Milliseconds 200
        } until (Get-Command Import-Module -ErrorAction Ignore)

        
        # other dot-sourced scripts...
        # . "$ProfileDirectory\CustomAlias.ps1"
        # . "$ProfileDirectory\CustomConstants.ps1"
        # . "$ProfileDirectory\CustomVariables.ps1"
        # . "$ProfileDirectory\CustomPrompt.ps1"

        # Import-Module -Name Sirtao
        Import-Module -Name Get-DirectoryItem

    }
}



$null = Register-ObjectEvent -InputObject $Job -EventName StateChanged -SourceIdentifier Job.Monitor -Action {
    # JobState: NotStarted = 0, Running = 1, Completed = 2, etc.
    if ($Event.SourceEventArgs.JobStateInfo.State -eq 'Completed') {
        $Result = $Event.Sender | Receive-Job
        if ($Result) {
            $Result | Out-String | Write-Host
        }

        $Event.Sender | Remove-Job
        Unregister-Event Job.Monitor
        Get-Job Job.Monitor | Remove-Job
    }
    elseif ($Event.SourceEventArgs.JobStateInfo.State -gt 2) {
        $Event.Sender | Receive-Job | Out-String | Write-Host
    }
}

Module loaded

function Get-DirectoryItem {
    [CmdletBinding(DefaultParameterSetName = 'BaseSet')]
    [Alias('Get-Dir', 'GD')]
    param (
    )

    process {
        1..3 | Where-Object { $_ }
        1..3 | ForEach-Object { $_ }
        $a = @('a', 'b', 'c') 
        $a | ForEach-Object { $_ }
        $a | Where-Object { $_ }

    }
}

What I did try: simply running the command.

What I was expecting: the script returning the values of the arrays

What I got: the errors ForEach-Object: Object reference not set to an instance of an object. and Where-Object: Object reference not set to an instance of an object.

Please note that Get-Item, Get-ChildItem and Get-FileHash, the only other examples of piping I used in my modules, do work as expected

How i did fix it: removing the import from the Job. The module was still imported automagically and everything worked as expected. But as i said, this is not about fixing, but understanding.

So... any ideas?

Answer 1

This answer attempts to explain why the issue happens but that does not mean implementing this for a "faster profile loading" is advisable.

---

First, an easier way to reproduce this issue:

$module = [psmoduleinfo]::new($false)
$module.SessionState = $ExecutionContext.SessionState

Start-ThreadJob {
    . $using:module {
        while (-not (Get-Command Import-Module -ErrorAction Ignore)) {
            Start-Sleep -Milliseconds 200
        }

        $newItemSplat = @{
            Value = 'function Test-Func { 0..10 | Where-Object { $_ } }'
            Path  = [guid]::NewGuid().ToString() + '.psm1'
        }

        $tmp = New-Item @newItemSplat
        Import-Module $tmp.FullName
        $tmp | Remove-Item
    }
} | Receive-Job -Wait -AutoRemoveJob

Test-Func

As pointed out by mclayton's helpful comment, adding .Ast.GetScriptblock() fixes the issue:

Value = 'function Test-Func { 0..10 | Where-Object { $_ }.Ast.GetScriptblock() }'

Why it fixes the issue?

Because .GetScriptblock() creates a new instance stripping out its Runspace affinity, this is something that has been discussed by Paul Higin (creator of the ThreadJob Module) in GitHub issue #4003. See also PR #18138 - Make PowerShell class not affiliate with Runspace when declaring the NoRunspaceAffinity attribute for more information on this subject.

Why it fails?

This is based on guessing but when doing . $using:module { ... Import-Module ... } we are creating a nested module in that PSModuleInfo instance and when trying to invoke Test-Func, the Where-Object scriptblock is trying to marshal back to the originating Runspace which is no longer alive because the ThreadJob already ended, thus getting the null reference exception.

How to prove this is the case?

Because if we keep the originating Runspace alive the issue is gone :)

$module = [psmoduleinfo]::new($false)
$module.SessionState = $ExecutionContext.SessionState
$rs = [runspacefactory]::CreateRunspace()
$rs.Open()

$ps = [powershell]::Create($rs).AddScript({
    . $args[0] {
        while (-not (Get-Command Import-Module -ErrorAction Ignore)) {
            Start-Sleep -Milliseconds 200
        }

        $newItemSplat = @{
            Value = 'function Test-Func { 0..10 | Where-Object { $_ } }'
            Path  = [guid]::NewGuid().ToString() + '.psm1'
        }

        $tmp = New-Item @newItemSplat
        Import-Module $tmp.FullName
        $tmp | Remove-Item
    }
}).AddArgument($module)

$task = $ps.BeginInvoke()
while (-not $task.AsyncWaitHandle.WaitOne(200)) { }

Test-Func

---

Additional info for anyone looking to dig deeper into the root cause, I believe mclayton has again provided the key source blocks causing this exception to be thrown:

• ExecutionContext.cs#L1144-L1155 sets Events = null; when the runspace is disposed.

• Then, in ScriptBlock.InvokeWithPipe(...), it tries to reference the .Events property: context.Events.SubscribeEvent(...) which is already null causing the NRE.