Serge Chegorian's System Center Blog

Serge Chegorian's System Center Blog

[PS]: Running parallel processes in PowerShell

October 15th, 2020

Many of you were in the situation when you have to run the same routine against multiple end points. If you run them in sequence, the process might take very long time depending on the routine complexity and the network performance. The wiser solution is to run all routines in parallel and than to collate data. The following cmdlets will help you:

Start-Job
Get-Job
Receive-Job
Remove-Job

So what each of them does?

Start-Job -ScriptBlock $scriptBlock -ArgumentList($ArgumentList)

Start-Job has two main parameters: ScriptBlock and ArgumentList. ScriptBlock is the main piece of code of your routine. This is what will be executed independently using ArgumentList as a parameter.

Get-Job gets the list of active jobs created by Start-Job and their status which could be ‘running’, ‘completed’ or ‘failed’.

Receive-Job uses Job ID as a parameter and gets all output from the $job created by Start-Job. This is important because Receive-Job simply returns the content of $job buffer. Whatever you put there using Write-Host, return or any other function will be returned as unindexed block of data.

Remove-Job uses $job handler or $job ID as a parameter and deletes the job. Remember that the job is not deleted automatically once it is completed and job results are sitting in the buffer until the job is deleted by Remove-Job cmdlet.

So the structure of your script should be as follows:

#Script Block
$scriptBlock = {
param ($MyParam)
function MyFunction
{
            return $Return
}
            MyFunction -MyParam $MyParam
}
#Starting jobs with variable parameter
for ($k=0;$k -lt $Something; $k++){
$job = Start-Job -ScriptBlock $scriptBlock -ArgumentList(MyParam)
}
# Checking jobs status. If at least one is running waiting for 10 seconds and checking again
While (Get-Job -State "Running") { 
$iScriptItterations++
$iRunning=0
$iCompleted=0
$iFailed=0
foreach($job in Get-Job) {
if ([string]$job.JobStateInfo -eq "Completed") {$iCompleted++}
if ([string]$job.JobStateInfo -eq "Running") {$iRunning++}
if ([string]$job.JobStateInfo -eq "Failed") {$iFailed++}
}
Write-Host "Total jobs: $iTotalJobs, $iRunning jobs are running... Completed:
$iCompleted, failed: $iFailed"
Start-Sleep 10 
}
#All jobs have fiinished. Adding job results to the array and deleting the job
$AllJobResults= @()
foreach($job in Get-Job){
$AllJobResults += Receive-Job -Job $job 
Remove-Job -Job $job 
}

One more bit. As I have mentioned Receive-Job returns unindexed results. What I have found practical is to return data in a form of generic PowerShell object. This will allow you later to sort your data and display them in a form of the table:

$objReturn = New-Object -Type PSObject -Property @{
  'DataFiled1' = $Data1
  'DataFiled2' = $Data2
  ...
  'DataFiledN' = $DataN
}
$AllJobResults | Sort-Object -Property Data1,Data2 |Format-Table -property Data1,Data2,DataN

Serge Chegorian's System Center Blog

Serge Chegorian's System Center Blog