PowerShell Scripting Techniques and Gems – Part 1

Windows PowerShell has become the most favorable shell language that a windows admin can not go without in the modern onprem and cloud world. As it has been developed and enhanced or in a better word evolved into today’s version 5.1 of windows PowerShell and PowerShell 6.2, I will be creating a  multi part blog post to look back at what I would call PowerShell Gems. (Not talking about Ruby here ). These PowerShell gems are techniques and functionality probably unknow to most starting admins or even PowerShell veterans. Some techniques are native PowerShell others can be utilized from .NET. I will collect and provide a platform for anyone who wants to contribute so consecutive posts might be coming from fellow PowerShell enthusiasts.

In this first part of the series I will start off with a array method that is not well documented. Most of the times when you are working with arrays you will find yourself using only a subset of the data or start looping thru a array and process each item of the array.

Arrays have a few nasty little habits. First of all an array is fixed size. This means that once creased you can not add items to is or remove items from it. Luckily PowerShell is often smart enough to understand this and will do that under the hood for you, However this does come at a price of performance (One of the things I will show in a different post.).

But we ain`t here to talk about the bad stuff so let`s look at the great things you can do with array. To start with here is the official .NET class of a array.

As you might have noticed when you look at the .NET class or at its members you will find there is 1 mystery method available. The “Where” method can not be found in any of the documentation of the class. This is because it has been added to PowerShell DSC in version 4 of PowerShell. Even though it was a DSC enhancement you can use it in regular PowerShell as well. This special method is added by the PS core and comes with some advantages over the “where-object” and is only available on collections like arrays.

So what are the main advantages of this method:

• It is faster than the “where-object” cmdlet.
• It has operational modes to also directly filter the data result set

The format of the “where” method

As there is not many documentations available of this method we can use PowerShell’s error handling to find out the proper command structure of this method by simply call it on a collection without providing proper input.

So the command expects the following input:

<strong>.Where( { Expression } [, mode ] [, numbertoreturn]])</strong>

If we break this apart it will leave us with the following:

• Expression : aka filter
  This is the filter script that you want to apply to the collection ( example: all that is true )
• Mode: System.Management.Automation.WhereOperatorSelectionMode as a enum
  This is a “WhereOperatorSelectionMode” class. We can look at the values of this class by calling its declared field names.
  The documented class can be found here:
  https://docs.microsoft.com/en-us/dotnet/api/system.management.automation.whereoperatorselectionmode?view=pscore-6.2.0  

• Numbertoreturn:  integer

  This is the amount of matches it should return as a number. Note you can use mathematical calculations in this part as long as the result returned is a integer.

  Note: you can use mathematical calculations in this part as long as the result returned is a integer. 

“Where” method in action

Lets take a look at the simple form with just only basic expression filtering. To make sure timings are accurate all timings performed are in a clean PS run space and where run on a Surface pro with a I7-6650U and 16 GB RAM. In the first test the cmdlet Where-object was run first and the “where” method gets the advantage of disk cache

As I want to do a fair comparison and eliminate caching  I repeated the test using a cold boot now running where method first and give where-object the benefit of disk cache this resulted in a different view as you can clearly see the disk cache effecting testing but still the performance gain is visible for the method VS the cmdlet.

Where method numbertoreturn property:

The “numbertoreturn” property does exactly what the name says it returns the number of matched properties. You have to specify this property as a integer but you can also use any mathematical calculations that result in a integer. Be aware that this property also messes with the mode that the where method is operating in but I will show this in the mode specific part of this post.

Lets look at some example lets say I only want to see the first 2 dll files from the windows directory as a result. You could have used select-object -first 2 but using the where method with the “numbertoreturn” property you can do it directly in the filter. Unfortunately the “numbertoreturn” is not as optimized as the select-object cmdlet in the pipeline. Also you will have to specify a mode and if you looked at the modes you can see there is a default mode available for standard filtering.

The “numbertoreturn” is useful when you are working with small collections that respond pretty directly due to hardly any compute time needed to process the collection. In all other scenarios its better to just use where-object with select-object and benefit from the select-object optimization.. Mixing “Select-object” with the where method will not have the optimization kick in and results in the same slow performance as using the “numbertoreturn”.

Where method mode modifiers:

If I zoom in on to the mode modifiers you will find that some of them might be hard to find a use case for while others fix / speedup operations that would otherwise need additional code or processing. So lets look at all the modes that can be used with the where method. The Mode is declare as a enum wich means instead of the named string you can also use the corresponding enum value as input for the mode

• Default: (Enum value: 0)

  This is the default mode and will be used when no mode is specified. Normal filtering will be applied as if there is no mode change. Note: Whenever you need to use the “numbertoreturn” property you have to specify this default mode.

• Split: (Enum value: 5)

  The split method is my personal favor of all mode you can use as it splits the results into 2 arrays. The results that pass the filter and the results that fail the filter. This way you never have to run a double query to get both the passed and failed items.

  If you would look at the result of a query in split mode you will notice that the result looks the same as without split but this is due to the fact that the result is a collection of 2 collections. If you look at the result below you will notice that the passing numbers are listed first as they are in the first collection.

  Example:filter all numbers with 2 digits in the range 5 to 15 and put the result in $result

Now that we have 2 collections in the result variable we can you any of the collection in our next pieces of code. Like any multi dimensional array you can index in the underlying collection using the proper index number.

A more practical example:

As you can not do a server side filtering for mailbox quota’s you could query for all mailboxes and filter the mailboxes that have a quota of 50GB. If you used the split mode you can directly use the result to pass all passing objects to a next command to increase their quota to 100GB and use the failing objects to set their quota to 75 GB. This way you only need to run the filter command once.

Note: Mixing Split with numbertoreturn

Whenever you mix split with number to return any object that passes the filter but fails the number to return will be added to the failing objects collection.

• First: (Enum value: 1)

  When you are running where in the “first” mode it will do the same as a Select-object -first , except for the pipeline optimization that select-object has. This works pretty well on small arrays or if you just need to test your expression.  If you want to return more than just the first 1 item you can use it in combination with the “numbertoreturn”.

  Example return first 3 items:

• Last: (Enum value: 2)

  When you are running where in the last mode it will do the same as a “Select-object -last”. There is no speed optimization possible as full filtering has to be done first before reverse enumeration can begin. This is the same as with select-object. You do get the speed performance increase VS where-object.  If you want to return more than just the last 1 item you can use it in combination with the “numbertoreturn”.

  Example return last 3 items:

• SkipUntil: (Enum value: 3)

  The “Skipuntil” mode seems a bit weird at start and use cases are hard to find if you work with random arrays. “Skipuntil” will search for the first object that matches the expression and stop processing. It will return all values both passing or failing from that point on.

  If you have large array that is sorted on the property in your expression this mode can give you a real speed optimization. As the array is already sorted you only need to test for the first item that is different meaning that no additional testing has to be performed. If you array had 20% of True and 80% of false and you run a “skipuntil” against the false value on 20% needs to be tested skipping the other 80% for evaluation.

  Example:

  “Skipuntil” does never need to run the expression after the first 10 numbers no matter if it has ten thousand or a million items in the array.

• Until: (Enum value: 4)

  The “Until” mode is the opposite of skip until as it will return All items Until the expression is true.

  Example:

  In this case “Until” is even faster than “skipuntil” as it only needs to return the first 10 items.

This is the end of the first part of this series. Let me know what you think or if you have any suggestions or if you want to participate on these Gems as I am planning to create a whitepaper once we have collected a lot of these Gems and techniques.