New-DepthGaugeFile.ps1: The Powershell Pipeline Is Neat, But It's Also Slow

If you know me or this blog at all, you know that first and foremost, I think Powershell is awesome.  It is essential to any Windows system administrator or I.T. pro's success.  The good news is that there are a dozen ways to accomplish any given task in Powershell.  The bad news is that eleven of those twelve techniques are typically as slow as a three-legged tortoise swimming through a vat of cold Aunt Jemima.

This is not the first, or the second, blog post I've made about Powershell performance pitfalls.  One of the fundamental problems with super-high-level languages such as Powershell, C#, Java, etc., is that they take raw performance away from you and give you abstractions in return, and you then have to fight the language in order to get your performance back.

Today I ran across another such example.

I'm creating a program that reads from files.  I need to generate a file that has "markers" in it that tell me where I am within that file at a glance.  I'll call this a "depth gauge."  I figured Powershell would be a simple way to create such a file.  Here is an example of what I'm talking about:

Depth Gauge or Yardstick File


The idea being that I'd be able to tell my program "show me what's at byte 0xFFFF of file.txt," and I'd be able to easily visually verify the answer because of the byte markers in the text file.  The random characters after the byte markers are just gibberish to take up space.  In the above example, each line takes up exactly 64 bytes - 62 printable characters plus \r\n.  (In ASCII.)

I reach for Powershell whenever I want to whip up something in 5 minutes that accomplishes a simple task.  And voila:

Function New-DepthGaugeFile
{
    [CmdletBinding()]
    Param([Parameter(Mandatory=$True)]
          [ValidateScript({Test-Path $_ -IsValid})]
          [String]$FilePath, 
          [Parameter(Mandatory=$True)]
          [Int64]$DesiredSize, 
          [Parameter(Mandatory=$True)]
          [ValidateRange(20, [Int32]::MaxValue)]
          [Int32]$BytesPerLine)
    Set-StrictMode -Version Latest
    [Int64]$CurrentPosition = 0
    $ValidChars = @('a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z',
                    'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z',
                    '0','1','2','3','4','5','6','7','8','9',
                    '!','@','#','$','%','^','&','*','(',')','_','+','-','=','?','<','>','.','[',']',';','`','{','}','\','/')
    
    Try
    {
        New-Item -Path $FilePath -ItemType File -Force -ErrorAction Stop | Out-Null
    }
    Catch
    {
        Write-Error $_
        Return
    }    
    
    [Int32]$BufferMaxLines = 64
    [Int32]$BufferMaxBytes = $BufferMaxLines * $BytesPerLine

    If ($DesiredSize % $BytesPerLine -NE 0)
    {
        Write-Warning 'BytesPerLine is not a multiple of DesiredSize.'
    }


    $LineBuffer = New-Object 'System.Collections.Generic.List[System.String]'
        
    While ($DesiredSize -GT $CurrentPosition)
    {
        [System.Text.StringBuilder]$Line = New-Object System.Text.StringBuilder
        
        # 17 bytes
        [Void]$Line.Append($("{0:X16} " -F $CurrentPosition))
        
        # X bytes
        1..$($BytesPerLine - 19) | ForEach-Object { [Void]$Line.Append($(Get-Random -InputObject $ValidChars)) }        
        
        # +2 bytes (`r`n)        

        [Void]$LineBuffer.Add($Line.ToString())
        $CurrentPosition += $BytesPerLine

        # If we're getting close to the end of the file, we'll go line by line.
        If ($CurrentPosition -GE ($DesiredSize - $BufferMaxBytes))
        {
            Add-Content $FilePath -Value $LineBuffer -Encoding Ascii
            [Void]$LineBuffer.Clear()
        }

        # If the buffer's full, and we still have more than a full buffer's worth left to write, then dump the
        # full buffer into the file now.
        If (($LineBuffer.Count -GE $BufferMaxLines) -And ($CurrentPosition -LT ($DesiredSize - $BufferMaxBytes)))
        {
            Add-Content $FilePath -Value $LineBuffer -Encoding Ascii
            [Void]$LineBuffer.Clear()
        }
    }
}

Now I can create a dummy file of any size and dimension with a command like this:

New-DepthGaugeFile -FilePath 'C:\testfolder1\largefile2.log' `
                   -DesiredSize 128KB -BytesPerLine 64

I thought I was being really clever by creating an internal "buffering" system, since I instinctively knew that performing a file write operation (Add-Content) on each and every loop iteration would slow me down.  I also knew from past experience that overuse of "array arithmetic" like $Array += $Element would slow me down because of the constant cloning and resizing of the array.  I also remembered that in .NET, strongly-typed lists are faster than ArrayLists because we want to avoid boxing and unboxing.

Despite all these little optimizations, here is the performance while writing a 1MB file:

Measure-Command { New-DepthGaugeFile -FilePath C:\testfolder1\largefile.log `
                                     -DesiredSize 1MB -BytesPerLine 128 }

TotalSeconds    : 103.8428624

Over 100 seconds to generate 1 megabyte of data.  I'm running on an SSD that is capable of copying hundreds of megabytes per second of data, so the storage is not the bottleneck.

To try to speed things up, I decided to focus on the line that appears to be doing the most amount of work:

1..$($BytesPerLine - 19) | ForEach-Object { 
             [Void]$Line.Append($(Get-Random -InputObject $ValidChars)) }

The idea behind this line of code is that we add some random characters to each line.  If we want each line to take up 64 characters, then we would add (64 - 19) characters to the line, because the byte marker at the beginning of the line, plus a space character, takes up 17 bytes.  Then then the newline and carriage return takes up 2 bytes.

My first instinct was that the Get-Random action was taking all the CPU cycles.  So I replaced it with static characters... and it made virtually no difference.  Maybe it's the pipe and Foreach-Object?  Let's change it to this:

For ($X = 0; $X -LT ($BytesPerLine - 19); $X++)
{
    [Void]$Line.Append($(Get-Random -InputObject $ValidChars))
}

And now the results:

Measure-Command { New-DepthGaugeFile -FilePath C:\testfolder1\largefile.log `
                                     -DesiredSize 1MB -BytesPerLine 128 }

TotalSeconds    : 61.0464638

Exact same output, almost twice as fast.

Certificate Store Backup and Cleanup (With a Little Powershell)

I was thinking about HTTPS packet inspection the other day.  The type of HTTPS packet inspection that could be performed with a product such as Forefront Threat Management Gateway, for instance.  Basically, you start by funneling everyone's network traffic through your gateway.  Second, you install an SSL certificate into the Trusted Root CA certificate store on all of the client computers whose encrypted traffic you wish to inspect. Now, your gateway is ready to act as a "man-in-the-middle" and decrypt everyone's outbound traffic.  Traffic to their personal email accounts on Gmail and Hotmail... traffic to their online banking websites... transparently, without their knowledge.

I.T. departments do this to their employees all the time.  But I wonder... if it's so easy for I.T. departments, what would stop something like a government agency from installing this same sort of gateway in an ISP datacenter, and sniffing everyone's HTTPS traffic?

If only the government had some way of getting a trusted CA certificate into the cert store on everyone's computer...


So anyway, this thought led me to think about cleaning up my own certificate stores on my personal machines.  We trust so many certificates by default, and we don't really know what they all are or where they came from.  Most of us who use Windows just rely on Microsoft's Windows Root Certificate Program to tell us which root CAs we should trust by default.

So first, we need to turn off Automatic Root Certificates Update via Group Policy (if you administer an Active Directory domain) or via local security policy if you are on a standalone PC:

Computer Configuration > Administrative Templates > System > Internet Communication settings > Turn off Automatic Root Certificates Update


If you leave this setting turned on, Windows will use the internet to re-download and replace any root CA certificates that it thinks you should trust.  So you'll delete them, and they'll just reappear.

Secondly, there are many certificates in your certificate stores for a reason.  Many of them are there for verifying signed code, such as kernel drivers, for example. If Windows cannot verify their digital signatures, they won't load. Windows might not even boot properly.  Nevertheless, Microsoft continues to ship some very old certificates with Windows, such as these:


They're there for "backwards compatibility," of course.  I decided I'd take the risk that I didn't need certificates from the 20th century anymore, and figured I would delete them.

Then there's this guy:


On the list of organizations that engender warm, fuzzy feelings of implicit trust, this one is pretty much at the rock bottom of that list... right above the Nigerian pirate running a CA on his laptop.  Nevertheless, this is one of those root certificate authority certs that is protected and automatically distributed by the Windows Root CA Program.  But since we've disabled the automatic certificate update, and I don't feel like I should be compelled to trust this certificate authority, it's time to delete it.

But, one last thing before we delete the certificates.  Let's make a backup of all of our certificate stores, so that if we accidentally delete a certificate that's required for something important, we can restore it.  It took about 15 minutes of Powershell to write the backup script.  With another 5 minutes, you could sprinkle a little decoration on it and make a cmdlet out of it.

# Backup-CertificateStores.ps1
Set-StrictMode -Version Latest
[String]$CertBackupLocation = "C:\Users\Ryan\Documents\CertStoreBackup_$([Environment]::MachineName)\"

If (-Not (Test-Path $CertBackupLocation))
    { New-Item $CertBackupLocation -ItemType Directory | Out-Null }

$AllCerts = Get-ChildItem Cert:\ -Recurse | Where PSIsContainer -EQ $False

Foreach ($Cert In $AllCerts)
{
    [String]$StoreLocation = ($Cert.PSParentPath -Split '::')[-1]    
    If (-Not (Test-Path (Join-Path $CertBackupLocation $StoreLocation)))
        { New-Item (Join-Path $CertBackupLocation $StoreLocation) -ItemType Directory | Out-Null }

    If (-Not $Cert.HasPrivateKey -And -Not $Cert.Archived)
    {
        Export-Certificate -Cert $Cert -FilePath ([String](Join-Path (Join-Path -Path $CertBackupLocation $StoreLocation) $Cert.Thumbprint) + '.crt') -Force
    }
}

Now you have backups of all your public certificates (this doesn't back up your private certs or keys,) so delete whichever ones you feel are unnecessary.

Modifying Permissions on Windows Services Pt I

I'm going to jot down some quick notes on modifying the permissions on Windows services, because I don't think I have written anything about it here before.

Many times, we find ourselves wanting to delegate some administrative activity on a server to another admin or group of admins, but we don't want to give them full administrative control over the entire server.  We need to delegate only specific activity.  For example, we might want to give our delegated users the right to stop, start and restart only a specific Windows service.  Modifying the ACL on a Windows service is a little more involved than modifying the ACL on a file or folder, though.

You can do this with Group Policy if it's a domain-joined machine.


Group Policy System Services

If the computer is not domain joined or if you only want to do this with the local security policy of one or two computers, you can also accomplish this task using Security Templates on the local computer:


Local Security Templates

You can also use the sc.exe utility:


sc sdshow and sc sdset

The sc sdshow servicename command displays the access control list of the Windows service, in SDDL (security descriptor definition language) format.

The SDDL string looks crazy at first, but it’s pretty simple after you analyze it for a second. There is a D: part, and an S: part. The D: part stands for Discretionary ACL. This is what we usually think of when we think of an ACL on a file, etc. The S: part is the system ACL that is used for things like object access auditing, and is not usually modified as much or thought about as much as the DACL.

With the second command, I am setting the new ACL on the service with sc sdset. I have inserted one Access Control Entry into the D: part of the ACL, before the S: part. The SID I specified is of a non-administrative user. I would recommend creating a security group called “IIS Delegated Administrators” or something like that, and using the SID of that security group. I have granted that account the RP, WP, and DT privileges. (Start service, stop service, and pause service.)  The A stands for Allow, as opposed to a Deny ACE.  And different types of objects such as services, files, MSDTC components, etc., all have slightly different rights strings.  In other words, the "RP" right means something different for a Directory Service object than it does for a Windows service.  Here are the rights strings for Windows services:

CC      SERVICE_QUERY_CONFIG

DC      SERVICE_CHANGE_CONFIG

LC      SERVICE_QUERY_STATUS

SW      SERVICE_ENUMERATE_DEPENDENTS

RP      SERVICE_START

WP      SERVICE_STOP

DT      SERVICE_PAUSE_CONTINUE

LO      SERVICE_INTERROGATE

CR      SERVICE_USER_DEFINED_CONTROL

SD      _DELETE

RC      READ_CONTROL

WD      WRITE_DAC

WO      WRITE_OWNER

You can find a lot more here.

Supersymmetry Outlook Add-In v1.1.3.17

You can see the original Supersymmetry 1.0 post here.

The Supersymmetry Outlook Add-In has been upgraded to version 1.1.3.17.  (Similarity to "31337" or "1337" is coincidental!) But this release is significantly cooler than 1.0 anyway.

In case you're not familiar, the purpose of the Supersymmetry Outlook Add-In is to prevent you from accidentally sending email messages in Outlook if they contain an uneven or unmatched pair of quotation marks, parentheses, curly braces or square brackets.  Read more about it and see more early screenshots in the 1.0 post I linked to earlier.

Improvements in this release include:

  • A new text file named ignore.txt should be placed at %USERPROFILE%\Supersymmetry\ignore.txt.  In this text file, the user can put character sequences ("tokens") for Supersymmetry to ignore in a message.  This is very handy for ignoring things like emoticons, because emoticons usually have parentheses in them, and the use of emoticons can make the message as a whole appear as though it contains an uneven number of parentheses, when the message is actually fine if you ignore the emoticons. A sample ignore.txt is included in the download package.
  • A new text file named divider.txt should be placed at %USERPROFILE%\Supersymmetry\divider.txt.  In this text file, the user can put a character or string token that splits or divides an email thread into pieces, and Supersymmetry will only scan up to the first occurrence of that token. The token or divider can be a single character, or a special string.  This is very useful so that Supersymmetry does not include all of the previous messages in the email thread during its scan.  I recommend putting your special delimiter character or string cleverly in your email signature for both new messages and replies, so that Outlook automatically inserts the token into every message you draft.  I'll leave that up to you.  A sample divider.txt is included in the download package.
  • If either of the two aforementioned files does not exist in the right place on the file system, the add-in will still work, but you will really miss those features.  Additionally, you will see some warning notifications pop up when Outlook loads the add-in:

Supersymmetry cannot locate its files

  • Notice that the recommended installation directory has changed from %APPDATA%\Supersymmetry to %USERPROFILE%\Supersymmetry, just because it's a little simpler.  If the add-in is installed correctly, and the configuration files were read successfully, you'll see these popups instead:

Supersymmetry loaded correctly

Installation

Download the package here:
Supersymmetry-v1.1.3.17.zip (296.8KB)

Make sure to Uninstall any old version of Supersymmetry first, by going to Programs and Features in your Control Panel and uninstalling Supersymmetry.  Also, make sure you've closed Outlook.  Then, unzip the package into your user profile directory, so that %USERPROFILE%\Supersymmetry exists and contains a file named setup.exe.  Next, run setup.exe.  In theory, that should help you download any required prerequisites such as .NET 4.5, and Visual Studio Tools for Office, and then install the add-in.  It's a "ClickOnce" deployment.  A great idea, when it works.

Uninstallation

Simply go to Programs and Features (aka Add/Remove Programs) in Control Panel, find Supersymmetry, and uninstall it.

Have fun!

Supersymmetry in action!

Supersymmetry Outlook Add-In v1.0

Update Oct. 4th, 2014: You want the updated version of this addin, v1.1.3.17!

Like millions of others, I use Outlook as an email client, especially at work.  I was drafting an email at work the other day, and after quickly proofreading it, I sent it out.  Only after sending it, of course, did I spot an error.  I had used a parenthesis to start a parenthetical clause (like this,) only I forgot to use the accompanying closing parenthesis at the end of the statement, so it came out (like this.

I realized that I do this quite a bit in my writing, particularly when I'm rapid-firing work emails.  And not just with parentheses, but also with quotation marks, and occasionally curly braces and square brackets.  There are no red squiggly underlines for this and spellcheck won't help you here.

So I wrote an Outlook 2013 Add-In that will catch me if I attempt to send an email that contains an unmatched set of quotation marks, parentheses, curly braces or square brackets.  Notice the popup when I hit the Send button:

Email draft with a mistake in it


It requires Outlook 2013, .NET 4.5, and Windows Vista or later. It should work on both 32-bit and 64-bit machines, though I didn't test it on 32-bit. You may need to install Visual Studio 2010 Tools for Office Runtime, depending on whether you already installed it when you installed Microsoft Office or not.  If you download the package, and your computer already recognizes the *.vsto file extension, then you probably already have the necessary VSTO runtime installed.  On my development machine, I had to uninstall VSTO, delete the "C:\Program Files\Common Files\Microsoft Shared\VSTO" directory, then reinstall VSTO, or else I got an error when trying to install the add-in.  However, on a fresh test machine that never had Visual Studio installed and only had MS Office installed, I did not get the error and only needed to double-click the *.vsto file and everything worked.

Installation

Download the ZIP archive here:

Supersymmetry-v1.0.zip (49.5KB)

Unpack the ZIP archive somewhere... I chose %APPDATA%\Supersymmetry because that's a good place to put per-user add-ins that doesn't require administrator privileges to write to.  Once you have unzipped the files to a directory, double-click the Supersymmetry.vsto file.

I signed the manifest using a code signing certificate that chains up to the Baltimore CyberTrust Root CA.

Publisher Has Been Verified

You may or may not have the certificate chain in your trusted CAs store.  If you would rather compile the code yourself, send me an email and I will just send you the source code.  The source code is so stupid-simple that I don't feel it deserves a Github repo.  Getting Visual Studio set up just right and figuring out the idiosyncrasies of "ClickOnce" deployment was way more involved than actually writing the code.

Uninstallation

Just go to "Programs and Features" in the Control Panel and click Supersymmetry from the list and click Uninstall.

Limitations

When you click the Send button on an email, the add-in currently scans the entire previous thread embedded with the message, not just the part that you just typed.  That means that the add-in will catch quotation mark and parentheses mistakes that other people made earlier on in the email thread, in addition to your own.  When I think of the best way to filter out these older original messages, I will add that to version 1.1.

Update Oct. 4th, 2014: You want the updated version of this addin, v1.1.3.17!