WTF is a GitHub?

I got my start in IT doing solely operations stuff: swapping out bad hard drives, terminating my own CAT5 cables, (I still carry a TIA-568 wiring diagram in my wallet in case I forget... had to throw out the condom to make room for it though, because let's be honest - I'll be using one way more than the other, just kidding, btw,) administering Active Directory, doing file server migrations... things like that.  And I still do mostly operations stuff today.  But I've also been playing around with programming since I was 14 years old, and I still do, even though my current job doesn't usually call for it.  Most of the time coding is just a hobby for me, but when there's a specific problem at work that I think I can solve with a little of my own programming elbow-grease, I'm all over it.  So I guess that makes me a natural participant in the DevOps movement, which is interesting because the age-old friction between IT pros and developers is still present.  On one hand, I get the idea of "don't half-ass two things; whole-ass one thing."  I think something that will ease that tension between Dev and Ops is the realization that developers aren't just developing boxed products anymore, but services.  Everything's a perpetual service now.  Which means the code and the hardware running that code have to evolve together now, at the same time and with the same goal. (Case in point: The OpenCompute Project, wherein we see entire datacenters being designed around the idea of running these super-scalable cloud workloads.)

Which means the developers and the IT pros need to hold hands and sing Kumbaya.

So Microsoft has been announcing tons of interesting things the past few weeks, such as Powershell 5 with OneGet, the open-sourcing of Roslyn the C# compiler, the introduction of .NET Native, etc.  And that got me into a bit of a mood for software the past few days.  Something I was way behind on was Git and GitHub.  Didn't know what they were, how they worked, how to integrate it into Visual Studio, etc.  So I've taken the past couple of days to educate myself, and the result is that now I have a public GitHub account.  I've only uploaded a few of my old personal projects - I will likely upload way more in the future.  I also have a ton of Powershell stuff built up, that I'll probably just make a single repository for.

Don't be too hard on me if you're going to scrutinize my code. I'm self-taught and I feel like I'm pretty terrible... but maybe I'm my own biggest critic.  I'm like the fat guy at the gym with all the other people with amazing bodies thinking to themselves "well at least he's trying..."

Git was originally designed by Linus Torvalds as a distributed source code revision tool.  GitHub is a place where you can store your repositories, and public code repositories are free!  You use Git to upload source code to GitHub.  Git/Github have established themselves as the premier service of its kind, which is evidenced by the fact that Visual Studio now integrates seamlessly with it and you can push commits to your branches (<- see me speaking the lingo there?) directly from within VS on your desktop.

To Scriptblock or Not to Scriptblock, That is the Question

I was doing some work with the Active Directory Powershell cmdlets recently.  Well, I work with them almost every day, but they still get me with their idiosyncrasies from time to time.

I needed to check some group memberships on various privileged groups within the directory.  I'll show you an abridged version of the code I started with to get the point across, the idea of which is that I iterate through a collection of groups (a string array) and perform some actions on each of the Active Directory groups in sequence:

Foreach($ADGroupName In [String[]]'Domain Admins',     `
                                  'Enterprise Admins', `
                                  'Administrators',    `
                                  'Account Operators', `
                                  'Backup Operators')
{
    $Group = Get-ADGroup -Filter { Name -EQ $ADGroupName } -Properties Members
    If ($Group -EQ $Null -OR $Group.PropertyNames -NotContains 'Members')
    {
        # This condition only occurs on the first iteration of the Foreach loop!
        Write-Error "$ADGroupName was null or was missing the Members property!"
    }
    Else
    {
        Write-Host "$ADGroupName contains $($Group.Members.Count) members." 
    }
}

Before I continue, I'd just like to mention that I typically do not mind very long lines of code nor do I avoid verbose variable names, mostly because I'm always using a widescreen monitor these days.  Long gone are the days of the 80-character-wide terminal.  And I agree with Don Jones that backticks (the Powershell escape character) are to be avoided on aesthetic grounds, but for sharing code in formats that are less conducive to long lines of code, such as this blog or a StackExchange site with their skinny content columns, I'll throw some backticks in to keep the horizontal scrolling to a minimum. 

Anyway, the above code exhibited the strangest bug. At least I'd call it a bug. (I'll make sure and let Jeffrey Snover know next time I see him. ;P) Only on the first iteration of the Foreach loop, I would get the "error" condition instead of the expected "Domain Admins contains 5 members" output.  The remaining iterations all behaved as expected.  It did not matter in what order I rearranged the list of AD groups; I always got an error on the first element in the array.

For a moment, I settled on working around the "bug" by making a "Dummy Group," including that as the first item in the array, gracefully handling the expected exception because Dummy Group did not exist, and then continuing normally with the rest of the legitimate groups.  This worked fine, but it didn't sit well with me.  Not exactly my idea of production-quality code.  I wanted to find the root cause of this strange behavior.

Stackoverflow has made me lazy.  Apparently I go to Serverfault when I want to answer questions, and Stackoverflow when I want other people to answer questions for me.  Simply changing line 7 above to this:

$Group = Get-ADGroup -Filter "Name -EQ '$ADGroupName'" -Properties Members

Made all the difference.  That is, using a string with an expandable variable inside it instead of the script block for a filter.  (Which itself is a little confusing since single quotes (') usually indicate non-expandable variables.  Oh well.  Just syntax to remember when playing with these cmdlets.

Nevertheless, if the code would not work correctly with a script block, I wish the parser would mark it as a syntax error, instead of acting weird.  (Behavior exists in PS 2 and PS 4, though in PS 4 the missing property is just ignored and I get 0 members, which is even worse.)

Powershell, Panchromatic Edition, Continued!

That is a weird title.  Anyway, this post is a continuation to my last post, here, in which I used Powershell to create a bitmap that contained each and every RGB color (24-bit, ~16.7 million colors) exactly once.  We learned that using dynamic arrays and the += operator are often not a good choice when working with large amounts of data that you'd like to see processed before your grandchildren graduate high school. Today we look at another performance pitfall.

So last time, I printed a 4096x4096 bitmap containing 16777216 colors. But the pixels were printed out in a very uniform, boring manner.  I wanted to at least see if I could randomize the pixels a little bit to make the image more interesting.  First, I attempted to do that like this:

Function Shuffle([System.Collections.ObjectModel.Collection[System.Drawing.Color]]$List)
{
    $NewList = New-Object 'System.Collections.ObjectModel.Collection[System.Drawing.Color]'
    While ($List.Count -GT 0)
    {
        [Int]$RandomIndex = Get-Random -Minimum 0 -Maximum $List.Count                
        $NewList.Add($List[$RandomIndex])
        $List.RemoveAt($RandomIndex)
        Write-Progress -Activity "Randomizing Pixels..." -Status "$($NewList.Count) of 16777216"
    }
    Return $NewList
}

Seems pretty solid, right?  I intend to shuffle or randomize the neatly ordered list of pixels that I've generated.  So I pass that neatly ordered list to a Shuffle function.  The Shuffle function randomly plucks an element out of the original list one at a time, inserts it into a new "shuffled" list, then removes the original element from the old list so that it is not reused. Finally, it returns the new shuffled list.

Yeah... so that runs at about 12 pixels per second.

So instead of waiting 16 days for that complete, (16.7 million elements at 12 per second...)  I decided that I had to come up with a better solution.  I thought on it, and I almost resorted to writing a pure C# type and adding that to my script using Add-Type, but then I decided that would be "cheating" since I wanted to write this in Powershell as best I could.

Then it suddenly hit me: maybe I was thinking about it this way too hard.  Let's try something crazy:

Write-Progress -Activity "Randomizing Pixels" -Status "Please wait..."
$RandomPixelList = $AllRGBCombinations | Get-Random -Count $AllRGBCombinations.Count

Done in about two minutes, which beats the hell out of 16 days.  What we have now is a "randomized" list of pixels. Let's paint them and see how it looks:

A slice at 1x magnification:

A slice at 6x magnification:

I call it the "Cosmic Microwave Background."

You'll likely see a third installment in this series as I work some interesting algorithm into the mix so that the image is more interesting than just a random spray of pixels.  Until then...

Powershell Dynamic Arrays Will Murder You, Also... A Pretty Picture! (Part 1 of X)

I was browsing the web a couple days ago and I saw this post, which I thought was a fun idea.  I didn't want to look at his code though, because, kind of like reading movie spoilers, I wanted to see if I could do something similar myself first before spoiling the fun of figuring it out on my own. The idea is that you create an image that contains every RGB color, using each color only once.

Plus I decided that I wanted to do it in Powershell, because I'm a masochist.

There are 256 x 256 x 256 RGB colors (in a 24-bit spectrum.) That equals 16777216 colors. Since each color will be used only once, if I only use 1 pixel per color, a 4096 x 4096 image would be capable of containing exactly 16777216 different colors.

First I thought to just generate a random color, draw the pixel in that color, then add it to a list of "already used" colors, and then on the next iteration, just keep generating random colors in a loop until I happen upon one that wasn't in my list of already used colors. But I quickly realized this would be horribly inefficient and slow.  Imagine: to generate that last pixel, I'd be running through the loop all day hoping for the 1 in ~16.7 million chance that I got the last remaining color that hadn't already been used. Awful idea.

So instead let's just generate a big fat non-random array of all 16777216 RGB colors:

#
$AllRGBCombinations = @()
For ([Int]$Red = 0; $Red -LT 256; $Red++)
{
    For ([Int]$Green = 0; $Green -LT 256; $Green++)
    {
        For ([Int]$Blue = 0; $Blue -LT 256; $Blue++)
        {
            $AllRGBCombinations += [System.Drawing.Color]::FromArgb($Red, $Green, $Blue)
        }
    }
}

That does generate an array of 16777216 differently-colored and neatly-ordered pixel objects... but guess how long it takes?

*... the following day...*

Well, I wish I could tell you, but I killed the script after it ran for about 20 hours. I put a progress bar in just to check that it wasn't hung or in an endless loop, and it wasn't... the code is just really that slow.  It starts out at a decent pace and then gradually slows to a crawl.

Ugh, those dynamic arrays and the += operator strike again. I suspect it's because the above method recreates and resizes the array every iteration... like good ole' ReDim back in the VBscript days.  It may be handy for small bits of data, but if you're dealing with large amounts of data, that you want processed this decade, you better strongly type your stuff and use lists.  Let's try the above code another way:

#
$AllRGBCombinations = New-Object 'System.Collections.ObjectModel.Collection[System.Drawing.Color]'
For ([Int]$Red = 0; $Red -LT 256; $Red++)
{
    For ([Int]$Green = 0; $Green -LT 256; $Green++)
    {        
        For ([Int]$Blue = 0; $Blue -LT 256; $Blue++)
        {
            $AllRGBCombinations.Add([System.Drawing.Color]::FromArgb($Red, $Green, $Blue))
        }
    }
    $PixelsGenerated += 65536
    Write-Progress -Activity "Generating Pixels..." -Status "$PixelsGenerated / 16777216" -PercentComplete (($PixelsGenerated / 16777216) * 100)
}

Only 5.2 seconds in comparison, including the added overhead of writing the progress bar. Notice how I only update the progress bar once every 256 * 256 pixels, because it will slow you down a lot if you try to update the progress bar after every single pixel is created.

Now I can go ahead and generate an image containing exactly one of every color that looks like this:

Yep, there really are 16.7 million different colors in there, which is why even a shrunken PNG of it is 384KB.  Hard to compress an image when there are NO identical colors! The original 4096x4096 bitmap is ~36MB.  And I ended up loading a shrunken and compressed JPG for this blog post, because I didn't want every page hit consuming a meg of bandwidth.

It kinda' makes you think about how limited and narrow a human's vision is, doesn't it?  24-bit color seems to be fine for us when watching movies or playing video games, but that image doesn't seem to capture how impressive our breadth of vision should be.

Next time, we try to randomize our set of pixels a little, and try to make a prettier, less computerized-looking picture... but still only using each color only once.  See you soon.

Verifying RPC Network Connectivity Like A Boss

Aloha.  I did some fun Powershelling yesterday and now it's time to share.

If you work in an IT environment that's of any significant size, chances are you have firewalls.  Maybe lots and lots of firewalls. RPC can be a particularly difficult network protocol to work with when it comes to making sure all the ports necessary for its operation are open on your firewalls. I've found that firewall guys sometimes have a hard time allowing the application guy's RPC traffic through their firewalls because of its dynamic nature. Sometimes the application guys don't really know how RPC works, so they don't really know what to ask of the firewall guys.  And to make it even worse, RPC errors can be hard to diagnose.  For instance, the classic RPC error 1722 (0x6BA) - "The RPC server is unavailable" sounds like a network problem at first, but can actually mean access denied, or DNS resolution failure, etc.

MSRPC, or Microsoft Remote Procedure Call, is Microsoft's implementation of DCE (Distributed Computing Environment) RPC. It's been around a long time and is pervasive in an environment containing Windows computers. Tons of Windows applications and components depend on it.

A very brief summary of how the protocol works: There is an "endpoint mapper" that runs on TCP port 135. You can bind to that port on a remote computer anonymously and enumerate all the various RPC services available on that computer.  The services may be using named pipes or TCP/IP.  Named pipes will use port 445.  The services that are using TCP are each dynamically allocated their own TCP ports, which are drawn from a pool of port numbers. This pool of port numbers is by default 1024-5000 on XP/2003 and below, and 49152-65535 on Vista/2008 and above. (The ephemeral port range.) You can customize that port range that RPC will use if you wish, like so:

reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v Ports /t REG_MULTI_SZ /f /d 8000-9000
reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v PortsInternetAvailable /t REG_SZ /f /d Y
reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v UseInternetPorts /t REG_SZ /f /d Y

And/Or

netsh int ipv4 set dynamicport tcp start=8000 num=1001
netsh int ipv4 set dynamicport udp start=8000 num=1001
netsh int ipv6 set dynamicport tcp start=8000 num=1001
netsh int ipv6 set dynamicport udp start=8000 num=1001

This is why we have to query the endpoint mapper first, because we can't just guess exactly which port we need to connect to for a particular service.

So, I wrote a little something in Powershell that will test the network connectivity of a remote machine for RPC, by querying the endpoint mapper, and then querying each port that the endpoint mapper tells me that it's currently using.


#Requires -Version 3
Function Test-RPC
{
    [CmdletBinding(SupportsShouldProcess=$True)]
    Param([Parameter(ValueFromPipeline=$True)][String[]]$ComputerName = 'localhost')
    BEGIN
    {
        Set-StrictMode -Version Latest
        $PInvokeCode = @'
        using System;
        using System.Collections.Generic;
        using System.Runtime.InteropServices;

        public class Rpc
        {
            // I found this crud in RpcDce.h

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcBindingFromStringBinding(string StringBinding, out IntPtr Binding);

            [DllImport("Rpcrt4.dll")]
            public static extern int RpcBindingFree(ref IntPtr Binding);

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcMgmtEpEltInqBegin(IntPtr EpBinding,
                                                    int InquiryType, // 0x00000000 = RPC_C_EP_ALL_ELTS
                                                    int IfId,
                                                    int VersOption,
                                                    string ObjectUuid,
                                                    out IntPtr InquiryContext);

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcMgmtEpEltInqNext(IntPtr InquiryContext,
                                                    out RPC_IF_ID IfId,
                                                    out IntPtr Binding,
                                                    out Guid ObjectUuid,
                                                    out IntPtr Annotation);

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcBindingToStringBinding(IntPtr Binding, out IntPtr StringBinding);

            public struct RPC_IF_ID
            {
                public Guid Uuid;
                public ushort VersMajor;
                public ushort VersMinor;
            }

            public static List QueryEPM(string host)
            {
                List ports = new List();
                int retCode = 0; // RPC_S_OK                
                IntPtr bindingHandle = IntPtr.Zero;
                IntPtr inquiryContext = IntPtr.Zero;                
                IntPtr elementBindingHandle = IntPtr.Zero;
                RPC_IF_ID elementIfId;
                Guid elementUuid;
                IntPtr elementAnnotation;

                try
                {                    
                    retCode = RpcBindingFromStringBinding("ncacn_ip_tcp:" + host, out bindingHandle);
                    if (retCode != 0)
                        throw new Exception("RpcBindingFromStringBinding: " + retCode);

                    retCode = RpcMgmtEpEltInqBegin(bindingHandle, 0, 0, 0, string.Empty, out inquiryContext);
                    if (retCode != 0)
                        throw new Exception("RpcMgmtEpEltInqBegin: " + retCode);
                    
                    do
                    {
                        IntPtr bindString = IntPtr.Zero;
                        retCode = RpcMgmtEpEltInqNext (inquiryContext, out elementIfId, out elementBindingHandle, out elementUuid, out elementAnnotation);
                        if (retCode != 0)
                            if (retCode == 1772)
                                break;

                        retCode = RpcBindingToStringBinding(elementBindingHandle, out bindString);
                        if (retCode != 0)
                            throw new Exception("RpcBindingToStringBinding: " + retCode);
                            
                        string s = Marshal.PtrToStringAuto(bindString).Trim().ToLower();
                        if(s.StartsWith("ncacn_ip_tcp:"))                        
                            ports.Add(int.Parse(s.Split('[')[1].Split(']')[0]));
                        
                        RpcBindingFree(ref elementBindingHandle);
                        
                    }
                    while (retCode != 1772); // RPC_X_NO_MORE_ENTRIES

                }
                catch(Exception ex)
                {
                    Console.WriteLine(ex);
                    return ports;
                }
                finally
                {
                    RpcBindingFree(ref bindingHandle);
                }
                
                return ports;
            }
        }
'@
    }
    PROCESS
    {
        ForEach($Computer In $ComputerName)
        {
            If($PSCmdlet.ShouldProcess($Computer))
            {
                [Bool]$EPMOpen = $False
                $Socket = New-Object Net.Sockets.TcpClient
                
                Try
                {                    
                    $Socket.Connect($Computer, 135)
                    If ($Socket.Connected)
                    {
                        $EPMOpen = $True
                    }
                    $Socket.Close()                    
                }
                Catch
                {
                    $Socket.Dispose()
                }
                
                If ($EPMOpen)
                {
                    Add-Type $PInvokeCode
                    $RPCPorts = [Rpc]::QueryEPM($Computer)
                    [Bool]$AllPortsOpen = $True
                    Foreach ($Port In $RPCPorts)
                    {
                        $Socket = New-Object Net.Sockets.TcpClient
                        Try
                        {
                            $Socket.Connect($Computer, $Port)
                            If (!$Socket.Connected)
                            {
                                $AllPortsOpen = $False
                            }
                            $Socket.Close()
                        }
                        Catch
                        {
                            $AllPortsOpen = $False
                            $Socket.Dispose()
                        }
                    }

                    [PSObject]@{'ComputerName' = $Computer; 'EndPointMapperOpen' = $EPMOpen; 'RPCPortsInUse' = $RPCPorts; 'AllRPCPortsOpen' = $AllPortsOpen}
                }
                Else
                {
                    [PSObject]@{'ComputerName' = $Computer; 'EndPointMapperOpen' = $EPMOpen}
                }
            }
        }
    }
    END
    {

    }
}

And the output will look a little something like this:


You can also query the endpoint mapper with PortQry.exe -n server01 -e 135, but I was curious about how it worked at a deeper level, so I ended up writing something myself. There weren't many examples of how to use that particular native API, so it was pretty tough.