DNS 101: Round Robin (Or Back When I was Young And Foolish Part II)

I learned something today. It's something that made me feel stupid for not knowing. Something that seemed elemental and trivial - yet, I did not know it. So please, allow me to relay my somewhat embarrassing learning experience in the hopes that it will save someone else from the same embarrassment.

I did know what DNS round robin was. Or at least, I would have said that I did.

Imagine you configure DNS1, as a DNS server, to use round robin. Then, you create 3 host (A or AAAA) records for the same host name, using different IPs. Let's say we create the following A records on DNS1:

server01 - A 10.0.0.4
server01 - A 10.0.0.5
server01 - A 10.0.0.6

Then on a workstation which is configured to use DNS1 as a DNS server, you ping server01. You receive 10.0.0.4 as a reply. You ping server01 again. With no hesitation, you get a reply from 10.0.0.4 again. We assume that your local workstation has cached 10.0.0.4 locally and will reuse that IP for server01 until the entry either expires, or we flush the DNS cache on the workstation with a command like ipconfig/flushdns.

I run ipconfig/flushdns. Then I ping server01 again.

This time I receive a response from 10.0.0.5. Now I assume DNS round robin is working perfectly. I go home for the day feeling like I know everything there is to know about DNS.

But was it that the DNS server is responding to DNS queries with the single next A/AAAA record that it has on file, in a round-robin type sequential fashion to every DNS query that it receives? That is what I assumed.

But the fact of the matter is that DNS servers, when queried for a host name, actually return a list of all A/AAAA records associated with that host name, every time that host name is queried for. (To a point - the list must fit within a UDP packet, and some firewalls/filters don't let UDP packets longer than 512 bytes through. That's changing though. Our idea of how big data is and should be allowed to be is always growing.)

I assume that www.google.com, being one of the busiest websites in the world, has not only some global load balancing and other advanced load balancing techniques employed, but probably also has more than one host record associated with it. To test my theory, I fire up Wireshark and start a packet capture. I then flush my local DNS cache with ipconfig/flushdns and then ping www.google.com.

Notice how I pinged it, got one IP address in response (.148), then flushed my DNS cache, pinged it again and got another different IP address (.144)? But despite what it may look like, that name server is not returning just one A/AAAA record each time I query it:


*Click for Larger*

My workstation is ::9. My workstation's DNS server is ::1. The DNS server is configured to forward DNS requests for zones for which it is not authoritative on to yet another DNS server. So I ask for www.google.com, my DNS server doesn't know, so it forwards the request. The forwardee finally finds out and reports back to my DNS server, which in turn relays back to me a list of all the A records for www.google.com. I get a long list containing not only a mess of A records, but a CNAME thrown in there too, all from a single DNS query! (We're not worried about the subsequent query made for an AAAA record right now. Another post perhaps.)

I was able to replicate this same behavior in a sanitary lab environment running a Windows DNS server and confirmed the same behavior. (Using the server01 example I mentioned earlier.)

Where round robin comes in is that it rotates the order of the list given to each subsequent client who requests it. Keep in mind that while round robin-ing the A records in your DNS replies does supply a primitive form of load distribution, it's a pretty poor substitute for real load balancing, since if one of the nodes in the list goes down, the DNS server will be none the wiser and will continue handing out the list with the downed node's IP address on it.

Lastly, since we know that our client is receiving an entire list of A records for host names which have many IP addresses, what does it actually do with the list?  Well, the ping utility doesn't do much. If the first IP address on the list is down, you get a destination unreachable message and that's it. (Leading to a lot of people not realizing they have a whole list of IPs they could try.) Web browsers however, have a nifty feature known as "browser retry" or "client retry," where they will continue trying the other IPs in the list until they find a working one. Then they will cache the working IP address so that the user does not continue to experience the same delay in web page loading as they did the first time. Yes, there are exploits concerning this feature, and yes it's probably a bad idea to rely on this since browser retry is implemented differently across every different browser and operating system. It's a relatively new mechanism actually, and people may not believe you if you tell them. To prove it to them, find (or create) a host name which has several bad IPs and one or two good ones. Now telnet to that hostname. Even telnet (a modern version from a modern operating system) will use getaddrinfo() instead of gethostbyname() and if it fails to connect the first IP, you can watch it continue trying the next IPs in the list.

More info here, here and here. That last link is an MSDN doc on getaddrinfo(). Notice that it does talk about different implementations on different operating systems, and that ppResult is "a pointer to a linked list of one or more addrinfo structures that contains response information about the host."

Das Keyboard

I've been using computers for a long time, but I never really put any thought into the keyboard I used. I had never used a keyboard that was worth more than about $10, because what's the point, right?

Well I was hanging out in the ##/r/sysadmin IRC chatroom when one of the keys on my latest $10 keyboard started sticking, and it occurred to me that instead of just turning the keyboard upside down and shaking it, hoping it was just a stray Dorito flake stuck in there again, it might be time to finally get a serious keyboard. Luckily there was no shortage of opinions on "good" keyboards there in the chatroom, and one in particular really caught my eye: The DasKeyboard.

It's a mechanical keyboard, and I had heard the buzz about mechanical keyboards - that the tactile feel, durability and key actuation was unparalleled, but on the other hand they were loud. (*CLACKCLACKCLACK*) So with that in mind, I ended up buying the Das Keyboard Model S Ultimate Silent, understanding that it was slightly silent-er than the regular version, but was still pretty loud.

UPS dropped it off a couple days later. I've been using it for about a week now. So without any further ado, let me just get down to exactly what I really love and don't love about this keyboard:

Pros:

  • Very heavy. This keyboard will stay where you put it. Feels solid and high quality. The keys don't rattle around in their sockets.
  • The blue LEDs for caps lock, num lock, and scroll lock are pleasantly dim and not distractingly bright. Really bright LEDs annoy me.
  • No key inscriptions. You might think that it is too hipster and/or nerdy to have a blank keyboard. But I'm truly urging you to try it. Even if you've been using PCs for 30 years, you don't realize how often you unknowingly glance at the keyboard as a subconscious crutch, and having a blank keyboard will increase your touch-typing skill and typing speed. (Can you reliably hit the % character or { character just by touch? Are you confident enough to hit the & character without looking and losing your stride mid-sentence? You will be after about a week of where looking doesn't help.) Plus it'll intimidate people to stay off your computer if they don't know what they're doing.
  • Built-in USB ports so you can plug in another USB keyboard into your keyboard once you realize that you can't type without letters on the keys.
  • Long, thick USB cord, with additional plug for a USB hub.
  • Feels great to type on. The keys are lightly textured and they spring back extremely quickly after you press them - which is what makes extremely fast typing so possible on this keyboard. There's no mushiness or uncertainty in your key presses. There is nothing like a mechanical keyboard for both fast typing and video games. Those Korean kids that play Starcraft tournaments and register 42 billion keystrokes per minute? Mechanical keyboards is how they do that.
  • The company is based in my home state of Texas!
  • N-key rollover (push as many keys at once as you want, the computer will register them all. Only mechanical keyboards can do this.)
  • Tactile feedback and not having to completely depress each key to get it to register. This means you can move on to the next key faster.
Cons:
  • It's loud. Even though the "silent" version is slightly less so, it's still the loudest keyboard I've ever owned. That said, you may grow fond of the satisfying sound of typing. If I were a film director, I would use this keyboard as a prop for when I want the audience to know that someone is typing.
  • It's expensive. However, take into account that this extremely heavy-duty keyboard will last for years, and is arguably the best keyboard that money can buy.
  • I am experiencing a little bit of stickiness on my left shift key when I strike it near its right edge with my left pinky finger. If I strike it more towards the center of the key it works perfectly.  I've only used the keyboard for about a week though, so I'm hoping that the left shift key either smooths out a little over time, or I subconsciously train my pinky finger to stretch out a little further during typing to strike the shift key more towards its center.  The backspace key is also a bit squeaky.  The rest of the keys are perfect
So all in all, I am quite pleased with my purchase.  This is by far the best keyboard I've ever laid hands on. The transition to no letters on the keys was much smoother than I thought and I never once found myself missing them. Furthermore, my typing speed truly has increased already.
 
One last interesting thing is that when using this keyboard, it tricks my mind via optical illusion, as sometimes I could swear I can see faint letter inscriptions on the keys because my mind is expecting to see letters on the keys. Spooky!
 
edit: I am happy to report that the left shift key did in fact smooth out over time with a couple weeks of use.  In fact, all of the keys are getting a little smoother and becoming more perfectly tuned to my fingers. My love of this keyboard is still growing.

Using Powershell to Monitor Windows Reliability Data

There's always a lot of talk about monitoring when stuff gets installed and uninstalled on a Windows machine, or when "configuration changes" take place on a system, or even when unplanned reboots (crashes) take place... how do we audit that? As awesome as the Windows event logs are, they can be a bit unwieldy to sift through all the noise and cryptic messages.

There are lots of third-party tools for auditing software changes. Those tools can cost a lot of money. But did you know Windows already does this for you? If you run perfmon /rel on your Vista/7/2008/R2 machine, you will be greeted with this pretty picture:

Notice that you can even export all the data as a nice little XML file. So that's pretty neat. You can see all the application crashes, system crashes, when software was installed and uninstalled, etc... but that's all GUI stuff. I know what really you want is something more programmatic, customizable and automatable. It just so happens that there's a WMI class called Win32_ReliabilityRecords. Let's use Powershell to take a peek:

# Looking at Windows software installations and uninstallations and other reliability data
# Ryan Ries, Jan 5 2012
#
# Usage: .\ReliabilityData.ps1 <argument>
# Valid arguments are "ShowAll", "ShowSystemCrashes", "ShowWhateverYourImaginationIsTheLimit", ...
# Arguments are not case sensitive.

param([parameter(Mandatory=$true)]
      [string]$Argument)

Function WMIDateStringToDateTime([String] $strWmiDate) 
{ 
    $strWmiDate.Trim() > $null 
    $iYear   = [Int32]::Parse($strWmiDate.SubString( 0, 4)) 
    $iMonth  = [Int32]::Parse($strWmiDate.SubString( 4, 2)) 
    $iDay    = [Int32]::Parse($strWmiDate.SubString( 6, 2)) 
    $iHour   = [Int32]::Parse($strWmiDate.SubString( 8, 2)) 
    $iMinute = [Int32]::Parse($strWmiDate.SubString(10, 2)) 
    $iSecond = [Int32]::Parse($strWmiDate.SubString(12, 2)) 
    $iMicroseconds = [Int32]::Parse($strWmiDate.Substring(15, 6)) 
    $iMilliseconds = $iMicroseconds / 1000 
    $iUtcOffsetMinutes = [Int32]::Parse($strWmiDate.Substring(21, 4)) 
    if ( $iUtcOffsetMinutes -ne 0 ) 
    { 
        $dtkind = [DateTimeKind]::Local 
    } 
    else 
    { 
        $dtkind = [DateTimeKind]::Utc 
    } 
    return New-Object -TypeName DateTime -ArgumentList $iYear, $iMonth, $iDay, $iHour, $iMinute, $iSecond, $iMilliseconds, $dtkind 
} 

If($Argument -eq "ShowAll")
{
       $reliabilityData = Get-WmiObject Win32_ReliabilityRecords
       ForEach ($entry in $reliabilityData)
       {
              Write-Host "Computer Name: " $entry.ComputerName
              Write-Host "Event ID:      " $entry.EventIdentifier
              Write-Host "Record Number: " $entry.RecordNumber
              Write-Host "Date and Time: " $(WMIDateStringToDateTime($entry.TimeGenerated))
              Write-Host "Source:        " $entry.SourceName
              Write-Host "Product Name:  " $entry.ProductName
              Write-Host "User:          " $entry.User
              Write-Host "Message:       " $entry.Message
              Write-Host " "
       }
}

If($Argument -eq "ShowSystemCrashes")
{
       $reliabilityData = Get-WmiObject Win32_ReliabilityRecords
       ForEach ($entry in $reliabilityData)
       {
              If($entry.Message.StartsWith("The previous system shutdown") -And $entry.Message.EndsWith("was unexpected."))
              {
                     Write-Host "Computer Name: " $entry.ComputerName
                     Write-Host "Event ID:      " $entry.EventIdentifier
                     Write-Host "Record Number: " $entry.RecordNumber
                     Write-Host "Date and Time: " $(WMIDateStringToDateTime($entry.TimeGenerated))
                     Write-Host "Source:        " $entry.SourceName
                     Write-Host "Product Name:  " $entry.ProductName
                     Write-Host "User:          " $entry.User
                     Write-Host "Message:       " $entry.Message
                     Write-Host " "             
              }
       }
}

If($Argument -eq "ShowApplicationInstalls")
{
       $reliabilityData = Get-WmiObject Win32_ReliabilityRecords
       ForEach ($entry in $reliabilityData)
       {
              If($entry.Message.StartsWith("Windows Installer installed the product."))
              {
                     Write-Host "Computer Name: " $entry.ComputerName
                     Write-Host "Event ID:      " $entry.EventIdentifier
                     Write-Host "Record Number: " $entry.RecordNumber
                     Write-Host "Date and Time: " $(WMIDateStringToDateTime($entry.TimeGenerated))
                     Write-Host "Source:        " $entry.SourceName
                     Write-Host "Product Name:  " $entry.ProductName
                     Write-Host "User:          " $entry.User
                     Write-Host "Message:       " $entry.Message
                     Write-Host " "             
              }
       }
}

So those are just some ideas that I threw together, but is by no means a complete solution. Use that as a starting point, play with the script, expand on it and make it even better! And one last thing, ideally I should not be using Write-Host here but instead be preserving the objects, that way I could combine this script with other commandlets on the pipeline, etc. I'll put that in as an enhancement request...

Windows XP, RDC7, Trusted Publishers, and You

Someone asked me for some help yesterday with a problem they were having at work. At their company they use Windows XP workstations, a 2003 Active Directory infrastructure, and *.rdp files that the employees use to establish remote connections to other servers. XP was pretty nice when it came out, but today it's old and just not exciting anymore. Same goes with Server 2003. I mean Windows 7 and 2008 R2 are both several years old by now and definitely proven technologies... but still, upgrading to a modern OS seems to be at the bottom of almost every company's list. Desktop admins around the globe are still puttering about supporting employees on WinXP, and server admins all over the world are still logging on to Server 2003 (or worse!) servers.

With the release of Windows 7 came the new Remote Desktop Client 7, which adds some nice new features, and supports the new and interesting Group Policies that come with a 2008+ Active Directory. One such Group Policy is "Specify SHA1 thumbprints of certificates representing trusted .rdp publishers." Enabling this setting allows you, as the administrator, to specify a list of SHA1 hashes that represent certificates that are from what are considered trusted sources.  When the recipient of this policy launches an *.rdp file, and it's signed by a trusted certificate whose hash is on the list, the user will not get prompted with a warning. When you locate this setting in the (2008 and above) GPO editor, it plainly states that this policy is for "Windows Vista SP1 and above." The thing is, you can install RDC7 on Windows XP.

Here's the rest of the detail on the GPO settings from Technet: http://technet.microsoft.com/en-us/library/cc771261(WS.10).aspx.

Furthermore, a signed *.rdp file will have these two lines at the end:

signscope:s:Full Address,Server
signature:s:THISISANSHA1THUMBPRINT

The problem is that the aforementioned Group Policy setting doesn't exist on 2003 Domain Controllers.

Nevertheless, the effect of the newer 2008 policy should still work since we've installed the new RDC7 client on the Windows XP machines. In theory. We just have to figure out how to deploy it. As it turns out, you can just navigate yourself to this registry key on the client:

HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client\PublisherBypassList

In Windows XP the PublisherBypassList key might not exist. Create it! Your SHA1 hashes go there as 32-bit dwords, no spaces, all caps. (This could be done in either HKLM or HKCU. The hashes in HKCU are just added onto the ones loaded from HKLM... just like the description of the GPO setting says.)

So even though you don't have that GPO Setting in Server 2003 like you do in 2008, you can push generic registry modifications such as this out to clients, thereby achieving the same effect.

And it works!

Replacing Task Manager with Process Explorer

Alright, everyone's back from the holidays, and I for one am ready to get my nose back to the grindstone!

In this post, I want to talk about a fairly recent discovery for me: Mark Russinovich's Process Explorer, not to be confused with his Process Monitor. Process Explorer has been around for years and is still being kept current so the fact that I had never really used it before now is a bit of an embarrassment for me, but I'm a total convert now and won't go without it from now on. Hopefully I'll be able to convert someone else with this post.

First, there were two videos that I watched recently that were instrumental in convincing me to keep Process Explorer permanently in my toolbox. It's a two-part series of talks given by Russinovich himself about the "Mysteries of Windows Memory Management." The videos are a little on the technical side, but they're extremely detailed and in-depth and if you're interested in hearing one of the top NT gurus in the world explicate the finer intricacies of how Windows uses physical and virtual memory, then you need to watch these videos. They're quite long, so you may want to save them for later:

Part 1
Part 2

One of the prevailing themes in the videos is that Russinovich doesn't seem to care much for the traditional Task Manager. We all know and love taskmgr and the three-fingered salute required to bring it up. (The three-fingered salute, CTRL+ALT+DEL, is officially referred to as the Secure Attention Sequence. Some free trivia for you.) He explains how some of the labels in Task Manager - especially the ones concerning memory usage - are a bit misleading and/or inaccurate. (What is memory free versus memory available?) He then shows us how he uses Process Explorer in lieu of Task Manager, which gives us a much clearer and more accurate (and cooler looking) picture of all the processes running on the machine, the memory that they're using, the ways in which they're using it, the handles, DLLs and files the processes are using, and so much more.

It's basically better than the regular Windows Task Manager in every way... and the best part? You can easily "replace" Task Manager with it such that when you hit Ctrl+Alt+Del and choose to bring up the "Task Manager," Process Explorer actually launches instead!

Awesome, right? Process Explorer provides an enormous wealth of information where the vanilla Task Manager falls short. Part of me wants to post more screen shots of this program to show you more examples of what you can see and do with Process Explorer, but those videos by Russinovich himself do a better job of showing off exactly how the program works and what all of it means than I can. In the videos, you'll learn what a Working Set is, Private Bytes, Bytes Committed, what a Hard Fault is and how it differs from a Soft Fault, etc.

And not to mention that as an added bonus, you can use this tool to troubleshoot the age-old conundrum of "what process is holding this file open so that I'm unable to delete it! Waaah!"

Needless to say, that if you ever hit Ctrl+Alt+Del on one of my machines and hit Start Task Manager, Process Explorer is going to show up instead.