Neat Windows Tricks (Or Back When I Was Young and Foolish Pt. III)

I work with Windows a lot. Almost every day of my life since Windows 3.1, I've been submersed in Windows for both work and play. Getting to know it inside and out. Learning new Windows applications. Being excited for new releases of Windows because I know it'll bring big changes to the operating system which will make me have to learn new things. Every now and then, I even start to think I'm pretty knowledgable about Windows...

Which is why I'm always flabbergasted when someone non-chalantly shows me a relatively mundane Windows trick I never knew about, yet it's been under my nose the whole time!  And so, I bring you two Windows tricks that I learned about today. If you already knew about them, you can just think of me as an amateur and move on.  But if you didn't already know about them, they might just change the way you do your daily work!

C:\> SomeCommand | clip

 I never knew about this! On the command line, you can pipe the output of any program or command to your Windows clipboard and Ctrl+V it anywhere!


You can also do something like

C:\> clip < textFile.txt to quickly copy the contents of the file to your clipboard.

Alright, trick #2:

Shift + Right-Click to Run as Different User

This one was down-right embarrassing for me to have not known. Maybe I did know it at one time, but then I used it so infrequently that I forgot about it... I don't know. Back in the XP/2003 days, you could right-click on an application and choose "Run as..." which would prompt you for the credentials of another user under which to to run that process. But then in Vista onwards it disappeared. Sure, there's still "Run as administrator," but sometimes you need to run a process as someone else besides Administrator. Well, I always just got around it by launching a command prompt and doing something like runas / notepad.exe.

But I completely forgot that it's still there in the GUI.  All you have to do is Shift+Right-click:


It's a Christmas Miracle!

Well I did something quite embarrassing the other day. But then I did something pretty cool to make up for it. I'll call it a Christmas miracle, because it involves me not knowing the domain admin password, having no other normal way of logging in, despairing for a couple minutes, then getting back in anyway. And also because I'm histrionic like that. It's a real story of tribulation and triumph.

A couple days ago, the domain admin password on one of my lab domains expired unexpectedly, and so I was prompted to reset it upon RDPing to a server. So I did. As a matter of habit, I saved the new password in my password vault software. (I'm one of those people that has a different password for every single account I have, but as a result I can't remember them all.)

At least, I thought I saved the new password.

Fast forward to a week later. I wanted to do some work in that lab again, and as I began the login process, I realized that I had no idea what I had set the password to. The domain admin password. The only Administrative account in that domain. "No problem," I thought. It's in my password vault...

It wasn't.

I went through several backup copies of the password file, thinking that it must have been overwritten. Nope. Dreadful thoughts of spending the whole weekend rebuilding the lab flashed before my eyes.

Then I had an idea.

I switched my monitor over to the physical domain controller, and plugged in a keyboard.

Server 2012 Login

It's a very calm and soothing login screen, isn't it? I like it.

Anyway, there's the ever-present "Accessibility" or "Ease of Access" icon down there in the corner. (I know many of you already know where I'm going with this, and you're right. But if you don't know, read on. Otherwise, you may want to skip to the "Afterthoughts" section below.) This domain controller is running Server 2012 Core. That accessibility button in the corner doesn't do anything. I can click it, but it doesn't do anything. Maybe it's because this is a Server Core installation with no GUI elements. Hm.

I had a Windows installation image on a bootable USB thumb drive handy, so I plugged that in to the domain controller and rebooted the machine (using the power button on the chassis obviously, since I wasn't going to be logging in.)

Once the machine had booted off the installation media, instead of choosing to install Windows, I chose "Repair your computer." From there, I chose the option to launch a Command Prompt. Voila. Sweet. I now had a shell on the computer.

Next I navigated on over to where everything interesting happens, \Windows\System32. (The drives had different drive letters than normal, but who cares.) Now, under regular circumstances, that little Accessibility button on the logon screen launches a program called Utilman.exe. From there you can get all the gizmos like the on-screen keyboard and magnifier and things like that to help you log in, in case your eyesight's not so good or you have no hands or something. But there was no Utilman.exe in the System32 folder. I guess that's why the Accessibility icon didn't do anything when I clicked it. My domain controllers are very inconsiderate of the handicapped.

So I just decided to try the command:

copy X:\Windows\System32\cmd.exe X:\Windows\System32\Utilman.exe

Then I rebooted. Once I was back at the Windows login screen, I hit the Accessibility icon one last time. Command Prompt. And not just any Command Prompt, but a Command Prompt running as Local System! Since this was a normal boot of the domain controller and therefore Active Directory was online, I just did:

net user DomainAdmin *

And I was greeted with a prompt to reset the domain administrator password to whatever I wanted.

This time I made damn sure that the new password was saved and backed up in my password vault.


So it's great that I was able to reset a domain administrator password with nothing but a bootable image and physical access to the machine, but, one can't help but wonder about the glaring security implications of this. Once a person has a Local System shell on a domain controller, they own the domain.

Furthermore, today's enterprise environments are more complex than ever. Having physical access to the console of the machine doesn't necessarily mean you have to actually be able to touch the machine, like I was able to do here. These days, "physical access" can mean logging in to an ILO or accessing a virtual machine management console from a thousand miles away. And are we, as enterprise administrators and IT directors and CTOs, sure that everyone in our organization with access to those things are people we also trust as potential domain admins? It's not just about the threat of outside attackers. We may be unwittingly exposing a massive attack surface to all of our front-line support personnel should they ever decide to go rogue.

Alright, so what should be done about this little "feature" of Windows? First off, Microsoft needs to not launch Utilman.exe as Local System. Launch it as an unprivileged account that doesn't have the ability to do things like reset Administrator passwords. I'm sure there's a technical reason why they think they "have" to run the Ease of Access stuff as Local System, so that it can simulate the secure attention sequence, etc., but I'm not buying it.  Just because it's hard doesn't mean they should get away with letting this gaping security hole go unpatched.  They could also include a hash check in the code behind that little icon so it would at least be a little more difficult to replace the real Utilman.exe with, say, a copy of Cmd.exe that was renamed to Utilman.exe.

To make matters worse, there is no "official" Group Policy setting to disable this button. You might be able to mitigate the risk somewhat by using Group Policy to do something like run a script that cacls Utilman.exe at startup and denies Everyone all access, or having Group Policy push out your own version of Utilman.exe at every boot of the computer... neither of which are ideal solutions nor are they likely to be bulletproof.

Oh well. I guess that goes along with that immutable law of computer security, that if someone else has physical access to your computer, then it's not your computer anymore.

"SSLv3.0/TLSv1.0 Protocol Weak CBC Mode Vulnerability"

Good afternoon, and sorry I haven't posted in a while.  I've been staying pretty busy.

So if you have been in IT or working with servers for very long, you're probably familiar with this guy:

*The most annoying appliance ever?*

So in case you're not familiar, this little guy sits in your datacenter, scanning your network, and spits out reports about all the potential vulnerabilities it finds on all your network devices and servers. Then you get to go fix all of those potential vulnerabilities so that you can maintain PCI compliance and such. Sometimes it's as easy as applying an OS patch. Sometimes it's making an obscure configuration change to an application that is just as likely to break the application as it is to plug the vulnerability.

"SSLv3.0/TLSv1.0 Protocol Weak CBC Mode Vulnerability" was a particularly annoying one. I'm making sure to put the exact title of the vulnerability as Qualys puts it so that maybe someday it will show up in somebody's Internet search and help them.  I wasn't so lucky.  There really wasn't much information out there on this particular vulnerability that applied to me; or so I thought at the time.  It seemed like the only information I could find on this vulnerability either pertained to Linux servers, or particularly to IIS on Windows Servers.  My server was a physical HP machine running Windows Server 2008 R2 with all the HP software installed... and I wasn't the only guy on the team who was hung up on this particular vulnerability.

Qualys will tell you that this vulnerability is tied to CVE-2011-3389 and so my first instinct was to look for the Microsoft-issued security advisory.  The particular Windows patch it was suggesting was already installed, and I didn't even have IIS installed on this server anyway.  This led me down the path of modifying system-wide registry settings like this, to no avail.  The same vulnerability kept showing up on subsequent scans.

So after taking a step back and thinking for a second, it occurred to me that Qualys was reporting this particular vulnerability on port 2381.  That's the port used by the HP System Management Homepage.  (A glorious piece of software... please note the sarcasm.) So maybe there's a configuration change I can make just to the SMH... and after Googling through some HP documentation I found this gem:

C:\hp\hpsmh\bin>smhconfig.exe -Z 'RC4-SHA'

That should restrict the cipher modes that the SMH is allowed to use to only RC4-SHA. (With a capital Z.)  But my version of smhconfig.exe didn't implement the -Z switch, so I updated it via the Proliant Support Pack, and then was able to successfully run the command.

Problem solved. Vulnerability gone.

It was only after I went through all that, I went back to the original CVE-2011-3389 page and noticed this.  :P

DNS 101: Round Robin (Or Back When I was Young And Foolish Part II)

I learned something today. It's something that made me feel stupid for not knowing. Something that seemed elemental and trivial - yet, I did not know it. So please, allow me to relay my somewhat embarrassing learning experience in the hopes that it will save someone else from the same embarrassment.

I did know what DNS round robin was. Or at least, I would have said that I did.

Imagine you configure DNS1, as a DNS server, to use round robin. Then, you create 3 host (A or AAAA) records for the same host name, using different IPs. Let's say we create the following A records on DNS1:

server01 - A
server01 - A
server01 - A

Then on a workstation which is configured to use DNS1 as a DNS server, you ping server01. You receive as a reply. You ping server01 again. With no hesitation, you get a reply from again. We assume that your local workstation has cached locally and will reuse that IP for server01 until the entry either expires, or we flush the DNS cache on the workstation with a command like ipconfig/flushdns.

I run ipconfig/flushdns. Then I ping server01 again.

This time I receive a response from Now I assume DNS round robin is working perfectly. I go home for the day feeling like I know everything there is to know about DNS.

But was it that the DNS server is responding to DNS queries with the single next A/AAAA record that it has on file, in a round-robin type sequential fashion to every DNS query that it receives? That is what I assumed.

But the fact of the matter is that DNS servers, when queried for a host name, actually return a list of all A/AAAA records associated with that host name, every time that host name is queried for. (To a point - the list must fit within a UDP packet, and some firewalls/filters don't let UDP packets longer than 512 bytes through. That's changing though. Our idea of how big data is and should be allowed to be is always growing.)

I assume that, being one of the busiest websites in the world, has not only some global load balancing and other advanced load balancing techniques employed, but probably also has more than one host record associated with it. To test my theory, I fire up Wireshark and start a packet capture. I then flush my local DNS cache with ipconfig/flushdns and then ping

Notice how I pinged it, got one IP address in response (.148), then flushed my DNS cache, pinged it again and got another different IP address (.144)? But despite what it may look like, that name server is not returning just one A/AAAA record each time I query it:

*Click for Larger*

My workstation is ::9. My workstation's DNS server is ::1. The DNS server is configured to forward DNS requests for zones for which it is not authoritative on to yet another DNS server. So I ask for, my DNS server doesn't know, so it forwards the request. The forwardee finally finds out and reports back to my DNS server, which in turn relays back to me a list of all the A records for I get a long list containing not only a mess of A records, but a CNAME thrown in there too, all from a single DNS query! (We're not worried about the subsequent query made for an AAAA record right now. Another post perhaps.)

I was able to replicate this same behavior in a sanitary lab environment running a Windows DNS server and confirmed the same behavior. (Using the server01 example I mentioned earlier.)

Where round robin comes in is that it rotates the order of the list given to each subsequent client who requests it. Keep in mind that while round robin-ing the A records in your DNS replies does supply a primitive form of load distribution, it's a pretty poor substitute for real load balancing, since if one of the nodes in the list goes down, the DNS server will be none the wiser and will continue handing out the list with the downed node's IP address on it.

Lastly, since we know that our client is receiving an entire list of A records for host names which have many IP addresses, what does it actually do with the list?  Well, the ping utility doesn't do much. If the first IP address on the list is down, you get a destination unreachable message and that's it. (Leading to a lot of people not realizing they have a whole list of IPs they could try.) Web browsers however, have a nifty feature known as "browser retry" or "client retry," where they will continue trying the other IPs in the list until they find a working one. Then they will cache the working IP address so that the user does not continue to experience the same delay in web page loading as they did the first time. Yes, there are exploits concerning this feature, and yes it's probably a bad idea to rely on this since browser retry is implemented differently across every different browser and operating system. It's a relatively new mechanism actually, and people may not believe you if you tell them. To prove it to them, find (or create) a host name which has several bad IPs and one or two good ones. Now telnet to that hostname. Even telnet (a modern version from a modern operating system) will use getaddrinfo() instead of gethostbyname() and if it fails to connect the first IP, you can watch it continue trying the next IPs in the list.

More info here, here and here. That last link is an MSDN doc on getaddrinfo(). Notice that it does talk about different implementations on different operating systems, and that ppResult is "a pointer to a linked list of one or more addrinfo structures that contains response information about the host."

Back When I Was Young and Foolish (Part I of Oh-So-Many)

I'd like to do a little reminiscing now, of a time when I was young and foolish. I anticipate many such posts, which I guess means I spend a lot of time being young and foolish.

It was my first real IT job, and it'd be fair to call me a sysadmin. I was basically just an assistant to the lead architect. I was doing things like setting up backup schedules in DPM and creating user accounts in AD. Just the usual stuff you'd expect a guy first getting started in IT would be doing.

One day, my boss and I were discussing one of the many issues caused by being in the middle of a domain migration, and that was that we currently had employees working in two different Windows domains simultaneously. For various reasons, there would not and could not be a trust relationship established between the two. To further complicate matters, we now needed two separate ISA (now known as TMG) servers at the office, one for each domain, so as to keep the employees off of Facebook and One might suggest that it was a social problem that should be dealt with by the employees' managers, but the managers were among the worst offenders. But I digress.

Now since both of these domains are on the same subnet, we can really only have one DHCP server. So if we only have one DHCP server, how are we going to push out two different sets of options to the clients, such as which proxy server to use, to members of either domain?

We settled on using two different DHCP classes.

So while we were fiddling around with the DHCP server, for no good reason we decided that right now was a great time to screw around with the basic settings of a DHCP server that had been serving us faithfully with no problems for years.

What is a good DHCP lease duration, really? Is it the default of 8 days? Well I suppose that depends on a lot of factors, such as how mobile your DHCP clients are and how often you expect them to be connecting to and disconnecting from the network, how many DHCP addresses you have to distribute, etc..  But I've already put more thought into it just now than we did on that day, for we almost completely arbitrarily set the DHCP lease duration to 30 minutes.

Fast forward a few months.

Everything had been working great and the office DHCP server, as DHCP servers should be, was all but forgotten about. I had just been offered a much higher-paying job with a much larger IT company, so I was now a short-timer at this job. Then, out of the blue, our primary domain controller (the one hosting all the FSMOs) at our remote datacenter goes dark.  I can still ILO into it, but both of its network adapters are just gone. No error messages in the logs, no sign that Windows had ever known that it did once have network connections, nothing.  It just looked like it had never had network adapters in the first place.  (This still perturbs me to this day. Why wasn't there an event in either the Windows logs or in the IML that said "Uhh dude where did your network adapters go!?")

After seizing the FSMO roles on the backup DC, we took a trip out to the datacenter to have a look. Sure enough, the link lights on the physical hardware were out, and upon rebooting, the BIOS of the machine didn't even recognize that it had ever had network adapters in it. So I had HP come out and replace the motherboard, which fixed the issue. I don't know what the point of having redundant NICs is if they're wired to both blow at the same time, but I digress again...

So now I have essentially a brand new server out of what used to be our primary domain controller. The question, my boss wanted to know, was how do we restore a former DC after its FSMO roles have been seized? My answer: You don't. You never bring it back. Ever. If I don't wipe the hard drive right now, it thinks it's still the owner of those FSMO roles, and the only way for it to learn that it no longer owns those FSMO roles is to connect it back to the domain network and let the KCC do its magic.  What's going to happen in that span of time that we now have two DCs in the domain that both simultaneously think that they own all the FSMOs? I didn't want to find out. So I argued that we just rebuild it and I eventually got my way.

Now since I had to rebuild the machine, I figured now was as good a time as any to upgrade it from Windows 2008 to Windows 2008 R2. Having spent the last several months studying my ass off for the MCITP tests and passing them, I was feeling pretty self-confident. I took the freshly rebuilt server back out to the datacenter. I re-cabled it and re-racked it, and powered it on. I grabbed the 2008 R2 version of adprep.exe and on the other domain controller, used it to run adprep /forestprep followed by adprep /domainprep. The domain was now ready to accept a 2008 R2 domain controller. I dcpromo'ed the new machine, and it went without a hitch. I then gracefully transferred the FSMOs back from the other DC. (I didn't alter the actual functional levels of the forest or domain.)

Now I had one new, fresh shiny 2008 R2 domain controller in the datacenter, with 5 other DCs in 2 other sites that were still on Windows 2008. Still high off my earlier success, I felt like this was a great time to do an in-place upgrade on the rest of those DCs.  One by one, I upgraded them from 2008 to 2008 R2, and you know what? I was actually pleasantly surprised at how smooth and painless it was. My hat goes off to Microsoft for making an in-place upgrade of a production domain controller so seamless.  I just remoted into the server, mounted the 2008 R2 media and hit "Upgrade."  That's all there was to it. The installation took about 30 minutes, and when the machine came back up, it was a pristine 2008 R2 box.

Everything was going great, until I got around to upgrading the last of the 6 DCs. That sixth DC was the DHCP server at the office. I followed the same procedure that had gone off without a hitch 5 times already. Confident that all was well, I set off to browse reddit while the DC upgrade finished. I was just thinking about how the upgrade was going kind of slow, when suddenly my Internet connection dropped. At the exact same instant, the phone rang.

Me: "Hello?"
Boss: "What did you do?"
Me: "Wha, I don't, I ..."
Boss: "RUN to the server room and set up a new DHCP scope. Now."

The entire office had just lost their Internet connections. In the intervening months, I had completely forgotten that we had set the DHCP lease duration on the domain controller to... oh, just a hair shorter than the amount of time it takes to complete an in-place upgrade on that domain controller. Had the lease time been just 5 minutes longer, the DC/DHCP server would have finished the upgrade and been back up serving renewals as if nothing had ever happened. By the time I had gotten the replacement scope set up on the other DC and done the dance of de-authorizing the first DC as as a DHCP server in Active Directory, and authorizing the new one, the original DC was back up and ready.

After about 30 minutes of repairing the damage, I sheepishly emerged from the server room, to the sound of ironic applause coming from all the employees in their cubicles.