Psst, You Want A Script To Backup Your Lab VMs?

I can hook you up...

So I'm always doing a lot of lab work with Hyper-V virtual machines. Every once in a while I want to just save the state of the entire lab all at once and back it up to a safe storage volume.  I suppose I could set up Windows Server Backup on each of the VMs, and find some disk to use as a pass-through disk for one of the virtual machines and then share that so that the VMs could back up to the network share... but that's a ton of hassle.

How about I just save the state of all the VMs, and export them directly to my backup volume, then resume the VMs, all from the hypervisor?  As a scheduled task, perhaps?

About 10 minutes in the Powershell ISE and I've done just that.  A couple things to be warned of - First, you can't do this in production. The virtual machines are frozen while they're being exported, and it can take several minutes to export a VM. Secondly, make sure you are running with full Administrator privileges, or else cmdlets such as Get-VM will silently return nothing.

 

# Ryan Ries, 2013
# Backs up some lab VMs. Takes several minutes at least.

[String]$BackupPath = "D:\Backups\Hyper-V"
[String]$ErrorLog   = "D:\Backups\Hyper-VBackups.log"
$Start = Get-Date
"$(Get-Date) - Hyper-V Backups Starting." | Out-File $ErrorLog -Append
Try
{
    Get-Childitem $BackupPath -Recurse -Force | Remove-Item -Recurse -Force -ErrorAction Stop
}
Catch
{
    "$(Get-Date) - Error during Get-Childitem or Remove-Item: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Stop-VM -ErrorAction Stop -Save
}
Catch
{
    "$(Get-Date) - Error during Get-VM Stop-VM -Save: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Export-VM -ErrorAction Stop -Path $BackupPath
}
Catch
{
    "$(Get-Date) - Error during Export-VM: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Start-VM -ErrorAction Stop
}
Catch
{    
    "$(Get-Date) - Error during Start-VM: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
$End = Get-Date
"$(Get-Date) - Hyper-V Backups completed in $([Math]::Round((New-TimeSpan $Start $End).TotalMinutes)) minutes." | Out-File $ErrorLog -Append

Neat Windows Tricks (Or Back When I Was Young and Foolish Pt. III)

I work with Windows a lot. Almost every day of my life since Windows 3.1, I've been submersed in Windows for both work and play. Getting to know it inside and out. Learning new Windows applications. Being excited for new releases of Windows because I know it'll bring big changes to the operating system which will make me have to learn new things. Every now and then, I even start to think I'm pretty knowledgable about Windows...

Which is why I'm always flabbergasted when someone non-chalantly shows me a relatively mundane Windows trick I never knew about, yet it's been under my nose the whole time!  And so, I bring you two Windows tricks that I learned about today. If you already knew about them, you can just think of me as an amateur and move on.  But if you didn't already know about them, they might just change the way you do your daily work!

C:\> SomeCommand | clip

 I never knew about this! On the command line, you can pipe the output of any program or command to your Windows clipboard and Ctrl+V it anywhere!

Clip

You can also do something like

C:\> clip < textFile.txt to quickly copy the contents of the file to your clipboard.

Alright, trick #2:

Shift + Right-Click to Run as Different User

This one was down-right embarrassing for me to have not known. Maybe I did know it at one time, but then I used it so infrequently that I forgot about it... I don't know. Back in the XP/2003 days, you could right-click on an application and choose "Run as..." which would prompt you for the credentials of another user under which to to run that process. But then in Vista onwards it disappeared. Sure, there's still "Run as administrator," but sometimes you need to run a process as someone else besides Administrator. Well, I always just got around it by launching a command prompt and doing something like runas /user:BobMarley@domain.com notepad.exe.

But I completely forgot that it's still there in the GUI.  All you have to do is Shift+Right-click:

Shift+Right-click

It's a Christmas Miracle!

Well I did something quite embarrassing the other day. But then I did something pretty cool to make up for it. I'll call it a Christmas miracle, because it involves me not knowing the domain admin password, having no other normal way of logging in, despairing for a couple minutes, then getting back in anyway. And also because I'm histrionic like that. It's a real story of tribulation and triumph.

A couple days ago, the domain admin password on one of my lab domains expired unexpectedly, and so I was prompted to reset it upon RDPing to a server. So I did. As a matter of habit, I saved the new password in my password vault software. (I'm one of those people that has a different password for every single account I have, but as a result I can't remember them all.)

At least, I thought I saved the new password.

Fast forward to a week later. I wanted to do some work in that lab again, and as I began the login process, I realized that I had no idea what I had set the password to. The domain admin password. The only Administrative account in that domain. "No problem," I thought. It's in my password vault...

It wasn't.

I went through several backup copies of the password file, thinking that it must have been overwritten. Nope. Dreadful thoughts of spending the whole weekend rebuilding the lab flashed before my eyes.

Then I had an idea.

I switched my monitor over to the physical domain controller, and plugged in a keyboard.

Server 2012 Login

It's a very calm and soothing login screen, isn't it? I like it.

Anyway, there's the ever-present "Accessibility" or "Ease of Access" icon down there in the corner. (I know many of you already know where I'm going with this, and you're right. But if you don't know, read on. Otherwise, you may want to skip to the "Afterthoughts" section below.) This domain controller is running Server 2012 Core. That accessibility button in the corner doesn't do anything. I can click it, but it doesn't do anything. Maybe it's because this is a Server Core installation with no GUI elements. Hm.

I had a Windows installation image on a bootable USB thumb drive handy, so I plugged that in to the domain controller and rebooted the machine (using the power button on the chassis obviously, since I wasn't going to be logging in.)

Once the machine had booted off the installation media, instead of choosing to install Windows, I chose "Repair your computer." From there, I chose the option to launch a Command Prompt. Voila. Sweet. I now had a shell on the computer.

Next I navigated on over to where everything interesting happens, \Windows\System32. (The drives had different drive letters than normal, but who cares.) Now, under regular circumstances, that little Accessibility button on the logon screen launches a program called Utilman.exe. From there you can get all the gizmos like the on-screen keyboard and magnifier and things like that to help you log in, in case your eyesight's not so good or you have no hands or something. But there was no Utilman.exe in the System32 folder. I guess that's why the Accessibility icon didn't do anything when I clicked it. My domain controllers are very inconsiderate of the handicapped.

So I just decided to try the command:

copy X:\Windows\System32\cmd.exe X:\Windows\System32\Utilman.exe

Then I rebooted. Once I was back at the Windows login screen, I hit the Accessibility icon one last time. Command Prompt. And not just any Command Prompt, but a Command Prompt running as Local System! Since this was a normal boot of the domain controller and therefore Active Directory was online, I just did:

net user DomainAdmin *

And I was greeted with a prompt to reset the domain administrator password to whatever I wanted.

This time I made damn sure that the new password was saved and backed up in my password vault.

Afterthoughts

So it's great that I was able to reset a domain administrator password with nothing but a bootable image and physical access to the machine, but, one can't help but wonder about the glaring security implications of this. Once a person has a Local System shell on a domain controller, they own the domain.

Furthermore, today's enterprise environments are more complex than ever. Having physical access to the console of the machine doesn't necessarily mean you have to actually be able to touch the machine, like I was able to do here. These days, "physical access" can mean logging in to an ILO or accessing a virtual machine management console from a thousand miles away. And are we, as enterprise administrators and IT directors and CTOs, sure that everyone in our organization with access to those things are people we also trust as potential domain admins? It's not just about the threat of outside attackers. We may be unwittingly exposing a massive attack surface to all of our front-line support personnel should they ever decide to go rogue.

Alright, so what should be done about this little "feature" of Windows? First off, Microsoft needs to not launch Utilman.exe as Local System. Launch it as an unprivileged account that doesn't have the ability to do things like reset Administrator passwords. I'm sure there's a technical reason why they think they "have" to run the Ease of Access stuff as Local System, so that it can simulate the secure attention sequence, etc., but I'm not buying it.  Just because it's hard doesn't mean they should get away with letting this gaping security hole go unpatched.  They could also include a hash check in the code behind that little icon so it would at least be a little more difficult to replace the real Utilman.exe with, say, a copy of Cmd.exe that was renamed to Utilman.exe.

To make matters worse, there is no "official" Group Policy setting to disable this button. You might be able to mitigate the risk somewhat by using Group Policy to do something like run a script that cacls Utilman.exe at startup and denies Everyone all access, or having Group Policy push out your own version of Utilman.exe at every boot of the computer... neither of which are ideal solutions nor are they likely to be bulletproof.

Oh well. I guess that goes along with that immutable law of computer security, that if someone else has physical access to your computer, then it's not your computer anymore.

Passive-Aggressive Configuration Management FTW

I was doing a little deep investigation of the Windows DNS Cache service today, and I discovered that the process checks for the existence of this registry value upon startup:

HKLM\System\CurrentControlSet\services\Dnscache\Parameters\DowncaseSpnCauseApiOwnerIsTooLazy

Needless to say, the configuration setting doesn't appear to be publicly documented.  Whatever it does though, I sense some latent hostility toward some API owner. It wouldn't be the first time Microsoft has let Registry settings with silly names slip through the cracks.

From the "No Not All My Stuff Is Patched Yet" Dept.

This is a public service announcement.

If you have a Windows 2008 R2 server without SP1 or without this hotfix, and you also have a monitoring application or script that uses WMI to query for service information from the Win32_Service WMI class on a regular basis, you might just get burned really badly by a memory leak that will have the WMI service gobbling up >500MB of memory until it crashes.  You might then have to roll back hours of work and hope the Microsoft bug didn't cause any cluster failovers or dropped Exchange mailbox stores across a couple dozen servers.