Psst, You Want A Script To Backup Your Lab VMs?

I can hook you up...

So I'm always doing a lot of lab work with Hyper-V virtual machines. Every once in a while I want to just save the state of the entire lab all at once and back it up to a safe storage volume.  I suppose I could set up Windows Server Backup on each of the VMs, and find some disk to use as a pass-through disk for one of the virtual machines and then share that so that the VMs could back up to the network share... but that's a ton of hassle.

How about I just save the state of all the VMs, and export them directly to my backup volume, then resume the VMs, all from the hypervisor?  As a scheduled task, perhaps?

About 10 minutes in the Powershell ISE and I've done just that.  A couple things to be warned of - First, you can't do this in production. The virtual machines are frozen while they're being exported, and it can take several minutes to export a VM. Secondly, make sure you are running with full Administrator privileges, or else cmdlets such as Get-VM will silently return nothing.

 

# Ryan Ries, 2013
# Backs up some lab VMs. Takes several minutes at least.

[String]$BackupPath = "D:\Backups\Hyper-V"
[String]$ErrorLog   = "D:\Backups\Hyper-VBackups.log"
$Start = Get-Date
"$(Get-Date) - Hyper-V Backups Starting." | Out-File $ErrorLog -Append
Try
{
    Get-Childitem $BackupPath -Recurse -Force | Remove-Item -Recurse -Force -ErrorAction Stop
}
Catch
{
    "$(Get-Date) - Error during Get-Childitem or Remove-Item: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Stop-VM -ErrorAction Stop -Save
}
Catch
{
    "$(Get-Date) - Error during Get-VM Stop-VM -Save: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Export-VM -ErrorAction Stop -Path $BackupPath
}
Catch
{
    "$(Get-Date) - Error during Export-VM: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Start-VM -ErrorAction Stop
}
Catch
{    
    "$(Get-Date) - Error during Start-VM: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
$End = Get-Date
"$(Get-Date) - Hyper-V Backups completed in $([Math]::Round((New-TimeSpan $Start $End).TotalMinutes)) minutes." | Out-File $ErrorLog -Append

In Which I Go Debuggin'!

Update May 18, 2014: This case was featured in Mark Russinovich's "Case of the Unexplained" session at TechEd 2014 in Houston, Texas!


So a couple days ago, I watched what has to be my favorite TechEd 2013 video. I very highly encourage you take an hour and watch it too - but only after you've read this post!  This video finally inspired me to take care of a problem that had been bugging me (no pun intended) on my own Windows machine for several weeks.

On my Windows 7 work laptop, I use Outlook for my work email.  A few weeks ago, I began to notice an odd behavior.  It would happen about twice a day.  I would be doing various different things on my computer at the time, but Outlook was always open, at least in the background.  Suddenly, the system would get sluggish and unresponsive for about 1 minute, and during that one minute, the fan inside the laptop would audibly spin up and my mouse cursor would rapidly switch between a normal cursor and a "wait" cursor with the little spinning ring next to the pointer... exactly like this:

Mouse cursor animation
(*This is the first and hopefully last animated gif I've ever used on this blog, sorry! But it really did look just like that.*)

First thing I did was look at the event logs:

Error in the Event Log

An application fault followed by a couple of Windows Error Reporting messages.  This event would get logged at the exact same time as the foul behavior.  I already had a hunch that 'SearchProtocolHost.exe' had something to do with the Windows Search service.  So I stopped and disabled the Windows Search service.  This completely eliminated the errors and the strange application faults... but... it also had the effect of disabling my ability to search my emails in Outlook.  I get tons of email and I rely on the ability to search my email folders for keywords, so this solution was inadequate.  I had to dig deeper.

Since the strange behavior and the application faults happened pretty regularly and lasted for a while, that gave me ample time to observe the process behavior in Process Explorer:

Process Explorer

The above screenshot doesn't show what I really wanted to show, and that was while this malfunction was taking place, I could see dozens of SearchProtocolHost.exe processes dying and spawning in rapid succession, and also dozens of WerFault.exe (the Windows Error Reporting tool) processes dying and spawning for every SearchProtocolHost.exe that would spawn and then immediately die.

I could see this happening because of the red highlighting and green highlighting in Process Explorer that indicates processes that have either just been created or just exited.

All this crazy process spawning and immediately dying activity in the background is what was causing my epileptic mouse cursor, as well as the general high CPU and disk usage.

At this point I decided I would set up procdump to capture process dumps of this SearchProtocolHost.exe thing whenever it crashed, by typing procdump -ma -i C:\dumps  I already had the Windows Debugging Tools installed, and I already had my symbols properly set up.  Andrew Richards teaches you how to set all that up in the video I told you about at the beginning of this post, as do various other sources you can find readily on the web.

A few minutes later, sure enough I started getting procdump process windows popping up and writing out crash dumps as quickly as SearchProtocolHost.exe was crashing.  It captured about 12 dumps, one for each time the process crashed in rapid succession, and each dump was about 90MB.

I opened one at random. Luckily, this dump was pretty easy. A perfect learning curve for a debugging novice such as myself:

Windbg

Well the helpful text tells you right off the bat that the .ecxr command will tell you something interesting. (Exception Context Record.)  The only interesting thing I see here is that Windbg does not find symbols for EVMSP32.dll. If your symbol server is already set up, then that is usually a pretty big tipoff that you're working with a non-Microsoft DLL.

Let's try !analyze -v: 

More debugging

This is a little more interesting info. What we have here is the thread stack that caused everything to go pear-shaped. Notice the last thing on the stack before death (stacks grow upward) is some function in EVMSP32.

So what the heck is EVMSP32 already?  Windbg must be reading my mind, because it provides a handy hyperlink to the details of that exact module. Let's click it!

 

 

EVMSP32.dll

Symantec!  How could you do this to me!?  It was you causing my system to go nuts while you tried to index my email!

Hey, I never asked for Symantec Enterprise Vault. It was a "gift" that corporate IT pushed onto my laptop, oh... about exactly the same time when I started having this problem.

I uninstalled the Symantec Enterprise Vault Outlook Add-in via Programs and Features in the Control Panel.  Problem solved.  No more annoying system behavior or background application faults, and I can still search my email in Outlook.

A Few Powershell Commands That Have Been Useful To Me Lately

I've been building lots of new Server 2012 machines lately, which means lots of Server Core, which means lots of command line interface, which means lots of Powershell.

So, a few quick tricks I've found useful the past couple days.

foreach($_ In Get-ADComputer -Filter *) { Invoke-Command -ComputerName $_.Name { Set-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\Filesystem -Name NtfsDisable8dot3NameCreation -Value 1 } }

This nifty one-liner grabs all the computer names from Active Directory, remotely disables 8.3 file name creation on each machine.  It's good for filesystem performance, as Windows no longer needs to maintain records of old DOS-style names like FILENA~1.TXT for every file with a long name. Better yet, the Best Practices Analyzer will stop complaining about it once you disable 8.3 file name creation. Unfortunately, MAX_PATH in Windows is still 260 characters. When you hit that limit, you will be extremely annoyed. .NET, and thus Powershell, are especially flummoxed by really long file paths. The Windows API does technically allow you to exceed MAX_PATH by using the \\?\ handle, but you also lose a lot of sanitizing and security features when you perform that bypass.  Note, you need to reboot the machine after changing the 8.3 file name policy.

New-NetIPAddress -AddressFamily IPv6 -InterfaceIndex 13 -IPAddress "2001:2c98:ee9c:279b::3" -PrefixLength 64

 Get used to setting your IP configs with Powershell. Not just IPv4, but IPv6 too. Hmm, speaking of TCP/IP configurations, what else do I need besides an IP address? Oh, yeah:

Set-DnsClientServerAddress -Addresses fd58:2c98:ee9c:279b::1,fd58:2c98:ee9c:279b::2 -InterfaceIndex 13

DNS servers! And of course, if you need to know the index of the network adapter you're working on, it's as simple as Get-NetAdapter.

 

Which Hyper-V VM Worker Process Belongs To Which VM, PART 2! (And also about IDE and SCSI controllers)

It is with great shame that I admit that the information in yesterday's post is not accurate.  Yesterday, I spoke about the "Hyper-V Virtual IDE Controller (Emulated)" performance counters as though they would give you I/O statistics for virtual machines using the virtual IDE controller.

Virtual IDE Controller

Au contraire!

There are only two scenarios in which those counters will show anything useful. Either when the VM very first boots up, for about 1 second before the OS is completely loaded... or, if the Hyper-V VM integration tools are not installed.

See, when the virtual machine has its integration tools installed, then even when the configuration screen about says "IDE Controller," it's no longer an emulated IDE controller. It's a synthetic virtual controller, just like you would get if you used the SCSI controller on the virtual machine.

If the VM did not have the integration tools installed, and was thus using the actual emulated IDE controller, that would mean that the I/O would need extra steps in processing, and it would travel through its vmwp.exe process on the Hyper-V host.  However, if the enlightenments are installed on the VM, the I/O travels through the VMBUS.  This makes the I/O faster, and it explains why when using synthetic devices as opposed to emulated ones means that you can no longer see I/O happening in vmwp.exe, (it should travel straight to the "System" process on the root partition,) nor will you see anything but zeros in the "Hyper-V Virtual IDE Controller (Emulated)" performance counters.

To reassure you that I'm not just making s*$@ up this time, this MSDN blog post says basically the same thing. And so does this even better one.

"The next thing to notice is the “Fast Path Filter”.  This is a filter driver that gets installed on all disk objects in the virtual machine – whether they are IDE or SCSI.  It allows us to pass directly to the VMBUS based path for everything except low level disk operations (like partitioning a disk)."

So there you have it. But unfortunately, this means those counters I mentioned are pretty much worthless now. But all is not lost!  You can still use the "Hyper-V Virtual Storage Device" counters, however, the counters seem to be about how much I/O is being done on the VHD/X of the virtual machine, and not the precise I/O being done by the virtual machine itself.

Not as good, in my opinion, but it's still something.

Also, I'd like to thank Chris S from Serverfault for enlightening me (pun fully intended) and putting me on the right path.

Which Hyper-V VM Worker Process Belongs To Which VM?

Warning! This info is not totally accurate! Please read the next day's post for corrections.

I wanted to track down which virtual machine on my Hyper-V host was causing an inordinate amount of disk I/O on my host without logging in to each one.

In Hyper-V, you will notice that on the root partition, you will see one instance of vmms.exe (VM Management Service,) and then you will see a separate instance of vmwp.exe (VM Worker Process) for each virtual machine that is currently running.

Notice that the vmwp.exe processes run under a special user context which contains the GUID that you would find in that virtual machine's configuration files on disk. The same GUID is also supplied to vmwp.exe as an argument as the process is created, like so:

"C:\Windows\System32\vmwp.exe" c83cdee4-1a6d-4f51-9d05-e57df8403ed4

It's not immediately apparent which vmwp.exe belongs to which VM. Furthermore, the I/O charged against each individual vmwp.exe process is not necessarily indicative of what's actually happening on the virtual machine that its hosting. So we'll need to go to the performance counters instead. The "Hyper-V Virtual IDE Controller (Emulated)" set of counters should do the trick, assuming you're using the IDE controller on your VMs.

I have all the information I need now to determine which virtual machine is responsible for the large amount of I/O... but I didn't want to just do it manually. Why not write a reusable tool that can also be run on Core servers with no GUI?

So this simple script, when run on a Hyper-V host, does just that. The output looks like this:

Get-VMPidAndIO | Out-GridView

My VMs were all idle then, hence all the zeros. The screenshot loses all its dramatic flair, but whatever.

The script could easily be enhanced by supporting remote Hyper-V hosts, alternate credentials, etc. But what do you want for 30 minutes?

#Requires -Version 3
function Get-VMPidAndIO
{
<#
.SYNOPSIS
	Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
.DESCRIPTION
	Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
    Currently only works for VMs using virtual IDE controllers.
    Requires Powershell 3 at a minimum.
.LINK
    http://myotherpcisacloud.com
.NOTES
    Written by Ryan Ries, June 2013.
    ryan@myotherpcisacloud.com
#>
    BEGIN
    {
        Try
        {
            $VMProcesses = Get-CimInstance -Query "Select ProcessId,CommandLine From Win32_Process Where Name ='vmwp.exe'" -ErrorAction Stop
        }
        Catch
        {
            Write-Error $_.Exception.Message
            Return
        }
    }
    PROCESS
    {

    }
    END
    {
        Foreach($_ In $VMProcesses) 
        {
            $VMName = $((Get-VM | Where Id -EQ $_.CommandLine.Split(' ')[-1]).Name)            
            [PSCustomObject]@{PID=$_.ProcessId;
                              VMName=$VMName; 
                              ReadBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Read Bytes/sec").CounterSamples.CookedValue, 2);
                              WriteBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Write Bytes/sec").CounterSamples.CookedValue, 2); }
        }

    }
}