Update May 18, 2014: This case was featured in Mark Russinovich's "Case of the Unexplained" session at TechEd 2014 in Houston, Texas!
So a couple days ago, I watched what has to be my favorite TechEd 2013 video. I very highly encourage you take an hour and watch it too - but only after you've read this post! This video finally inspired me to take care of a problem that had been bugging me (no pun intended) on my own Windows machine for several weeks.
On my Windows 7 work laptop, I use Outlook for my work email. A few weeks ago, I began to notice an odd behavior. It would happen about twice a day. I would be doing various different things on my computer at the time, but Outlook was always open, at least in the background. Suddenly, the system would get sluggish and unresponsive for about 1 minute, and during that one minute, the fan inside the laptop would audibly spin up and my mouse cursor would rapidly switch between a normal cursor and a "wait" cursor with the little spinning ring next to the pointer... exactly like this:
(*This is the first and hopefully last animated gif I've ever used on this blog, sorry! But it really did look just like that.*)
First thing I did was look at the event logs:
An application fault followed by a couple of Windows Error Reporting messages. This event would get logged at the exact same time as the foul behavior. I already had a hunch that 'SearchProtocolHost.exe' had something to do with the Windows Search service. So I stopped and disabled the Windows Search service. This completely eliminated the errors and the strange application faults... but... it also had the effect of disabling my ability to search my emails in Outlook. I get tons of email and I rely on the ability to search my email folders for keywords, so this solution was inadequate. I had to dig deeper.
Since the strange behavior and the application faults happened pretty regularly and lasted for a while, that gave me ample time to observe the process behavior in Process Explorer:
The above screenshot doesn't show what I really wanted to show, and that was while this malfunction was taking place, I could see dozens of SearchProtocolHost.exe processes dying and spawning in rapid succession, and also dozens of WerFault.exe (the Windows Error Reporting tool) processes dying and spawning for every SearchProtocolHost.exe that would spawn and then immediately die.
I could see this happening because of the red highlighting and green highlighting in Process Explorer that indicates processes that have either just been created or just exited.
All this crazy process spawning and immediately dying activity in the background is what was causing my epileptic mouse cursor, as well as the general high CPU and disk usage.
At this point I decided I would set up procdump to capture process dumps of this SearchProtocolHost.exe thing whenever it crashed, by typing procdump -ma -i C:\dumps I already had the Windows Debugging Tools installed, and I already had my symbols properly set up. Andrew Richards teaches you how to set all that up in the video I told you about at the beginning of this post, as do various other sources you can find readily on the web.
A few minutes later, sure enough I started getting procdump process windows popping up and writing out crash dumps as quickly as SearchProtocolHost.exe was crashing. It captured about 12 dumps, one for each time the process crashed in rapid succession, and each dump was about 90MB.
I opened one at random. Luckily, this dump was pretty easy. A perfect learning curve for a debugging novice such as myself:
Well the helpful text tells you right off the bat that the .ecxr command will tell you something interesting. (Exception Context Record.) The only interesting thing I see here is that Windbg does not find symbols for EVMSP32.dll. If your symbol server is already set up, then that is usually a pretty big tipoff that you're working with a non-Microsoft DLL.
Let's try !analyze -v:
This is a little more interesting info. What we have here is the thread stack that caused everything to go pear-shaped. Notice the last thing on the stack before death (stacks grow upward) is some function in EVMSP32.
So what the heck is EVMSP32 already? Windbg must be reading my mind, because it provides a handy hyperlink to the details of that exact module. Let's click it!
Symantec! How could you do this to me!? It was you causing my system to go nuts while you tried to index my email!
Hey, I never asked for Symantec Enterprise Vault. It was a "gift" that corporate IT pushed onto my laptop, oh... about exactly the same time when I started having this problem.
I uninstalled the Symantec Enterprise Vault Outlook Add-in via Programs and Features in the Control Panel. Problem solved. No more annoying system behavior or background application faults, and I can still search my email in Outlook.