Best-Practices Remediation Tips for Server 2012 Pt I.

I'm calling this Part 1 because I realized as I started writing that this is a lot of work, and can easily be split into 2 or more articles.

Like most of the IT Pro community, I've been getting comfortable with Server 2012 the past several weeks now, and the journey is still ongoing. As I talked about last time, I do like those Best Practices Analyzers for Windows Server. Here's me running it in Server Manager:

Best Practices Analyzer Server Manager 2012*BPA in Server Manager 2012*

Getting any of these results back means that I have some work to do in remediating them. It's not uncommon for a Server 2012 system that was just built fresh with no applications loaded or configuration changes to still have one or two compliance issues in the Best Practices Analyzers. There is a balance to be maintained between compatibility and performance optimization. Also, many of these issues that popped up for me personally were not role-specific, but rather apply to a base component of the OS. Now I'll go over some of the interesting ones I've gotten and how I fixed them:

 

[Hyper-V]: Avoid storing Smart Paging files on a system disk.

Smart paging is new with Hyper-V 2012. Read about it here. Basically, you now enter a minimum amount of RAM a virtual machine can have, you enter a maximum amount of RAM a virtual machine can have, and you also now enter the amount of startup RAM a virtual machine can have. The Windows OS can boot up more comfortably with a larger amount of RAM, but once it reaches cruising altitude and is idle, the RAM requirements go back down, which will allow the Hyper-V host to gradually start reclaiming memory from the VM. With all this dynamic shrinking and growing of the memory on all your VMs, that's where the "smart paging file" comes in. And just like you can improve performance by putting your traditional Windows paging file on its own disk, the same goes with the Smart Paging file.

[Hyper-V]: Use RAM that provides error correction.

Microsoft doesn't support Hyper-V environments on hardware that isn't using ECC RAM. This is just a lab using desktop-grade hardware, so there's nothing I can really do about this. If you're using real server gear, this should not be an issue for you.

[Hyper-V]: Virtual machines should be backed up at least once every week.

This one I still don't understand. I have all the guest operating systems backing up nightly, and then I am backing them up again through the backup of the Hyper-V host. So go away, error.

[Windows]: Short file name creation should be disabled.

Then why is it enabled by default? Oh, I know why... it's because of your crappy line-of-business apps that were written back in 1998 that you can't get rid of, right? Well I'm in a pure 2012 environment right now, so I have no such worries. Good bye 8.3 filenames. Change this registry value to 1 to disable short file name creation on all volumes: HKLM\System\CurrentControlSet\Control\FileSystem\NtfsDisable8dot3NameCreation.

[Windows]: IrpStackSize should have the recommended value.

Irp stands for I/O Request Packet. Mark Russinovich did a great job of explaining IRPs and their role in the Windows I/O system in his book Windows Internals. All the various components or layers that an I/O packet traverses on its way to and from a disk, for example, are collectively referred to as "a stack." Each filter driver you add to the file system means that the IrpStackSize needs to be increased in order to accommodate it. A common example of this is when you install an antivirus product that uses a file system filter driver. If your IrpStackSize is set too small, certain operations might fail, such as attempting to access that machine's file system remotely. Conversely, it doesn't need to be set too high, either.  It was at 11 by default on my 2012 systems. The Best Practices Analyzer says it should be at 15, so I'll set it to 15. HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\IRPStackSize.

 

Alright, that's where I'll leave off for part 1. Part 2 will focus on some more network-specific optimizations, so check back soon!

Best Practices Analyzer in Server Core 2012

I always liked the Best Practices Analyzers in Windows Server, where you could scan most of the installed roles for "best practices" and it would inform you if you had any errors or warnings in your setup.  It's just a nice little sanity check to make sure you didn't forget any major steps when installing AD Directory Services, or a DNS server, etc.  Traditionally, you'd initiate the Best Practices Analyzer from within Server Manager.

So what about Server Core, when you have no GUI? Well I'm glad you asked.

Get-BPAModel

That command will list all of the available Best Practices Analyzer modules that are installed on the current system. You use the module ID, such as "Microsoft/Windows/DirectoryServices" to run the BPA, like so:

Invoke-BPAModel Microsoft/Windows/DNSServer

The above command will run the Best Practices Analyzer on the specified role. The results from the BPA are written to disk, so to retrieve the results from your last BPA scan at any time, you do:

Get-BPAResult Microsoft/Windows/Hyper-V

If you wanted to only see the results where a potential problem was found, try this:

Get-BPAResult Microsoft/Windows/DirectoryServices | ? {$_.Problem -ne $null}

 

Native NIC Teaming In Server 2012

Built-in NIC teaming was one of my personally most anticipated features of Server 2012.  NIC teaming, whether for redundancy or for more bandwidth, has always been a cool concept and one of the foundations of highly-available systems, but it has historically required 3rd party vendor software to enable.  Probably the most popular example I can think of is the HP Network Configuration Utility software:

HP Network Configuration Utility

Almost every IT pro is going to be familiar with that screen.  Up until now, to team network adapters, one had to use vendor software such as the HP software pictured above. But starting with Windows Server 2012, the ability is built right in to the operating system, bringing the feature to new sets of hardware and without the need for any 3rd party vendor drivers or software! (Also of note is that Microsoft supports their NIC teaming, whereas they do not support the HP Network Configuration Utility.)

You can use the graphical Server Manager to configure NIC teams, but you can also do it all right from within Powershell. And since I typically prefer to keep my servers in straight-up Server Core mode, I wanted to figure out how to do it all from Powershell. My test machine for this experiment is a SuperMicro SYS-5015A-H 1U. It has two embedded GbE adapters (Realtek based.) Before Server 2012, I always just kept one of the NICs disabled since I had no use for it, and no teaming software. But now, I've installed a fresh copy of Windows Server 2012 Standard edition on it. 

Get-NetAdapter*Get-NetAdapter*

To make a team out of these two network adapters, simply do

New-NetLbfoTeam -Name Team -TeamMembers Ethernet,"Ethernet 2"

That's it! (Just put quotes around 'Ethernet 2' because it contains a space.) Now keep in mind that you'll probably have to re-do the IP configuration for your new NIC team now, so you'll want physical or DRAC/ILO access to the machine so you can do that. (Or do it via script. I set the IP configuration on my new NIC team via sconfig.) Here is what the new team looks like in Powershell: 

Get-NetLbfoTeam*Get-NetLbfoTeam*

The TeamingMode and LoadBalancingAlgorithm default to SwitchIndependent and TransportPorts, respectively, but of course can be configured to whatever you want as you create the team with the New-NetLbfoTeam command. Check this Technet article for explanations on the different options and what they do. If you later want to add another NIC to the existing team, you can use the Add-NetLbfoTeamMember command and specify the NIC you want to add.

 

Get-NetLbfoTeamMember*Get-NetLbfoTeamMember*

Beautiful.

Log Parser 2.2 and Log Parser Studio

At first I thought to title this post the same as the catchphrase of Log Parser: "The Whole World Is Your Database!"

But then I decided that was a bit too exciting for what I actually wanted to talk about.

So I just discovered Log Parser Studio a few days ago. LPS is a graphical frontend to Log Parser; quite similar to how SQL Management Studio is a GUI frontend to interacting with SQL Server.  I am, quite frankly, ashamed that I didn't already know about it. It's fantastic.

The thing is... Log Parser is a command-line utility that uses a very SQL-esque language to interact with logs. What kind of logs, you ask?  Any kind of logs! That's right... you can use it to query the Windows Security Event Log, or you can use it to query a folder full of IIS web server logs, or you can use it to query a log full of your own personal electric utility bills from last year!

However, Log Parser itself is a very complex, albeit powerful and flexible, command-line utility. Maybe you want something a little more user-friendly to get you started. That's exactly where Log Parser Studio, the GUI frontend, comes in to play.

As a little demonstration, I installed Log Parser 2.2 on my workstation. Then I downloaded Log Parser Studio to my workstation. I fired it up as a Windows application, and I pointed it to the remote IIS logs directory of this very web server. I then right-clicked on "IIS: Request per Hour" and chose "Run report now." As if I had just run a SQL query in SQL Management Studio, this window popped up:

 

Log Parser Studio Query*Click for Bigger*

This data is probably every single HTTP GET request made per hour, rather than a count of hits made by unique IP addresses, but the point is you now have this amazing utility that will parse practically any amount of data you can think of from any source of data you can think of. Go check it out and see how Log Parser is even capable of generating pie charts and bar charts and all sorts of crazy things using this data!

Mystery of the Performance Counters with Negative Denominators!

So I finally solved a bit of a mystery, after wondering about it off an on for about 6 months.

A lot of us out there use some sort of monitoring program like SCOM that gathers some performance counter data from all of the servers in your environment, and uses that data to produce messages such as:

WARNING: Hey man, the CPU usage on SERVER01 is really high and has been for a while, so, you might wanna' go check that out.

But every once in a while, if for some reason the performance counters needed are unavailable, your monitoring application might say something like this:

ERROR: Unable to get performance counter data for SERVER01.

Well that happens to me somewhat frequently. Sometimes it's an error in the monitoring application, or sometimes the agent machine needs a lodctr /R or winmgmt /resync or something like that. But every once in a while it's more difficult to solve than that. I logged on to SERVER01 to see if I could simply add the performance counters needed to Perfmon. The objects and instances were there, but one of them looked odd. Then I decided to use typeperf.exe to get some data from the counters:

 

perfmon error

 

typeperf error

A counter with a negative denominator value was detected? Now contrary to that screenshot above, the -1's were intermittent, and would actually change to normal values for a few seconds, and then go back to negative ones. It was as if the scale of that counter was offset, and whenever real CPU load would occur, the counter would increase to positive values, but then as the CPU usage dropped back down closer to zero, the counter would fall below zero again.  Hmm... if you do a search for something like "missing performance counters," or "a counter with a negative denominator value," you'll get plenty of good links, as missing or corrupt performance counters are a somewhat common issue. Such as this, or this. Notice in that last link, that there is a vague reference to it being caused by "intermittent timing issues in the performance data handlers for many counters." The article fails however to offer any advice or anything more concrete than that. lodctr /R and winmgmt /resyncperf and all that usual stuff did nothing to help.

So I sat back and thought some more about the scope of the problem. I have only seen this issue two or three times in my career. The two cases I could remember both seemed to be on VMware virtual machines. One of them was a Windows 2003 operating system, and then a couple months later I saw it again on a Windows 2008 R2 VM. That phrase from that one KB stuck with me... "intermittent timing issues," and that made me think of hardware timing issues. But since these were VMs, they didn't exactly have hardware per se... but maybe it had something to do with the way in which the virtual machines communicate with their physical host. I was so close to figuring out the answer, but at the time I shrugged it off and shelved the issue in my mind for a while.

Then about a week ago the issue popped up again. Long story short, it turns out it was this all along:

VMware tools

Unchecking that "Time synchronization between the virtual machine and the ESX server" in the VMware tools app on the VM made the problem go away. Ugh. The answer had been so simple all along, yet it alluded me for months.

I feel better now that I at least have a solution for that particular scenario.  Now... back to Hyper-V.