"SSLv3.0/TLSv1.0 Protocol Weak CBC Mode Vulnerability"

Good afternoon, and sorry I haven't posted in a while.  I've been staying pretty busy.

So if you have been in IT or working with servers for very long, you're probably familiar with this guy:

*The most annoying appliance ever?*

So in case you're not familiar, this little guy sits in your datacenter, scanning your network, and spits out reports about all the potential vulnerabilities it finds on all your network devices and servers. Then you get to go fix all of those potential vulnerabilities so that you can maintain PCI compliance and such. Sometimes it's as easy as applying an OS patch. Sometimes it's making an obscure configuration change to an application that is just as likely to break the application as it is to plug the vulnerability.

"SSLv3.0/TLSv1.0 Protocol Weak CBC Mode Vulnerability" was a particularly annoying one. I'm making sure to put the exact title of the vulnerability as Qualys puts it so that maybe someday it will show up in somebody's Internet search and help them.  I wasn't so lucky.  There really wasn't much information out there on this particular vulnerability that applied to me; or so I thought at the time.  It seemed like the only information I could find on this vulnerability either pertained to Linux servers, or particularly to IIS on Windows Servers.  My server was a physical HP machine running Windows Server 2008 R2 with all the HP software installed... and I wasn't the only guy on the team who was hung up on this particular vulnerability.

Qualys will tell you that this vulnerability is tied to CVE-2011-3389 and so my first instinct was to look for the Microsoft-issued security advisory.  The particular Windows patch it was suggesting was already installed, and I didn't even have IIS installed on this server anyway.  This led me down the path of modifying system-wide registry settings like this, to no avail.  The same vulnerability kept showing up on subsequent scans.

So after taking a step back and thinking for a second, it occurred to me that Qualys was reporting this particular vulnerability on port 2381.  That's the port used by the HP System Management Homepage.  (A glorious piece of software... please note the sarcasm.) So maybe there's a configuration change I can make just to the SMH... and after Googling through some HP documentation I found this gem:

C:\hp\hpsmh\bin>smhconfig.exe -Z 'RC4-SHA'

That should restrict the cipher modes that the SMH is allowed to use to only RC4-SHA. (With a capital Z.)  But my version of smhconfig.exe didn't implement the -Z switch, so I updated it via the Proliant Support Pack, and then was able to successfully run the command.

Problem solved. Vulnerability gone.

It was only after I went through all that, I went back to the original CVE-2011-3389 page and noticed this.  :P

SQL Server - Unable to Generate SSPI Context

The different sorts of authentication mechanisms in play in a Windows network can be pretty complex.  So when someone asked me, "Why do I get a 'Could not generate SSPI context' error when I try to log in to a SQL server?" I knew that there could be several answers to that question.  Go ahead and Google it yourself -- you won't get a definite *This is absolutely your problem* sort of answer.

First I remembered that there was a situation where RSA SecureID tokens (essentially certificates for our purposes) were used for various authentication tasks in the domain, and if one tried to authenticate to a SQL Server with Windows authentication without having one's RSA token plugged in, the "Could not generate SSPI context" error would be generated. Plug the SecureID device into a USB slot, and you'd log in to the SQL server just fine. But I knew that policy was not an issue in this situation...

Then I thought about how services not having their SPNs registered with Active Directory can cause authentication problems.  Specifically, if a SQL Server doesn't have its SPN registered properly in Active Directory, Kerberos authentication cannot be used.  But that still shouldn't prevent you from authenticating whatsoever... it'll just drop you down to NTLM instead of Kerberos.

Also, I was able to perform a logon with Windows auth to the same SQL Server at the same privilege and security level as the user, so I knew it had to be something at their end.

The only other thing I could think of was that something was just wrong with their security token that was confusing their SSPI?  Maybe it was corrupt somehow?  I'm not sure.  So, I recommended that the user run "klist purge" to purge all their domain controller-issued tickets, knowing that they would be refreshed as soon as they requested access to a domain resource...

Bingo.  Problem solved.

Server 2012 Goodies

Windows Server 2012 is getting close now.  Of course I've been playing with the release candidate.  One of my favorite things so far is how easy it is to turn off the GUI, therefore turning the server into a Core installation. You can also turn off the GUI, but leave the Server Manager enabled.  Another thing that I think is going to be awesome is the Hyper-V Extensible Virtual Switch. It's obvious that Microsoft listened to its customers when they complained that the Hyper-V Virtual Switch in 2008 R2 was a bit anemic/featureless.  This is a huge enhancement, in my opinion, and is going to introduce some very interesting opportunities, such as migrating a VM to another datacenter without changing its IP address, etc.

There are a lot of other new enhancements.  I found this free eBook entitled Introducing Windows Server 2012. Check it out; there are tons of neat and enticing bits of information in there.

Windows 2008 R2 + SQL 2008 R2 + Password Policy = Security Event Log Out of Control

I was asked to troubleshoot an interesting problem today where the Windows Security event log was being "flooded" by one particular sort of event.  By flooded, I mean about 20 duplicate entries logged per second.  The biggest problem with this is that it was making the Security log on that server useless, as the log would fill up within 45 minutes at that rate.  Click to enlarge the screenshot below:

event log ss*Names were changed to protect the guilty*

 The operating system is Windows 2008 R2. The server is a domain member. The server also runs SQL Server 2008 R2.  The server is a cluster node in a failover cluster.  I started Googling and Binging the event ID and description, and I didn't get much at first.

As an aside, I can't believe I just used the word "Binging," and I much prefer Google for almost everything, but if you want to search Technet, MSDN, and other Microsoft sites, the built-in Bing search on those sites actually does tend to produce better results on those sites than a general Google search. For me anyway. Maybe a "site:technet.com xyz" search on Google would do just as well. Anyway... onward:

So the only solid clue I found in my searching was this Technet article. The Windows Password Policy Checking API was being called at a staggering rate, but why?  By whom?  How do I make it stop?  The same SQL service account was being named in the events, so it obviously must have something to do with SQL.  Well, I could turn off the auditing of "Other Account Management Events," either by way of domain GPO or local security policy on that server... but that would only suppress the logging of that behavior.  It does nothing to stop the actual behavior.  Plus I would also lose any other events of that same category on the system.

I also knew that there was an "Enforce Password Policy" option that can be configured on each SQL account.  So I fired up SQL Management Studio on that server and did some testing, and as it turns out, that option was enabled on several accounts, including service accounts that are designed to hit the database rapidly.  It appears that every time an authentication attempt is made by one of those SQL accounts that has that Enforce Policy option checked, the SQL service makes a call to that Windows API to do some password policy checking, and that event is logged.

I tactically identified a few key service accounts that I knew to be very active on that database, and I disabled the "Enforce Password Policy" option on those accounts one by one.  I confirmed that with each account I changed, the rate at which those Security event 4793's were coming in decreased. Until finally, they stopped completely.

 

That's all for today.  On one final note, I wish we did cool things like this in the United States, especially as a resident of a state that cuts science funding.

From the "No Not All My Stuff Is Patched Yet" Dept.

This is a public service announcement.

If you have a Windows 2008 R2 server without SP1 or without this hotfix, and you also have a monitoring application or script that uses WMI to query for service information from the Win32_Service WMI class on a regular basis, you might just get burned really badly by a memory leak that will have the WMI service gobbling up >500MB of memory until it crashes.  You might then have to roll back hours of work and hope the Microsoft bug didn't cause any cluster failovers or dropped Exchange mailbox stores across a couple dozen servers.