Metal Whiskers? In *My* Datacenter?

Metal Whiskers*This image copyright NASA*

Metal whiskers, also referred to as tin whiskers or zinc whiskers, are something that I've read about with some curiosity before. Science does not currently fully understand why many metals and alloys form these tiny whiskers over time. The phenomenon has been known since the early 20th century, but we still don't know much about why it happens or how to effectively stop it.  It is still being studied today, and you can find metal whiskers in the news being blamed for things like fires aboard aircraft.  Obviously, these whiskers can wreak havoc in an electronic system whose components are packed tightly together. These whiskers can grow out of the solder used to manufacture electronic equipment, and they can also grow out of other non-electrified pieces of metal like server rack rails and the metal parts of datacenter raised flooring tiles. These tiny little metal whiskers can then be shaken loose or scraped off by such actions as lifting the floor tile and sliding it across the surface of an adjacent tile, then blown into the air by the datacenter ventilation system, and then subsequently sucked into the power supplies of the computers housed within the datacenter.  Resulting short-circuits can cause electronic component failure, and even fire.

This question was asked today by someone on ServerFault, which rekindled my interest in the subject.  I also recommend reading the Wikipedia on it.  And I highly recommend visiting this NASA page - an entire page devoted to the phenomenon of metal whiskers.

From that page, if you just watch or read one thing from it, I specifically recommend this video, which is specifically about the damage metal whiskers can do in a datacenter environment.

Users.exe v1.0.0.3

In my last post, I showed how to get RDP/TS session information from a local or remote computer, including ClientName, on the command line using Powershell. (But it's not really Powershell. It's a PS wrapper around some C# that P/Invokes a native API. Which is pretty nuts but it works.)  Since I can't think of any other way to get this information on the command line, I thought it would be useful to convert the thing to native code. I named the program users.exe. It's written in C. Here is what it looks like:

C:\Users\joe\Desktop>users /?

USAGE: users.exe [hostname]

Not specifying a hostname implies localhost.
This command will return information about users currently logged onto
a local or remote computer, including the client's hostname and IP.
Users.exe was written by Ryan Ries.

C:\Users\joe\>users SERVER01

Session ID  : 0
Domain\User : System
Client Name : Local
Net Address : n/a
Conn. State : Disconnected

Session ID  : 1
Domain\User : System
Client Name : Local
Net Address : n/a
Conn. State : Connected

Session ID  : 25
Domain\User : DOMAIN\joe
Client Name : JOESLAPTOP
Net Address : 10.122.124.21 (AF_INET)
Conn. State : Active

I had a pretty rough time shaking the rust off of my native/unmanaged code skills, but I pulled it off.  That said, if you wanna give it a try, I would very much appreciate any bug reports/feature requests.

users.exe (63.50 kb)

Getting RDP Sessions with Client Computer Name

I came across this question on ServerFault today, which I thought was quite interesting.  And so I spent all morning formulating a solution to it.  How do you go about getting the list of currently connected users of a Windows machine including the remote client's machine name, on the command line? You can type query session, but that doesn't tell you the client's computer name or IP address.  The various Terminal Services and Remote Desktop-related Windows event logs are of very limited help. The Users tab in Task Manager tells you the usernames and their session IDs, but not their computer name.  You've got a bunch of WMI classes like Win32_LoggedOnUser and Win32_LogonSession, but none of them seem to contain any data about the connected client's machine name or IP address.

I wanted to come up with a Powershell solution.  Something that could also be run over the network and not just on the local machine.  So after a couple hours of research, finding the WTSQuerySessionInformation documentation, and spending some quality time on pinvoke.net, here is the "Powershell" solution that I came up with:

(I put "Powershell" in quotes because there's one single actual Powershell command in the whole thing.  Add-Type.  It's really all .NET code, which P/Invokes an unmanaged DLL.  Nevertheless, it's pretty sweet that you can do all that from within Powershell.)

 

# QuerySessionInformation.ps1
# Written by Ryan Ries, Jan. 2013, with help from MSDN and Stackoverflow.

$Code = @'
using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
public class RDPInfo
{
    [DllImport("wtsapi32.dll")]
    static extern IntPtr WTSOpenServer([MarshalAs(UnmanagedType.LPStr)] String pServerName);

    [DllImport("wtsapi32.dll")]
    static extern void WTSCloseServer(IntPtr hServer);

    [DllImport("wtsapi32.dll")]
    static extern Int32 WTSEnumerateSessions(
        IntPtr hServer,
        [MarshalAs(UnmanagedType.U4)] Int32 Reserved,
        [MarshalAs(UnmanagedType.U4)] Int32 Version,
        ref IntPtr ppSessionInfo,
        [MarshalAs(UnmanagedType.U4)] ref Int32 pCount);

    [DllImport("wtsapi32.dll")]
    static extern void WTSFreeMemory(IntPtr pMemory);

    [DllImport("Wtsapi32.dll")]
    static extern bool WTSQuerySessionInformation(System.IntPtr hServer, int sessionId, WTS_INFO_CLASS wtsInfoClass, out System.IntPtr ppBuffer, out uint pBytesReturned);

    [StructLayout(LayoutKind.Sequential)]
    private struct WTS_SESSION_INFO
    {
        public Int32 SessionID;
        [MarshalAs(UnmanagedType.LPStr)]
        public String pWinStationName;
        public WTS_CONNECTSTATE_CLASS State;
    }

    public enum WTS_INFO_CLASS
    {
        WTSInitialProgram,
        WTSApplicationName,
        WTSWorkingDirectory,
        WTSOEMId,
        WTSSessionId,
        WTSUserName,
        WTSWinStationName,
        WTSDomainName,
        WTSConnectState,
        WTSClientBuildNumber,
        WTSClientName,
        WTSClientDirectory,
        WTSClientProductId,
        WTSClientHardwareId,
        WTSClientAddress,
        WTSClientDisplay,
        WTSClientProtocolType
    }

    public enum WTS_CONNECTSTATE_CLASS
    {
        WTSActive,
        WTSConnected,
        WTSConnectQuery,
        WTSShadow,
        WTSDisconnected,
        WTSIdle,
        WTSListen,
        WTSReset,
        WTSDown,
        WTSInit
    }

    public static IntPtr OpenServer(String Name)
    {
        IntPtr server = WTSOpenServer(Name);
        return server;
    }

    public static void CloseServer(IntPtr ServerHandle)
    {
        WTSCloseServer(ServerHandle);
    }

    public static List<string> ListUsers(String ServerName)
    {
        IntPtr serverHandle = IntPtr.Zero;
        List<String> resultList = new List<string>();
        serverHandle = OpenServer(ServerName);

        try
        {
            IntPtr SessionInfoPtr = IntPtr.Zero;
            IntPtr userPtr = IntPtr.Zero;
            IntPtr domainPtr = IntPtr.Zero;
            IntPtr clientNamePtr = IntPtr.Zero;
            Int32 sessionCount = 0;
            Int32 retVal = WTSEnumerateSessions(serverHandle, 0, 1, ref SessionInfoPtr, ref sessionCount);
            Int32 dataSize = Marshal.SizeOf(typeof(WTS_SESSION_INFO));
            Int32 currentSession = (int)SessionInfoPtr;
            uint bytes = 0;
            if (retVal != 0)
            {
                for (int i = 0; i < sessionCount; i++)
                {
                    WTS_SESSION_INFO si = (WTS_SESSION_INFO)Marshal.PtrToStructure((System.IntPtr)currentSession, typeof(WTS_SESSION_INFO));
                    currentSession += dataSize;

                    WTSQuerySessionInformation(serverHandle, si.SessionID, WTS_INFO_CLASS.WTSUserName, out userPtr, out bytes);
                    WTSQuerySessionInformation(serverHandle, si.SessionID, WTS_INFO_CLASS.WTSDomainName, out domainPtr, out bytes);
                    WTSQuerySessionInformation(serverHandle, si.SessionID, WTS_INFO_CLASS.WTSClientName, out clientNamePtr, out bytes);

                    if(Marshal.PtrToStringAnsi(domainPtr).Length > 0 && Marshal.PtrToStringAnsi(userPtr).Length > 0)
                    {
                        if(Marshal.PtrToStringAnsi(clientNamePtr).Length < 1)                       
                            resultList.Add(Marshal.PtrToStringAnsi(domainPtr) + "\\" + Marshal.PtrToStringAnsi(userPtr) + "\tSessionID: " + si.SessionID + "\tClientName: n/a");
                        else
                            resultList.Add(Marshal.PtrToStringAnsi(domainPtr) + "\\" + Marshal.PtrToStringAnsi(userPtr) + "\tSessionID: " + si.SessionID + "\tClientName: " + Marshal.PtrToStringAnsi(clientNamePtr));
                    }
                    WTSFreeMemory(clientNamePtr);
                    WTSFreeMemory(userPtr);
                    WTSFreeMemory(domainPtr);
                }
                WTSFreeMemory(SessionInfoPtr);
            }
        }
        catch(Exception ex)
        {
            Console.WriteLine("Exception: " + ex.Message);
        }
        finally
        {
            CloseServer(serverHandle);
        }
        return resultList;
    }
}
'@

Add-Type $Code

Copy all of that into a file named QuerySessionInformation.ps1. Now launch the 32 bit version of Powershell in C:\Windows\SysWOW64\WindowsPowershell\v1.0. The code above uses pointers that will not work in a native 64 bit environment.

Now run the script. If you've never run the 32 bit version of Powershell on that server before, you will need to modify the script execution policy with Set-ExecutionPolicy, as 32 bit and 64 bit Powershell have separate execution policies. Note that there should be no output from the script itself, as all it is doing is compiling the .NET code and adding it to the current environment. Also note that once a type is added with Add-Type, you can not unload it without exiting that Powershell session... AFAIK. It makes debugging this sort of stuff really annoying as you have to restart Powershell every time you modify the code.

Now that the code is loaded, type this:

PS C:\> [RDPInfo]::ListUsers("REMOTESERVER") 

If there are any active user sessions on REMOTESERVER, the output will look like this:

DOMAIN\UserName SessionID: 2 ClientName: RYAN-PC 

This will work on remote computers as well as the local computer, but beware that if the user running this does not have sufficient permissions to the remote computer, it will fail silently (no output.)

PS: There are other bits of info in WTS_INFO_CLASS that may be of interest to you, such as WTSConnectState and WTSClientAddress. All you have to do is query for them with WTSQuerySessionInformation().

An Interesting DFSR Change (that probably everyone knew about but me)

First, wooo I hit 5K on ServerFault today.

I'm embarrassed to say that something I read about recently but didn't pay enough attention to at the time officially just bit me in the butt.  

A significant change occurred in January 2012 in the way that DFS Replication behaves.  Windows Server 2008 R2 SP1 post KB2663685 and Windows Server 2012 have changed the default behavior of DFSR.  Auto-recovery of DFSR replicated folders after unexpected shutdown is now disabled.  In other words, if a computer that hosts a DFSR replicated folder experiences an unexpected shutdown, DFSR will not automatically resume upon reboot.  (This includes Sysvol!

On older versions of Windows, DFSR auto-recovery was enabled.  I'm sure the reason for this change involves auto-recovery leading to unexpected rollbacks and unauthoritative conflict resolutions between replication partners, especially in wide-spread domains with high end-to-end replication latency and frequent changes... but even though the news was published, I for one didn’t pay enough attention to it and it has a very real effect on the way we manage our Windows systems that utilize DFSR going forward. 

So what if a domain controller or a file server with a DFSR share on it running 2008R2 or 2012 crashes unexpectedly, leaving the DFS database and the NTFS USN journal out of sync?  Then Sysvol no longer receives updates on that DC.  The DFSR file share no longer receives updates on that file server.  It's up to you to manually restart replication, and to resolve any conflicts with replication partners if changes took place during the time that the crashed server wasn't replicating. 

Luckily that is easy to do, and it's also possible to set the behavior back to auto-recovery if that is what you wish. 

How will I know if this effects my server?

While this first example is just a symptom of the problem, here is how it first came to my attention, triggering the investigation:

(Click on images for a better view.)

Errors in my event log

Application of Group Policy was failing, but only on DC02 and servers which were using DC02 as a domain controller.  Not DC01 or any server logged on by DC01.  As it turns out, the GPO referenced by that error event, a new GPO that I had just created on DC01, didn’t exist on DC02, hence the errors.  Sysvol did not seem to be replicating anymore.

Here is the actual event log event to let you know that DFS Replication has stopped on one or more volumes:

DFSR Error

Luckily, starting replication back up again is easy and the command to do it with your actual GUID, is right there in the event:

wmic /namespace:\\root\microsoftdfs path dfsrVolumeConfig where volumeGuid="12345678-ABCD-1234-EFGH-1A2B3C4E5F" call ResumeReplication

 You can also turn auto-recovery back on with wmic or by modifying the registry if you don’t have the time to be bothered by this:

HKLM\System\CurrentControlSet\Services\DFSR\Parameters\StopReplicationOnAutoRecovery = 0

Just be aware that auto-recovery can lead to unwanted rollbacks of DFSR data in some circumstances.

D-Root IPv4 Change - Do You Have the New One?

D-Root changed its IPv4 address back on January 3rd.  The old one is still up for now, but will be phased out in the future.  The old IPv4 address is 128.8.10.90.  The new address is 199.7.91.13. (d.root-servers.net IPv6 address is not changing.)

Today, someone asked me how to figure out, on the command-line, whether their Windows DNS server had the new updated d-root IP address or not.

After about 10 minutes of fiddling, here is what I came up with:

PS C:\> If($(Get-WmiObject -Namespace root\MicrosoftDNS MicrosoftDNS_ResourceRecord | ? DomainName -Match "root-servers.net" | ? OwnerName -Match "d.root-servers.net." | ? ContainerName -Match "..RootHints").IPAddress -ne "199.7.91.13") { Write-Host "YOUR D-ROOT IS OLD!" }

If your DNS server is still configured with the old root hints address for d-root, the command will notify you.