Finding Hidden Processes with Volatility and Scanning with Sysinternals Sigcheck

Happy Memory-Forensics-Thursday!

I get called in to go malware hunting every once in a while.  It's usually after an automatic vulnerability scanner has found something unusual about a particular computer on the network and threw up a red flag.  Once a machine is suspected of being infected, someone needs to go in and validate whether what the vulnerability scanner found is truly a compromise or a false positive, the nature of the infection, and clean it if possible.  I know that the "safest" reaction to the slightest whiff of malware is to immediately disconnect the machine from the network, format it and reinstall the operating system, but in a busy production environment, that extreme approach isn't always feasible or necessary.

We all know that no antivirus product can catch everything, nor is any vulnerability scanner perfect.  But a human with a bit of skill and the right tools can quickly sniff out things that AV has missed.  Malware hunting and forensic analysis really puts one's knowledge of deep Windows internals to the test, possibly more so than anything else, so I find it extremely fun and rewarding.

So today we're going to talk about two tools that will aid you in your journey.  Volatility and Sigcheck.

Volatility is a wondrous framework for analyzing Windows memory dumps.  You can find it here. It's free and open-source.  It's written in Python, but there is also a compiled exe version if you don't have Python installed.  Volatility is a framework that can run any number of plugins, and these plugins perform data analyses on memory dumps, focused on pointing out specific indicators of compromise, such as API hooks, hidden processes, hooked driver IRP functions, interrupt descriptor table hooks, and so much more.  It's not magic though, and it doesn't do much that you could not also do manually (and with much more painstaking effort) with WinDbg, but it does make it a hell of a lot faster and easier.  (We have to wait until 2014 for Win8/8.1 and Server 2012/2012R2 support.)

But first, before you can use Volatility, you must have a memory dump.  (There is a technology preview branch of Volatility that can read directly from the PhysicalMemory device object.)  There are many tools that can dump memory, such as WinPMem, which you can also find on the Volatility downloads page that I linked to earlier.  It can dump in both RAW format and DMP (Windows crash dump) formats.  Make sure that you download a version with signed drivers, as WinPmem loads a driver to do its business, and modern versions of Windows really don't like you trying to install unsigned drivers.  You can also use LiveKd to dump memory using the command .dump -f C:\memory.dmp.

Since Volatility is such a huge and versatile tool, today I'm only going to talk about one little piece of it - finding "hidden" processes.

When a process is created, the Windows kernel assigns it an _EPROCESS data structure.  Each _EPROCESS structure in turn contains a _LIST_ENTRY structure.  That _LIST_ENTRY structure contains a forward link and a backward link, each pointing to the next _EPROCESS structure on either side of it, creating a doubly-linked list that makes a full circle.  So if I wanted to know all of the processes running on the system, I could start with any process and walk through the _EPROCESS list until I got back to where I started.  When I use Task Manager, tasklist.exe or Process Explorer, they all use API functions that in turn rely on this fundamental mechanism.  Behold my awesome Paint.NET skills:


So if we wanted to hide a process from view, all we have to do is overwrite the backward link of the process in front of us and the forward link of the process behind us to point around us.  That will effectively "unlink" our process of doom, causing it to be hidden:


This is what we call DKOM - Direct Kernel Object Manipulation.  A lot of rootkits and trojans use this technique.  And even though modern versions of Windows do not allow user mode access to the \\Device\PhysicalMemory object, which is where the _EPROCESS objects will always be because they're in a non-paged pool, we don't need it, nor do we need to load a kernel mode driver, because we can pull off a DKOM attack entirely from user mode by using the ZwSystemDebugControl API.  But we can root out the rootkits with Volatility.  With the command

C:\> volatility.exe --profile=Win7SP0x86 -f Memory.raw psscan

That command shows a list of running processes, but it does it not by walking the _EPROCESS linked list, but by scanning for pool tags and constrained data items (CDIs) that correspond to processes.  The idea is that you compare that list with a list of processes that you got via traditional means, and processes that show up as alive and well on Volatility's psscan list but not Task Manager's list are hidden processes probably up to no good.

There are other methods of finding hidden processes.  For instance, scanning for DISPATCHER_HEADER objects instead of looking at pool tags.  Even easier, a handle to the hidden process should still exist in the handle table of csrss.exe (Client/Server Runtime Subsystem) even after it's been unlinked from the _EPROCESS list, so don't forget to look there.  (There's a csrss_pslist plugin for Volatility as well.)  Also, use the thrdscan plugin to check for threads that belong to processes that don't appear to exist, which would be another sign of tomfoolery.

Alright, so now you've located an executable file that you suspect is malware, but you're not sure.  Scan that sucker with Sigcheck!  Mark Russinovich recently added VirusTotal integration into Sigcheck, with the ability to automatically upload unsigned binaries and have them scanned by 40+ antivirus engines and give you back reports on whether the file appears to be malicious!  Sigcheck can automatically scan through an entire directory structure, just looking for suspicious binaries, uploading them to VirusTotal, and showing you the results.

Remember that you must accept VirusTotal's terms and conditions before using the service.

Uploading suspicious files to VirusTotal is practically a civic responsibility, as the more malicious signatures that VirusTotal has on file, the more effective the antivirus service is for the whole world.

An Ode to Server Fault

Server FaultToday marks a momentous occasion, as I have finally attained 20,000 reputation on Server Fault!  20k is by no means the reputation limit, and there are still plenty of other badges to be earned as well, but it is the last reputation-based milestone in my journey.  It comes with the title of "Trusted User" and grants an extra layer of powers on the site just shy of full moderator power.

It took me almost two years and 490 answers to achieve it. 

In case you don't know, Server Fault is a question and answer site that I have referenced many times before on this blog. It is one site that is part of a larger network of question and answer sites known collectively as Stack Exchange.  Server Fault is specifically aimed at IT professionals.  People who work with servers and networks in an administrative, engineering or architectural capacity to support a business's IT operations.  It is not about programming, nor is it about the enthusiast user at home setting up a Linksys router... though the lines can sometimes be blurry.  People come and ask questions on the site, such as "Halp I broke Active Directory" or "How do I SharePoint?", and we gain reputation for providing answers to those questions and have the community vote on them based on the quality of our answers. (Or lose reputation if your answers suck!)

20k reputation is actually just a drop in the bucket on some other Stack Exchange sites such as Stack Overflow, but the difference is that Server Fault only gets a fraction of the traffic that Stack Overflow gets.  I've chosen to focus on SF as it's most closely aligned with my own professional ambitions and interests.

Q: Why did I choose Server Fault over the TechNet forums or Experts Exchange?

It's been long enough that I barely remember first stumbling upon the site, but I know I stumbled upon it while researching some problem with WSUS or DHCP or Active Directory or something like that.  The site's aesthetic design was very attractive, and the layout made sense to me and it was clean and neat.  The questions covered a wide range of interesting things that were right up my alley.  I liked the idea of being "rewarded" for giving people good answers and rewarding others for their insight. Even if the reputation is totally intangible and practically meaningless, it still gives me a sense of progression and of having earned something.  In a way, it makes a game out of answering people's questions.  I know that the TechNet forums does reputation too, but the website doesn't look and feel as nice or have as many features, the questions aren't usually as varied and interesting, and the community (both the askers and the answerers) generally seem lower caliber.  Serverfault is chock full of features, including a sweet chat room where you can go and shoot the bull with other sysadmin-type people.

I quickly signed up, and before I knew it I was visiting the site every day to see what types of technology people were discussing and if there were any questions there that I could answer.  And after I found out the site ran on a mainly Microsoft stack (IIS, .NET, MS SQL, etc.,) I was totally in love.

My very first answer on SF*My very first answer on SF. The question was from a Windows Server admin, asking what scripting language he should learn.*

One of the quirks about Server Fault that I wouldn't see on the TechNet forums is that there are a lot of questions about Unix and Unix/Linux applications too.  That's a challenge for me because, in case you haven't noticed, I'm a Microsoft evangelist.  But that doesn't mean I'm a Linux hater.  I know that it's a very solid platform used by millions of people around the world and I want to learn about it too.  Even though I tend to opt for using Microsoft platforms and tools, I also get to see other people bringing *nix and Microsoft tech together in fascinating ways, such as this guy, who is setting up 1400 Samba4-based Active Directory Read Only Domain Controllers!

Q: Why would I waste my time answering other people's questions on the internet?

Ah.  This is where it gets interesting, you see, because it's not a waste of time.  In fact, spending time on Server Fault keeps my skills sharp.  Being constantly exposed to new problems, and people applying technology in interesting ways that I had never thought about and running into new types of issues that I had never needed to solve before.  Spending time on Server Fault is an investment in myself.  I know more about my industry because of that site.  It happens again and again that I'll end up reading 3000-word TechNet articles and digging through MSDN documentation on the Active Directory schema just in order to be able to answer someone else's Server Fault question.  That's personal enrichment.

And more importantly, I've made friends there.  People that I've had the pleasure of talking to over the phone and doing business with in real life.  I've stayed up many late, alcohol-fueled nights in the chat room with these guys talking about everything from FusionIO cards to the U.S. Constitution to why I should quit my job and go work with Mark. ;)

In fact, I'm hoping to meet up with some of these guys at TechEd 2014!


IPv4Address Attribute In Get-ADComputer

Guten Tag, readers!

Administrators who use Microsoft's Active Directory module for Powershell are most likely familiar with the Get-ADComputer cmdlet.  This cmdlet retrieves information from the Active Directory database about a given computer object.  Seems pretty straightforward, but recently I started wondering about something in Get-ADComputer's output:

Get-ADComputer IPv4Address

IPv4Address?  I don't recall that data being stored in Active Directory... well, not as an attribute of the computer objects themselves, anyway.  If you take a look at a computer object with ADSI Edit, the closest thing you'll find is an ipHostNumber attribute, but it appears to not be used:

ADSI Edit Computer Properties

Hmm... well, by this point, if you're anything like me, you're probably thinking that a DNS query is about the only other way that the cmdlet could be getting this data.  But I wasn't satisfied with just saying "it's DNS, dummy," and forgetting about it.  I wanted to know exactly what was going on under the hood.

So I started by disassembling the entire Microsoft.ActiveDirectory.Management assembly.  (How did I know which assembly to look for?)

After searching the resulting source code for ipv4, it started to become quite clear.  From Microsoft.ActiveDirectory.Management.Commands.ADComputerFactory<T>:

internal static void ToExtendedIPv4(string extendedAttribute, string[] directoryAttributes, ADEntity userObj, ADEntity directoryObj, CmdletSessionInfo cmdletSessionInfo)
  if (directoryObj.Contains(directoryAttributes[0]))
    string dnsHostName = directoryObj[directoryAttributes[0]].Value as string;
    userObj.Add(extendedAttribute, (object) IPUtil.GetIPAddress(dnsHostName, IPUtil.IPVersion.IPv4));
    userObj.Add(extendedAttribute, new ADPropertyValueCollection());

Alright, so now we know that Get-ADComputer is using another class named IPUtil to get the IP address of a computer as it runs. Let's go look at IPUtil:

internal static string GetIPAddress(string dnsHostName, IPUtil.IPVersion ipVersion)
  if (string.IsNullOrEmpty(dnsHostName))
    return (string) null;
    foreach (IPAddress ipAddress in Dns.GetHostEntry(dnsHostName).AddressList)
      if (ipAddress.AddressFamily == (AddressFamily) ipVersion && (ipVersion != IPUtil.IPVersion.IPv6 || !ipAddress.IsIPv6LinkLocal && !ipAddress.IsIPv6SiteLocal))
        return ipAddress.ToString();
    return (string) null;
  catch (SocketException ex)
    return (string) null;

Ahh, there it is.  The ole' trusty, tried and true System.Net.Dns.GetHostEntry() method.  The cmdlet is running that code every time you look up a computer object.  Also notice that the method returns on the first valid IP address that it finds, so we know that this cmdlet isn't going to work very well for computers with multiple IP addresses.  It would have been trivial to make the cmdlet return an array of all valid IP addresses instead, but alas, the Powershell developers did not think that was necessary.  And of course if the DNS query fails for any reason, you simply end up with a null for the IPv4Address field.

I've noticed that Microsoft's Active Directory cmdlets have many little "value-added" attributes baked into their cmdlets, but sometimes they can cause confusion, because you aren't sure where the data is coming from, or the "friendly" name that Powershell ascribes to an attribute doesn't match the attribute's name in Active Directory, etc.

Windows Emergency Management Services

BSODToday we're going to talk about one of the more esoteric features of Windows.  A feature that even some seasoned sysadmins don't know about, and that almost nobody outside of kernel debuggers and device driver writers in Redmond ever use...

Emergency Management Services!

Imagine you have a Windows computer that has suffered a blue screen of death. If you want to sound more savvy, you might call it a STOP error or a bug check. Pictured is a very old example of a BSoD, but it's just so much more iconic than the pretty new Win8 one with the giant frowny face on it.

So you're sitting there staring at a blue screen on the computer's console... can you still reboot the machine gracefully?  Or even crazier, could you still run, for example, Powershell scripts on this machine even after it has suffered some massive hardware failure?

Don't reach for that power button just yet, because yes you can!

You might have thought that once a Windows computer has blue-screened, then it's done. It's stopped forever and it cannot execute any more code, period.  I thought that myself for a long time. But lo and behold, there's still a little juice left even after you've blue-screened, and all you need is a serial or USB cable.  That's where Emergency Management Services comes in.

As the name implies, EMS is typically there for when all else fails. For when your computer has already gone to hell in a handbasket. You could consider it an out-of-band management solution.

Of course you need to have already enabled it beforehand, not after a bug check has already occurred. You'd enable it on Vista/2008 and above like so:


If using a USB port, or


If using an RS-232 serial port. (How quaint.)

Now that it's enabled, you can connect to the Special Administration Console (SAC.)

SAC Special Administration Console

From here, you can launch a command prompt (Cmd.exe,) and from there, you can launch Powershell.exe!  All over a serial or USB cable connection. If the regular SAC mode cannot be entered for some reason, then EMS will put you in !SAC mode, where you can still at least read the event logs and reboot the server in a more graceful manner than just pulling the plug.

Mark Russinovich has this to say about the Windows boot up process as it concerns EMS:

"At this point, InitBootProcessor enumerates the boot-start drivers that were loaded by Winload and calls DbgLoadImageSymbols to inform the kernel debugger (if attached) to load symbols for each of these drivers. If the host debugger has configured the break on symbol load option, this will be the earliest point for a kernel debugger to gain control of the system. InitBootProcessor now calls HvlInit System, which attempts to connect to the hypervisor in case Windows might be running inside a Hyper-V host system’s child partition. When the function returns, it calls HeadlessInit to initialize the serial console if the machine was configured for Emergency Management Services (EMS)."
Mark Russinovich, David Solomon, Alex Ionescu, Windows Internals 6th Ed.

So there you have it. Even when faced with a BSoD, if you have an opportunity to shut down or reboot the machine in a more graceful manner than just pulling the electricity from it, then you should do it.

More Windows and AD Cryptography Mumbo-Jumbo

I've still had my head pretty deep into cryptography and hashing as far as Windows and Active Directory is concerned, and I figured it was worth putting here in case you're interested.  We're going to talk about things like NTLM and how Windows stores hashes, and more.

The term NTLM is a loaded one, as the acronym is often used to refer to several different things.  It not only refers to Microsoft’s implementation of another standard algorithm for creating hashes, but it also refers to a network protocol.  The NTLM used for storing password hashes on disk (aka NT hash) is a totally different thing than the NTLM used to transmit authentication data across a TCP/IP network.  There’s the original LAN Manager protocol, which is worse than NT LAN Manager (NTLM or NTLMv1,) which is worse than NTLMv2, which is worse than NTLMv2 with Session Security and so on…  but an NT hash is an NT hash is an NT hash.  When we refer to either NTLMv1 or NTLMv2 specifically, we’re not talking about how the password gets stored on disk, we’re talking about network protocols.

Also, this information refers to Vista/2008+ era stuff.  I don’t want to delve into the ancient history Windows NT 4 or 2000, so let’s not even discuss LAN Manager/LM.  LM hashes are never stored or transmitted, ever, in an environment that consists of Vista/2008+ stuff.  It’s extinct.

Unless some bonehead admin purposely turned it back on.  In which case, fire him/her.

Moving on…


So here we talk about what goes on in a regular Windows machine with no network connection.  No domain.  All local stuff. No network.


You might ask.  An NT hash is simply the MD4 hash of the little endian UTF-16 encoded plaintext input.  So really it’s MD4.

"So the Windows Security Accounts Manager (SAM) stores MD4 hashes of passwords to disk, then?"

Well no, not directly.  The NT hashes, before being stored, are encrypted with RC4 using the machine’s "boot key," which is both hard to get at as well as being unique to each OS install.  By "hard to get at," I mean that the boot key is scattered across several areas of the registry that require system level access and you have to know how to read data out of the registry that cannot be seen in Regedit, even if you run it as Local System. And it must be de-obfuscated on top of that.  After hashing the boot key and other bits of data with MD5, you then create another RC4 key from the MD5 hash of the hashed bootkey, plus some more information such as the specific user's security identifier.

So the final answer is that Windows stores local SAM passwords to disk in the registry as RC4-encrypted MD4 hashes using a key that is unique to every machine and is difficult to extract and descramble, unless you happen to be using one of the dozen or so tools that people have written to automate the process.

Active Directory is a different matter altogether.  Active Directory does not store domain user passwords in a local SAM database the same way that a standalone Windows machine stores local user passwords.  Active Directory stores those password hashes in a file on disk named NTDS.dit.  The only password hash that should be stored in the local SAM of a domain controller is the Directory Services Restore Mode password.  The algorithms used to save passwords in NTDS.dit are much different than the algorithms used by standalone Windows machines to store local passwords.  Before I tell you what those algorithms are, I want to mention that the algorithms AD uses to store domain password hashes on disk should not be in scope to auditors, because NTDS.dit is not accessible by unauthorized users.  The operating system maintains an exclusive lock on it and you cannot access it as long as the operating system is running.  Because of that,  the online Directory Services Engine and NTDS.dit together should be treated as one self-contained ‘cryptographic module’ and as such falls under the FIPS 140-2 clause:

"Cryptographic keys stored within a cryptographic module shall be stored either in plaintext form or encrypted form.  Plaintext secret and private keys shall not be accessible from outside the cryptographic module to unauthorized operators…"

So even plaintext secrets are acceptable to FIPS 140, as long as they stay within the cryptographic module and cannot be accessed by or sent to outsiders.

Active Directory stores not only the hashed password of domain users, but also their password history.  This is useful for that “Remember the last 24 passwords” Group Policy setting.  So there are encrypted NT hashes stored in NTDS.dit.  Let’s just assume we have an offline NTDS.dit – again, this should not be of any concern to auditors – this is Microsoft proprietary information and was obtained through reverse engineering.  It’s only internal to AD.  FIPS should not be concerned with this because this all takes place "within the cryptographic module."  Access to offline copies of NTDS.dit should be governed by how you protect your backups.

To decrypt a hash in NTDS.dit, first you need to decrypt the Password Encryption Key (PEK) which is itself encrypted and stored in NTDS.dit.  The PEK is the same across all domain controllers, but it is encrypted using the boot key (yes the same one discussed earlier) which is unique on every domain controller.  So once you have recovered the bootkey of a domain controller (which probably means you have already completely owned that domain controller and thus the entire domain so I'm not sure why you'd even be doing this) you can decrypt the PEK contained inside of an offline copy of NTDS.dit that came from that same domain controller.  To do that, you hash the bootkey 1000 times with MD5 and then use that result as the key to the RC4 cipher.  The only point to a thousand rounds of hashing is to make a brute force attack more time consuming.

OK, so now you’ve decrypted the PEK.  So use that decrypted PEK, plus 16 bytes of the encrypted hash itself as key material for another round of RC4.  Finally, use the SID of the user whose hash you are trying to decrypt as the key to a final round of DES to uncover, at last, the NT (MD4) hash for that user.

Now you need to brute-force attack that hash.  Using the program ighashgpu.exe, which uses CUDA to enable all 1344 processing cores on my GeForce GTX 670 graphics card to make brute force attempts on one hash in parallel, I can perform over 4 billion attempts per second to eventually arrive at the original plaintext password of the user.  It doesn’t take long to crack an NT hash any more.

As a side-note, so-called "cached credentials" are actually nothing more than password verifiers.  They’re essentially a hash of a hash, and there is no reversible information contained in a "cached credential" or any information that is of any interest to an attacker.  "Cached credentials" pose no security concern, yet most security firms, out of ignorance, still insist that they be disabled.

So there you have it.  You might notice that nowhere in Windows local password storage or Active Directory password storage was the acronym SHA ever used.  There is no SHA usage anywhere in the above processes, at all. 



Now passing authentication material across the network is an entirely different situation!

I’ll start with the bad news.  Remember earlier when I talked about brute-forcing the decrypted NT hash of another user?  Well that last step is often not even necessary.  NT hashes are password-equivalent, meaning that if I give Windows your hash, it’s as good as giving Windows your password in certain scenarios.  I don’t even need to know what your actual password is.  This is the pass-the-hash attack that you might have heard of.  But it’s not as bad as it sounds.

The good news is that neither Windows nor Active Directory ever sends your bare NT hash over the wire during network transmission.  And you cannot begin a pass-the-hash attack until you’ve already taken over administrative control of some domain joined machine.  That means there is no such thing as using pass-the-hash to own an entire networked AD environment just from an anonymous observer sniffing network packets.  That’s not how it works.  (Fun video on PtH)

Now that we know what an NT hash is, it’s a good time to draw the distinction that whenever we talk about specifically NTLMv1 and NTLMv2, we’re not actually talking about NT hashes anymore.  We’re talking about network communication protocols.  The whole mess is often just called NTLM as a blanket term because it’s all implemented by Microsoft products and it’s all interrelated.

Both NTLMv1 and NTLMv2 are challenge-response protocols, where Client and Server challenge and respond with each other such that Server can prove that Client knows what his/her password is, without ever actually sending the password or its hash directly over the network.  This is because Server already knows Client’s password (hash), either because it’s stored in the local SAM in the case of local Windows accounts, or because Server can forward Client’s data to a domain controller, and the domain controller can verify and respond to Server with “Yep, that’s his password, alright.”

With NTLMv1, you’ll see some usage of DES during network communication. 

With NTLMv2, you’ll see some usage of HMAC-MD5.

There’s also NTLM2 Session, aka NTLMv2 With Session Security, but it uses the same encryption and hashing algorithms as NTLMv2.

It is possible to completely remove the usage of NTLM network protocols from an Active Directory domain and go pure Kerberos, but it will break many applications.  Here is a fantastic article written by one of my favorite Microsoft employees about doing just that.

So let’s assume that hypothetically we blocked all usage of NTLM network protocols and went pure Kerberos. Kerberos in AD supports only the following encryption:

DES_CBC_CRC    (Disabled by default as of Win7/2008R2)[Source]
DES_CBC_MD5    (Disabled by default as of Win7/2008R2)
RC4_HMAC_MD5   (Disabled by default as of Win7/2008R2)
Future encryption types


Of course, there are plenty of other Windows applications that pass authentication traffic over the network besides just AD.  Remote Desktop is a great example.  Remote Desktop traditionally uses RC4, but modern versions of Remote Desktop will negotiate a Transport Layer Security (TLS) connection wherever possible.  (Also known as Network Level Authentication (NLA).)   This is great news because this TLS connection uses the computer’s digital certificate, and that certificate can be automatically created and assigned to the computer by an Enterprise Certificate Authority, and that certificate can be capable of SHA256, SHA384, etc.  However the Certificate Authority administrator defines it.

If you turn on FIPS mode, Remote Desktop can only use TLS 1.0 (as opposed to SSL) when NLA is negotiated, and it can only use 3DES_CBC instead of RC4 when TLS is not negotiated.

Other ciphers that are turned off when FIPS mode is turned on include:

-          TLS_RSA_WITH_RC4_128_SHA

-          TLS_RSA_WITH_RC4_128_MD5

-          SSL_CK_RC4_128_WITH_MD5

-          SSL_CK_DES_192_EDE3_CBC_WITH_MD5

-          TLS_RSA_WITH_NULL_MD5



That will apply to all applications running on Windows that rely upon Schannel.dll.  The application will crash if it calls upon one of the above ciphers when FIPS mode is enabled.

So anyway, that’s about all I got right now.  If you made it to the bottom of  this post I should probably buy you a drink!