Finding Hidden Processes with Volatility and Scanning with Sysinternals Sigcheck

Happy Memory-Forensics-Thursday!

I get called in to go malware hunting every once in a while.  It's usually after an automatic vulnerability scanner has found something unusual about a particular computer on the network and threw up a red flag.  Once a machine is suspected of being infected, someone needs to go in and validate whether what the vulnerability scanner found is truly a compromise or a false positive, the nature of the infection, and clean it if possible.  I know that the "safest" reaction to the slightest whiff of malware is to immediately disconnect the machine from the network, format it and reinstall the operating system, but in a busy production environment, that extreme approach isn't always feasible or necessary.

We all know that no antivirus product can catch everything, nor is any vulnerability scanner perfect.  But a human with a bit of skill and the right tools can quickly sniff out things that AV has missed.  Malware hunting and forensic analysis really puts one's knowledge of deep Windows internals to the test, possibly more so than anything else, so I find it extremely fun and rewarding.

So today we're going to talk about two tools that will aid you in your journey.  Volatility and Sigcheck.

Volatility is a wondrous framework for analyzing Windows memory dumps.  You can find it here. It's free and open-source.  It's written in Python, but there is also a compiled exe version if you don't have Python installed.  Volatility is a framework that can run any number of plugins, and these plugins perform data analyses on memory dumps, focused on pointing out specific indicators of compromise, such as API hooks, hidden processes, hooked driver IRP functions, interrupt descriptor table hooks, and so much more.  It's not magic though, and it doesn't do much that you could not also do manually (and with much more painstaking effort) with WinDbg, but it does make it a hell of a lot faster and easier.  (We have to wait until 2014 for Win8/8.1 and Server 2012/2012R2 support.)

But first, before you can use Volatility, you must have a memory dump.  (There is a technology preview branch of Volatility that can read directly from the PhysicalMemory device object.)  There are many tools that can dump memory, such as WinPMem, which you can also find on the Volatility downloads page that I linked to earlier.  It can dump in both RAW format and DMP (Windows crash dump) formats.  Make sure that you download a version with signed drivers, as WinPmem loads a driver to do its business, and modern versions of Windows really don't like you trying to install unsigned drivers.  You can also use LiveKd to dump memory using the command .dump -f C:\memory.dmp.

Since Volatility is such a huge and versatile tool, today I'm only going to talk about one little piece of it - finding "hidden" processes.

When a process is created, the Windows kernel assigns it an _EPROCESS data structure.  Each _EPROCESS structure in turn contains a _LIST_ENTRY structure.  That _LIST_ENTRY structure contains a forward link and a backward link, each pointing to the next _EPROCESS structure on either side of it, creating a doubly-linked list that makes a full circle.  So if I wanted to know all of the processes running on the system, I could start with any process and walk through the _EPROCESS list until I got back to where I started.  When I use Task Manager, tasklist.exe or Process Explorer, they all use API functions that in turn rely on this fundamental mechanism.  Behold my awesome Paint.NET skills:

EPROCESS Good

So if we wanted to hide a process from view, all we have to do is overwrite the backward link of the process in front of us and the forward link of the process behind us to point around us.  That will effectively "unlink" our process of doom, causing it to be hidden:

EPROCESS BAD

This is what we call DKOM - Direct Kernel Object Manipulation.  A lot of rootkits and trojans use this technique.  And even though modern versions of Windows do not allow user mode access to the \\Device\PhysicalMemory object, which is where the _EPROCESS objects will always be because they're in a non-paged pool, we don't need it, nor do we need to load a kernel mode driver, because we can pull off a DKOM attack entirely from user mode by using the ZwSystemDebugControl API.  But we can root out the rootkits with Volatility.  With the command

C:\> volatility.exe --profile=Win7SP0x86 -f Memory.raw psscan

That command shows a list of running processes, but it does it not by walking the _EPROCESS linked list, but by scanning for pool tags and constrained data items (CDIs) that correspond to processes.  The idea is that you compare that list with a list of processes that you got via traditional means, and processes that show up as alive and well on Volatility's psscan list but not Task Manager's list are hidden processes probably up to no good.

There are other methods of finding hidden processes.  For instance, scanning for DISPATCHER_HEADER objects instead of looking at pool tags.  Even easier, a handle to the hidden process should still exist in the handle table of csrss.exe (Client/Server Runtime Subsystem) even after it's been unlinked from the _EPROCESS list, so don't forget to look there.  (There's a csrss_pslist plugin for Volatility as well.)  Also, use the thrdscan plugin to check for threads that belong to processes that don't appear to exist, which would be another sign of tomfoolery.

Alright, so now you've located an executable file that you suspect is malware, but you're not sure.  Scan that sucker with Sigcheck!  Mark Russinovich recently added VirusTotal integration into Sigcheck, with the ability to automatically upload unsigned binaries and have them scanned by 40+ antivirus engines and give you back reports on whether the file appears to be malicious!  Sigcheck can automatically scan through an entire directory structure, just looking for suspicious binaries, uploading them to VirusTotal, and showing you the results.

Remember that you must accept VirusTotal's terms and conditions before using the service.

Uploading suspicious files to VirusTotal is practically a civic responsibility, as the more malicious signatures that VirusTotal has on file, the more effective the antivirus service is for the whole world.

ShareDiscreetlyWebServer v1.0.1.2

Several improvements over the last release the past few days:

  • Some code optimizations. Pages are rendering about an order of magnitude faster now.
  • Just about all the HTML and Javascript has been exported to editable files so that an administrator can change up the code, color schemes, branding, etc., without needing to recompile the code.
  • The server can now send an S/MIME, digitally signed email to the person you want to send the URL to. Unfortunately some email clients (such as the Gmail web client) don't natively understand S/MIME, but Outlook handles it just fine. You can also get various plugins and programs to read S/MIME emails if your email client doesn't understand it. I'd rather lean toward more security-related bells and whistles than max compatibility for this project.

You can access the secret server at https://myotherpcisacloud.com.

Users.exe v1.0.0.3

In my last post, I showed how to get RDP/TS session information from a local or remote computer, including ClientName, on the command line using Powershell. (But it's not really Powershell. It's a PS wrapper around some C# that P/Invokes a native API. Which is pretty nuts but it works.)  Since I can't think of any other way to get this information on the command line, I thought it would be useful to convert the thing to native code. I named the program users.exe. It's written in C. Here is what it looks like:

C:\Users\joe\Desktop>users /?

USAGE: users.exe [hostname]

Not specifying a hostname implies localhost.
This command will return information about users currently logged onto
a local or remote computer, including the client's hostname and IP.
Users.exe was written by Ryan Ries.

C:\Users\joe\>users SERVER01

Session ID  : 0
Domain\User : System
Client Name : Local
Net Address : n/a
Conn. State : Disconnected

Session ID  : 1
Domain\User : System
Client Name : Local
Net Address : n/a
Conn. State : Connected

Session ID  : 25
Domain\User : DOMAIN\joe
Client Name : JOESLAPTOP
Net Address : 10.122.124.21 (AF_INET)
Conn. State : Active

I had a pretty rough time shaking the rust off of my native/unmanaged code skills, but I pulled it off.  That said, if you wanna give it a try, I would very much appreciate any bug reports/feature requests.

users.exe (63.50 kb)

EventLogClearer v1.1.3.22

I have released an updated version of my EventLogClearer, bringing it up to version 1.1.3.22. For the original release, see this post.

EventLogClearer 1.1.3.22

Improvements made in this version include:

  • Fixed a bug where the application acted weird if you ran the log clearing procedure two or more times in a row.
  • Added a new mechanism for supplying alternate credentials, instead of only being able to run as the currently logged on user. This applies to both auto-populating the list of computers from AD, and running the event log clearing procedure. If you leave the credentials blank or as the default, "username," the current user will be used.
  • Added the ability to clear a ton more Applications and Services logs than before, due to me realizing the potential of the EventLogSession class.

As before, .NET 4.5 is required to run the application. The project was built in Visual Studio 2012.

Here is the executable: EventLogClearer-1.1.3.22-exe.zip (68.71 kb)

Here is the source code: EventLogClearer-1.1.3.22-source.zip (308.11 kb)

Log Parser 2.2 and Log Parser Studio

At first I thought to title this post the same as the catchphrase of Log Parser: "The Whole World Is Your Database!"

But then I decided that was a bit too exciting for what I actually wanted to talk about.

So I just discovered Log Parser Studio a few days ago. LPS is a graphical frontend to Log Parser; quite similar to how SQL Management Studio is a GUI frontend to interacting with SQL Server.  I am, quite frankly, ashamed that I didn't already know about it. It's fantastic.

The thing is... Log Parser is a command-line utility that uses a very SQL-esque language to interact with logs. What kind of logs, you ask?  Any kind of logs! That's right... you can use it to query the Windows Security Event Log, or you can use it to query a folder full of IIS web server logs, or you can use it to query a log full of your own personal electric utility bills from last year!

However, Log Parser itself is a very complex, albeit powerful and flexible, command-line utility. Maybe you want something a little more user-friendly to get you started. That's exactly where Log Parser Studio, the GUI frontend, comes in to play.

As a little demonstration, I installed Log Parser 2.2 on my workstation. Then I downloaded Log Parser Studio to my workstation. I fired it up as a Windows application, and I pointed it to the remote IIS logs directory of this very web server. I then right-clicked on "IIS: Request per Hour" and chose "Run report now." As if I had just run a SQL query in SQL Management Studio, this window popped up:

 

Log Parser Studio Query*Click for Bigger*

This data is probably every single HTTP GET request made per hour, rather than a count of hits made by unique IP addresses, but the point is you now have this amazing utility that will parse practically any amount of data you can think of from any source of data you can think of. Go check it out and see how Log Parser is even capable of generating pie charts and bar charts and all sorts of crazy things using this data!