Anti–Dynamic Analysis Techniques

None of the anti–static analysis techniques covered in the past few sections have any effect whatsoever on whether a program will actually execute or not. In fact, while they may make it difficult for you to comprehend the true behavior of a program using static analysis techniques alone, they can’t prevent the program from executing, or they would render a program useless from the start and therefore eliminate the need to analyze the program at all.

Given that a program must run in order for it to do any work, dynamic analysis aims to observe the behavior of a program in motion (while it is running) rather than observe the program at rest (using static analysis while the program is not running). In this section we briefly summarize some of the more common anti–dynamic analysis techniques. For the most part, these techniques have little effect on static analysis tools; however, where there is overlap, we will point this out. We will return to discuss the impact of many of these techniques on IDA’s integrated debugger beginning in Chapter 24.

Detecting Virtualization

One of the most common choices for configuring a sandbox environment is to make use of virtualization software, such as VMware, to provide an execution environment for malicious software (or, for that matter, any other software of interest). The advantage of such environments is that they typically offer checkpoint and rollback capabilities that facilitate rapid restoration of the sandbox to a known clean state. The primary disadvantage of using such environments as the foundation for a sandbox is the fact that it is fairly easy (especially on 32-bit x86 platforms) for a program to detect that it is running within a virtualized environment. Under the assumption that virtualization equates to observation, many programs that want to remain undetected simply choose to shut down once they determine that they are running within a virtual machine.

The following list describes a few of the techniques that have been used by programs running in virtualized environments to determine that they are running within a virtual machine rather than on native hardware.

Detection of virtualization–specific software

Users often install helper applications within virtual machines to facilitate communications between a virtual machine and its host operating system or simply to improve performance within the virtual machine. The VMware Tools collection is one example of such software. The presence of such software is easily detected by programs running within the virtual machine. For example, when VMware Tools is installed into a Microsoft Windows virtual machine, it creates Windows registry entries that can be read by any program. VMware Tools is rarely required in order to run malware within a virtual environment and should not be installed so as to eliminate such trivially detectable traces of the virtual machine.

Detection of virtualization–specific hardware

Virtual machines make use of virtual hardware abstraction layers to provide the interface between the virtual machine and the host computer’s native hardware. Characteristics of the virtual hardware are often easily detectable by software running within the virtual machine. For example, VMware has been assigned its own organizationally unique identifiers (OUI)^[170] for use with its virtualized network adapters. Observing a VMware-specific OUI is a good indication that a program is running within a virtual machine. Note that it is usually possible to modify the MAC address assigned to virtual network adapters using configuration options on the host computer.

Detection of virtual machine–specific behaviors

Some virtualization platforms contain backdoor-style communications channels to facilitate communications between a virtual machine and its host software. For example, the following five lines may be used to determine if you are running within a VMware virtual machine:^[171]

mov  eax, 0x564D5868    ; 'VMXh'
  mov  ecx, 10
  xor  ebx, ebx
  mov  dx,  0x5658        ; 'VX'
 in   eax, dx

The sequence will result in the EBX register containing the value 0x564D5868 if you are inside a virtual machine. If you are not within a virtual machine, the code will result in either an exception or no change to EBX, depending on the host operating system in use. This instruction sequence takes advantage of the fact that the x86 in instruction is generally not used or allowed in user-space programs; however, within VMware, the instruction sequence can be used to test for the presence of the channel used by VMware guest operating systems to communicate with their host operating system. This channel is used by VMware Tools, for example, to facilitate the exchange of data (such as clipboard contents) between the host and guest operating systems.

Detection of processor-specific behavioral changes

Perfect virtualization is a difficult thing to achieve. Ideally a program should not be able to detect any difference between a virtualized environment and native hardware. However, this is seldom the case. Joanna Rutkowska developed her redpill^[172] VMware-detection technique after observing behavioral differences between the operation of the x86 sidt instruction on native hardware and the same instruction executed within a virtual machine environment.

Though it is not the first paper on the topic, “On the Cutting Edge: Thwarting Virtual Machine Detection” by Tom Liston and Ed Skoudis^[173] presents a nice overview of virtual machine–detection techniques.

Detecting Instrumentation

Following creation of your sandbox environment and prior to executing any program you want to observe, you need to ensure that instrumentation is in place to properly collect and record information about the behavior of the program you are analyzing. A wide variety of tools exists for performing such monitoring tasks. Two widely used examples include Process Monitor,^[174] from the Sysinternals group^[175] at Microsoft, and Wireshark.^[176] Process Monitor is a utility capable of monitoring certain activities associated with any running Windows process, including accesses to the Windows registry and file system activity. Wireshark is a network packet capture and analysis tool often used to analyze the network traffic generated by malicious software.

Malware authors with a sufficient level of paranoia may program their software to search for running instances of such monitoring programs. Techniques range from scanning the active process list for process names known to be associated with such monitoring software to scanning the title bar text for all active Windows applications to search for known strings. Deeper searches can be performed, with some software going so far as to search for specific characteristics associated with Windows GUI components used within certain instrumentation software. For example, the WinLicense obfuscation/protection program uses the following function call to attempt to determine whether the Filemon (a predecessor of Process Monitor) utility is currently executing:

if (FindWindow("FilemonClass", NULL)) {
   //exit because Filemon is running
}

In this case, the FindWindow function is being used to search for a top-level application window based on the registered class name ("FilemonClass") of the window rather than the window’s title. If a window of the requested class is located, then Filemon is assumed to be executing, and the program terminates.

Detecting Debuggers

Moving beyond simple observation of a program, the use of a debugger allows an analyst to take complete control of the execution of program that requires analyzing. A common use of a debugger with obfuscated programs is to run the obfuscated program just long enough to complete any decompression or decryption tasks and then utilize the debugger’s memory-access features to extract the de-obfuscated process image from memory. In most cases, standard static analysis tools and techniques can be used to complete the analysis of the extracted process image.

The authors of obfuscation utilities are well aware of such debugger-assisted de-obfuscation techniques, so they have developed measures to attempt to defeat the use of debuggers for execution of their obfuscated programs. Programs that detect the presence of a debugger often choose to terminate rather than proceed with any operations that might allow an analyst to more easily determine the behavior of the program.

Techniques for detecting the presence of debuggers range from simple queries to the operating system via well-known API functions, such as the Windows IsDebuggerPresent function, to lower-level checks for memory or processor artifacts resulting from the use of a debugger. An example of the latter includes detecting that a processor’s trace (single-step) flag is set. Detection of specific debuggers is also possible in some cases. For example, SoftIce, a Windows kernel debugger, can be detected through the presence of the "\\.\NTICE" device, which is used to communicate with the debugger.

As long as you know what to look for, there is nothing terribly tricky about trying to detect a debugger, and attempts to do so are easily observed during static analysis (unless anti–static analysis techniques are employed simultaneously). For more information on debugger detection, consult Nicolas Falliere’s article “Windows Anti-Debug Reference,”^[177] which provides a comprehensive overview of Windows anti-debugging techniques.^[178] In addition, OpenRCE maintains an Anti Reverse Engineering Techniques Database,^[179] which contains a number of debugger-specific techniques.

Preventing Debugging

If a debugger manages to remain undetectable, there are still a number of techniques available to thwart its use. These additional techniques attempt to confound the debugger by introducing spurious breakpoints, clearing hardware breakpoints, hindering disassembly to make selection of appropriate breakpoint addresses difficult, or preventing the debugger from attaching to a process in the first place. Many of the techniques discussed in Nicolas Falliere’s article are geared toward preventing debuggers from operating correctly.

Intentionally generating exceptions is one means by which a program may attempt to hinder debugging. In most cases, an attached debugger will catch the exception, and the user of the debugger is faced with the task of analyzing why the exception occurred and whether to pass the exception along to the program being debugged. In the case of a software breakpoint such as the x86 int 3, it may be difficult to distinguish a software interrupt generated by the underlying program from one that results from an actual debugger breakpoint. This confusion is exactly the effect that is desired by the creator of the obfuscated program. In such cases, careful analysis of the disassembly listing to understand the true program flow is usually possible, though the level of effort for static analysis is raised somewhat.

Encoding portions of a program in some manner has the dual effect of hindering static analysis because disassembly is not possible and of hindering debugging because placing breakpoints is difficult. Even if the start of each instruction is known, software breakpoints cannot be placed until the instructions have actually been decoded, as altering the instructions by inserting a software breakpoint is likely to result in a failed decryption of the obfuscated code and a resulting crash of the program when execution reaches the intended breakpoint.

Alternatively, some de-obfuscation routines compute checksum values over ranges of bytes within the process. If one or more software breakpoints have been set within the range over which a checksum is being computed, the resulting checksum will be incorrect, and the program is likely to abort.

The Shiva ELF obfuscation tool for Linux makes use of a technique called mutual ptrace to prevent the use of a debugger in analyzing Shiva’s behavior.

PROCESS TRACING

The ptrace, or process tracing, API is available on many Unix systems and provides a mechanism for one process to monitor and control the execution of another process. The GNU debugger (gdb) is one of the more well-known applications that makes use of the ptrace API. Using the ptrace API, a ptrace parent process may attach to and control the execution of a ptrace child process. In order to begin controlling a process, a parent process must first attach to the child process that it seeks to control. Once attached, the child process is stopped anytime it receives a signal, and the parent is notified of this fact via the POSIX wait function, at which point the parent may choose to alter or inspect the state of the child process before instructing the child process to continue execution. Once a parent process has attached to a child process, no other process may attach to the same child process until the tracing parent chooses to detach from the child process.

Shiva takes advantage of the fact that a process may be ptraced by only one other process at any given time. Early in its execution, the Shiva process forks to create a copy of itself. The original Shiva process immediately performs a ptrace attach operation on the newly forked child. The newly forked child process, in turn, immediately attaches to its parent process. If either attach operation fails, Shiva terminates under the assumption that another debugger is being used to monitor the Shiva process. If both operations succeed, then no other debugger can be used to attach to the running Shiva pair, and Shiva can continue to run without fear of being observed. While operating in this manner, either Shiva process may alter the state of the other, making it difficult to determine, using static analysis techniques, what the exact control flow path is through the Shiva binary.

^[170]An OUI makes up the first three bytes of a network adapter’s factory-assigned MAC address.

^[171]See http://www.codeproject.com/KB/system/VmDetect.aspx by Elias Bachaalany.

^[172]See http://www.invisiblethings.org/papers/redpill.html

^[173]See http://handlers.sans.org/tliston/ThwartingVMDetection_Liston_Skoudis.pdf

^[174]See http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

^[175]See http://technet.microsoft.com/en-us/sysinternals/default.aspx

^[176]See http://www.wireshark.org/.

^[177]See http://www.symantec.com/connect/articles/windows-anti-debug-reference/.

^[178]See http://pferrie.tripod.com/papers/unpackers.pdf/ by Peter Ferrie.

^[179]See http://www.openrce.org/reference_library/anti_reversing/

Previous Chapter

21. Obfuscated Code Analysis