Windows Debugger Detection

Malware uses a variety of techniques to scan for indications that a debugger is attached, including using the Windows API, manually checking memory structure for debugging artifacts, and searching the system for residue left by a debugger. Debugger detection is the most common way that malware performs anti-debugging.

Using the Windows API

The use of Windows API functions is the most obvious of the anti-debugging techniques. The Windows API provides several functions that can be used by a program to determine if it is being debugged. Some of these functions were designed for debugger detection; others were designed for different purposes but can be repurposed to detect a debugger. A few of these functions use functionality not documented in the API.

Typically, the easiest way to overcome a call to an anti-debugging API function is to manually modify the malware during execution to not call these functions or to modify the flag’s post call to ensure that the proper path is taken. A more difficult option would be to hook these functions, as with a rootkit.

The following Windows API functions can be used for anti-debugging:

IsDebuggerPresent

The simplest API function for detecting a debugger is IsDebuggerPresent. This function searches the Process Environment Block (PEB) structure for the field IsDebugged, which will return zero if you are not running in the context of a debugger or a nonzero value if a debugger is attached. We’ll discuss the PEB structure in more detail in the next section.

CheckRemoteDebuggerPresent

This API function is nearly identical to IsDebuggerPresent. The name is misleading though, as it does not check for a debugger on a remote machine, but rather for a process on the local machine. It also checks the PEB structure for the IsDebugged field; however, it can do so for itself or another process on the local machine. This function takes a process handle as a parameter and will check if that process has a debugger attached. CheckRemoteDebuggerPresent can be used to check your own process by simply passing a handle to your process.

NtQueryInformationProcess

This is a native API function in Ntdll.dll that retrieves information about a given process. The first parameter to this function is a process handle; the second is used to tell the function the type of process information to be retrieved. For example, using the value ProcessDebugPort (value 0x7) for this parameter will tell you if the process in question is currently being debugged. If the process is not being debugged, a zero will be returned; otherwise, a port number will be returned.

OutputDebugString

This function is used to send a string to a debugger for display. It can be used to detect the presence of a debugger. For example, Example 16-1 uses SetLastError to set the current error code to an arbitrary value. If OutputDebugString is called and there is no debugger attached, GetLastError should no longer contain our arbitrary value, because an error code will be set by the OutputDebugString function if it fails. If OutputDebugString is called and there is a debugger attached, the call to OutputDebugString should succeed, and the value in GetLastError should not be changed.

Example 16-1. OutputDebugString anti-debugging technique

DWORD errorValue = 12345;
SetLastError(errorValue);

OutputDebugString("Test for Debugger");

if(GetLastError() == errorValue)
{
  ExitProcess();
}
else
{
  RunMaliciousPayload();
}

Manually Checking Structures

Using the Windows API may be the most obvious method for detecting the presence of a debugger, but manually checking structures is the most common method used by malware authors. There are many reasons why malware authors are discouraged from using the Windows API for anti-debugging. For example, the API calls could be hooked by a rootkit to return false information. Therefore, malware authors often choose to perform the functional equivalent of the API call manually, rather than rely on the Windows API.

In performing manual checks, several flags within the PEB structure provide information about the presence of a debugger. Here, we’ll look at some of the commonly used flags for checking for a debugger.

Checking the BeingDebugged Flag

A Windows PEB structure is maintained by the OS for each running process, as shown in the example in Example 16-2. It contains all user-mode parameters associated with a process. These parameters include the process’s environment data, which itself includes environment variables, the loaded modules list, addresses in memory, and debugger status.

Example 16-2. Documented Process Environment Block (PEB) structure

typedef struct _PEB {
  BYTE Reserved1[2];
  BYTE BeingDebugged;
  BYTE Reserved2[1];
  PVOID Reserved3[2];
  PPEB_LDR_DATA Ldr;
  PRTL_USER_PROCESS_PARAMETERS ProcessParameters;
  BYTE Reserved4[104];
  PVOID Reserved5[52];
  PPS_POST_PROCESS_INIT_ROUTINE PostProcessInitRoutine;
  BYTE Reserved6[128];
  PVOID Reserved7[1];
  ULONG SessionId;
} PEB, *PPEB;

While a process is running, the location of the PEB can be referenced by the location fs:[30h]. For anti-debugging, malware will use that location to check the BeingDebugged flag, which indicates whether the specified process is being debugged. Table 16-1 shows two examples of this type of check.

Table 16-1. Manually Checking the BeingDebugged Flag

`mov` method	`push`/`pop` method
mov eax, dword ptr fs:[30h] mov ebx, byte ptr [eax+2] test ebx, ebx jz NoDebuggerDetected	push dword ptr fs:[30h] pop edx cmp byte ptr [edx+2], 1 je DebuggerDetected

In the code on the left in Table 16-1, the location of the PEB is moved into EAX. Next, this offset plus 2 is moved into EBX, which corresponds to the offset into the PEB of the location of the BeingDebugged flag. Finally, EBX is checked to see if it is zero. If so, a debugger is not attached, and the jump will be taken.

Another example is shown on the right side of Table 16-1. The location of the PEB is moved into EDX using a push/pop combination of instructions, and then the BeingDebugged flag at offset 2 is directly compared to 1.

This check can take many forms, and, ultimately, the conditional jump determines the code path. You can take one of the following approaches to surmount this problem:

Force the jump to be taken (or not) by manually modifying the zero flag immediately before the jump instruction is executed. This is the easiest approach.
Manually change the BeingDebugged flag to zero.

Both options are generally effective against all of the techniques described in this section.

Note

A number of OllyDbg plug-ins change the BeingDebugged flag for you. The most popular are Hide Debugger, Hidedebug, and PhantOm. All are useful for overcoming the BeingDebugged flag check and also help with many of the other techniques we discuss in this chapter.

Checking the ProcessHeap Flag

An undocumented location within the Reserved4 array (shown in Example 16-2), known as ProcessHeap, is set to the location of a process’s first heap allocated by the loader. ProcessHeap is located at 0x18 in the PEB structure. This first heap contains a header with fields used to tell the kernel whether the heap was created within a debugger. These are known as the ForceFlags and Flags fields.

Offset 0x10 in the heap header is the ForceFlags field on Windows XP, but for Windows 7, it is at offset 0x44 for 32-bit applications. Malware may also look at offset 0x0C on Windows XP or offset 0x40 on Windows 7 for the Flags field. This field is almost always equal to the ForceFlags field, but is usually ORed with the value 2.

Example 16-3 shows the assembly code for this technique. (Note that two separate dereferences must occur.)

Example 16-3. Manual ProcessHeap flag check

mov eax, large fs:30h
mov eax, dword ptr [eax+18h]
cmp dword ptr ds:[eax+10h], 0
jne DebuggerDetected

The best way to overcome this technique is to change the ProcessHeap flag manually or to use a hidedebug plug-in for your debugger. If you are using WinDbg, you can start the program with the debug heap disabled. For example, the command windbg –hd notepad.exe will start the heap in normal mode as opposed to debug mode, and the flags we’ve discussed won’t be set.

Checking NTGlobalFlag

Since processes run slightly differently when started with a debugger, they create memory heaps differently. The information that the system uses to determine how to create heap structures is stored at an undocumented location in the PEB at offset 0x68. If the value at this location is 0x70, we know that we are running in a debugger.

The value of 0x70 is a combination of the following flags when a heap is created by a debugger. These flags are set for the process if it is started from within a debugger.

(FLG_HEAP_ENABLE_TAIL_CHECK | FLG_HEAP_ENABLE_FREE_CHECK | FLG_HEAP_VALIDATE_PARAMETERS)

Example 16-4 shows the assembly code for performing this check.

Example 16-4. NTGlobalFlag check

mov eax, large fs:30h
cmp dword ptr ds:[eax+68h], 70h
jz DebuggerDetected

The easiest way to overcome this technique is to change the flags manually or with a hidedebug plug-in for your debugger. If you are using WinDbg, you can start the program with the debug heap option disabled, as mentioned in the previous section.

Checking for System Residue

When analyzing malware, we typically use debugging tools, which leave residue on the system. Malware can search for this residue in order to determine when you are attempting to analyze it, such as by searching registry keys for references to debuggers. The following is a common location for a debugger:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug

This registry key specifies the debugger that activates when an application error occurs. By default, this is set to Dr. Watson, so if it is changed to something like OllyDbg, malware may determine that it is under a microscope.

Malware can also search the system for files and directories, such as common debugger program executables, which are typically present during malware analysis. (Many backdoors already have code in place to traverse filesystems.) Or the malware can detect residue in live memory, by viewing the current process listing or, more commonly, by performing a FindWindow in search of a debugger, as shown in Example 16-5.

Example 16-5. C code for FindWindow detection

if(FindWindow("OLLYDBG", 0) == NULL)
{
//Debugger Not Found
}
else
{
//Debugger Detected
}

In this example, the code simply looks for a window named OLLYDBG.

Previous Chapter

16. Anti-Debugging

Next Chapter

Identifying Debugger Behavior

Table of Contents for Practical Malware Analysis