Debugging is the act of locating bugs in software. Generally this is done by developers as bugs are worked out of their software. Debugging can take many forms. Beginning programmers often use output as a rudimentary form of debugging. This output can be printf statements in the case of C.

The most popular story on the use of debugging involves actual bugs. The origin of the use of “bug” to refer to a programming mistake is attributed to Admiral Grace Murray Hopper. A moth got caught in one of the relays from the Harvard University’s Mark II computer. The removal of the moth was coined debugging.

Debuggers are programs themselves that run and monitor the execution of other programs. The debugger can control and alter the execution of the target program. Memory and variables can be monitored and altered as well.

Debuggers are an essential tool in the reverse engineer’s toolbox. The ability to perform runtime analysis speeds up program understanding and reverse engineering. Certain tasks are easier within a debugger. Call chains can be watched instead of guessed.

Tracking indirect calls is much easier during debugging. A call through a register is an example of an indirect call. IDA Pro’s static analysis tracks indirect calls in a very limited fashion. Cross references are not created.

Debugging allows us to watch, observe, and guide our reverse engineering. We do not want to reverse engineer entire programs, but rather the interesting parts.

_try
{
      open(file)
}
_except
{
      printerror
}

:
Exception C0000005 (ACCESS_VIOLATION reading [41414141])

user@redbull:~$ strace ls
execve("/bin/ls", ["ls"], [/* 31 vars */]) = 0
brk(0) = 0x805c000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0xb7eec000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3

IDA Pro comes with a built-in debugger, which was introduced in version 4.50. The debugger is implemented as a plug-in. This is a true testament to the extensibility of IDA Pro.

IDA isn’t limited to debugging the local system. The debugger can operate locally as well as remotely over the network. The debugging clients allow IDA to debug other machines running different operating systems and even CPUs. Authentication is available, but best practices suggest debugging only over the local network.

IDA Pro supports the following debugging environments:

The debugger menu option is made available if the binary being analyzed matches one of the previous targets listed. Debugger settings are available from the debugger menu option as shown in Figure 5.1.

Figure 5.1. Debugger Setup Options

Some of the notable options are:

In order to demonstrate debugging within IDA Pro we will use Netcat as an example. Netcat is a network tool, whose name comes from the combination of network and the UNIX command cat. You can pipe data to and from other programs over a network, which is why it is known as the “TCP/IP Swiss army knife.”

In December 2004, vulnerability was reported in Netcat for Windows 1.1 (www.vulnwatch.org/netcat/netcat-111.txt). We will be analyzing the vulnerable version 1.1 (http://packetstormsecurity.org/UNIX/netcat/nc11nt.zip), while the patched version is 1.11 (www.vulnwatch.org/netcat/).

The vendor advisory describes a remote buffer overflow when using the “-e” option. The vulnerability resides in the SessionWriteShellThreadFn function within the dosexec.c source file. Select parts of the vulnerable function are shown in the following code snippet:

static VOID
SessionWriteShellThreadFn(LPVOID Parameter)
{
      PSESSION_DATA Session = Parameter;
      BYTE RecvBuffer[1];
      BYTE Buffer[BUFFER_SIZE];
      BYTE EchoBuffer[5];
      DWORD BytesWritten;
      DWORD BufferCnt, EchoCnt;
      DWORD TossCnt = 0;
      BOOL PrevWasFF = FALSE;
      BufferCnt = 0;

      //Loop, reading one byte at a time from the socket.
      while (recv(Session->ClientSocket, RecvBuffer, sizeof(RecvBuffer), 0) != 0)
           {
              EchoCnt = 0;

              Buffer[BufferCnt++] = EchoBuffer[EchoCnt++] = RecvBuffer[0];
              if (RecvBuffer[0] == '\r')
                   Buffer[BufferCnt++] = EchoBuffer[EchoCnt++] = '\n';

              //Trap exit as it causes problems
              if (strnicmp(Buffer, "exit\r\n", 6) == 0)
                   ExitThread(0);
              //
              //If we got a CR, it's time to send what we've buffered up down to the
              //shell process.
              if (RecvBuffer[0] == '\n' || RecvBuffer[0] == '\r') {
                 if (! WriteFile(Session->WritePipeHandle, Buffer, BufferCnt,
                 &BytesWritten, NULL))
                 {
                  break;
                 }
                 BufferCnt = 0;
              }
      }
      ExitThread(0);
}

The function contains a receive loop with two possible exits. The first exit occurs if the buffer contains an exit\r\n string. The second exit requires either a \n or \r as the received byte and a failure for the WriteFile call.

After opening nc.exe in IDA Pro, the debugger tab is available. If an executable format is supported by one of the debuggers, the debugger tab will be visible.

Process options need to be configured. Specifically, we need to configure the command line arguments. The vulnerability is only present when the -e option is used. In order to use this option we must also supply the -l (listening) options as well as the -p (port number option). The -e option executes a program passing any input it receives over the network. We don’t need the executed program to do anything, so we can use the more program. Figure 5.2 shows typical process options using our command line arguments.

Figure 5.2. Debugger Application Setup

Debugger hotkeys are:

We find the SessionWriteShellThreadFn function by looking at the imports for WriteFile. From the cross references we determine that the function address is .text:00401520. We then set a breakpoint at the beginning of the function. Figure 5.3 shows the graph view of the function. The function has been renamed to SessionWriteShellThreadFn and stack variable buf has been renamed to RecvBuffer for readability.

Figure 5.3. SessionWriteShellThreadFn Function

Push F9 and the debugger will start. The different windows will rearrange themselves. The debugger is running nc.exe using the passed arguments. Since there hasn’t been a connection yet, our breakpoint hasn’t been hit.

Start a cmd.exe shell and we will use another instance of Netcat to connect to the debugged one. This Netcat will be called the Netcat client in order to differentiate between the debugged Netcat. Use the command line:

nc localhost 2323

At this point we will hit our breakpoint. The register window will contain values similar to Figure 5.4. A stack window will be displayed similar to Figure 5.5.

Figure 5.4. Registers

Figure 5.5. Stack

The register window shows the register values on the left and the right side contains any interpretation of the values. Registers can be changed by either right-clicking in the value box or typing directly into the value box. New views are available by right-clicking on registers and most addresses. The views can be assembly or hex.

Stepping with the F8 key (step over) will avoid going into the actual recv call. The recv call will block until it receives data. Type hello\n into the Netcat client. The recv call will return now that it has received data. We can continue single stepping.

We can set another breakpoint on the instruction cmp al, 0x0d in the basic block from Figure 5.6. Pressing F9 (continue) runs the program until another breakpoint is hit. Execution will stop at the breakpoint we just set. Notice the value of esi has incremented after each character. The character can still be seen in register al. Right-click the breakpoint and select disable breakpoint.

After the entire command from the Netcat client has been copied into the buffer, WriteFile is called. Figure 5.7 shows the basic block containing the call. Set a breakpoint on test eax, eax, which is the instruction following the WriteFile call. When we continue (F9), the debugger will stop on this instruction.

Figure 5.7. Basic Block Containing the Call

We know the size of the buffer; it is 0xc8 bytes according to Figure 5.7. However, the stack looks different than expected since this function is called by CreateThread().

We can set a conditional breakpoint on the instruction cmp al, 0x0d from the basic block shown in Figure 5.8. In order to set a conditional breakpoint, right-click on the disabled breakpoint and select Edit breakpoint. Enter esi == 0xc8 ‖ esi == 0xcc in the condition box as in Figure 5.8. The breakpoint will hit at the last location of the buffer and then upon overwriting the next DWORD.

Figure 5.8. Conditional Breakpoint

We need to send more data. From the stack view, it appears that 272 bytes will write past the end of the page. Other machines or operating systems may have different memory layout. The easiest way to send the data is to build a string in a text editor and then paste it into the Netcat client.

After the breakpoint hits, the stack looks like Figure 5.9. There is still data to be read and there is very little space left on the page. The next breakpoint is the first of the stack corruption. When the program is allowed to continue, we see the warning and then the exception will come up as shown in Figure 5.10.

Figure 5.9. Stack

Figure 5.10. Exceptions

The exception is configured to stop the program. We can go into Change exception definition and select Pass to application. See Figure 5.11.

Figure 5.11. Handling Options

The program will terminate with this exception. IDA Pro has recorded the exception in the log window:

nc.exe: The instruction at 0x401555 referenced memory at 0xF50000. The memory could not be written (0x00401555 -> 00F50000)

Debugger: Thread terminated: id=00001660 (exit code = 0xC0000005).
Debugger: Thread terminated: id=000017AC (exit code = 0xC0000005).
Debugger: Process terminated (exit code = C0000005h).

Detecting memory corruption is important to the reverse engineer. Stack and heap overflows are attacks that overwrite and corrupt memory.

The debugger can be used to detect memory corruption. Some ways of detecting corruption can be done manually while others are more applicable to being scripts or plug-ins.

Microsoft began adding stack cookies to their compiler beginning with Visual Studio 2003, using the GS command line switch. At the entry of a function, a stack cookie is placed on the stack. The cookie is calculated by taking a global security cookie, __security_cookie, and XORing it with the esp register. During an exit of the function, the stack cookie is XORed with the esp register. The result of the operation should be __security_cookie. This value is passed to the __security_check_cookie( ) function. If the passed value matches __security_cookie, then __security_check_cookie( ) returns allowing the original function to continue as designed. A more detailed explanation is available here: http://uninformed.org/index.cgi?v=7&a=2&p=1.

The idea of the protection is that the cookie check will fail if the stack has been corrupted. If the check fails, the process will be terminated with an exit code of 0xc0000409. In order to catch the stack corruption, we can set a breakpoint in __security_check_cookie( ), as shown in Figure 5.12. Alternately, the breakpoint can be set directly on the __report_gsfailure( ) function. The __security_check_cookie( ) function is compiled in statically and the address will change depending on the binary.

Figure 5.12. Setting a Breakpoint

Checking for heap corruption is more dependent on the operating system being used. Windows XP SP2, Windows 2003, and Vista have various methods of heap protection built in. However, unlike the GS stack protection, the heap protections are part of the operating system. Various techniques have been developed to pass heap protections, but data from a fuzzer will most likely be caught by these protections.

There are multiple checks and simple breakpoints may not be sufficient. Debugger based scripting or a plug-in would be ideal. The heap functions can be hooked to provide allocation data. The protection functions can be hooked to report corruption. For systems without such thorough protection functions, hooked functions containing checks could be added. Rather than stopping attacks, the checks notify us of corruption as soon as possible.

set _NO_DEBUG_HEAP=1

The debuggers within IDA Pro are very useful. You have full access to the static analysis, renamed functions, and other parts of code that have been reverse engineered. Like any type of tool, people have preferences for different tools. There are different debuggers available to the reverse engineer.

Each debugger has advantages and disadvantages. It usually comes down to a matter of personal preference. The following paragraphs provide a brief overview of some other debuggers.

IDA Pro’s debugger is very powerful and allows for much greater program understanding than static analysis alone. The debugger can operate locally as well as remotely with the most common operating systems.

A benefit to using the IDA Pro debugger over other debuggers is the availability of any reverse engineering work we have done. This includes renaming functions, tables, and local variables.

There are times when IDA Pro’s debugger is not the best solution. Various other debuggers are available.