Exceptions are the principal way that a debugger gains control of a running program. Under the hood, even breakpoints generate exceptions, but nondebugging related events, such as invalid memory accesses and division by zero, will do so as well.
Exceptions are not specific to malware, malware analysis, or debugging. They are often caused by bugs, which is why debuggers usually handle them. But exceptions can also be used to govern the flow of execution in a normal program without involving a debugger. There is functionality in place to ensure that the debugger and the program being debugged can both use exceptions.
Debuggers are usually given two opportunities to handle the same exception: a first-chance exception and a second-chance exception.
When an exception occurs while a debugger is attached, the program being debugged stops executing, and the debugger is given a first chance at control. The debugger can handle the exception or pass it to the program. (When debugging a program, you will need to decide how to handle exceptions, even if they are unrelated to the code you’re interested in.)
If the program has a registered exception handler, that is given a chance to handle the exception after the debugger’s first chance. For example, a calculator program could register an exception handler for the divide-by-zero exception. If the program executes a divide-by-zero operation, the exception handler can inform the user of the error and continue to execute. This is what happens when a program runs without a debugger attached.
If an application does not handle the exception, the debugger is given another chance to handle it—the second-chance exception. When the debugger receives a second-chance exception, it means that program would have crashed if the debugger were not attached. The debugger must resolve the exception to allow the program to run.
When analyzing malware, you are generally not looking for bugs, so first-chance exceptions can often be ignored. (Malware may intentionally generate first-chance exceptions in order to make the program difficult to debug, as you’ll learn in Chapter 15 and Chapter 16.)
Second-chance exceptions cannot be ignored, because the program cannot continue running. If you encounter second-chance exceptions while debugging malware, there may be bugs in the malware that are causing it to crash, but it is more likely that the malware doesn’t like the environment in which it is running.
There are several common exceptions. The most common exception is one that occurs when the
INT 3 instruction is executed. Debuggers have special code to
handle INT 3 exceptions, but OSs treat these as any other
exception.
Programs may include their own instructions for handling INT
3 exceptions, but when a debugger is attached, it will get the first chance. If the
debugger passes the exception to the program, the program’s exception handler should handle
it.
Single-stepping is also implemented as an exception within the OS. A flag in the flags register called the trap flag is used for single-stepping. When the trap flag is set, the processor executes one instruction and then generates an exception.
A memory-access violation exception is generated when code tries to access a location that it cannot access. This exception usually occurs because the memory address is invalid, but it may occur because the memory is not accessible due to access-control protections.
Certain instructions can be executed only when the processor is in privileged mode. When the program attempts to execute them outside privileged mode, the processor generates an exception.