The first technique modifies the Microsoft PE header of a binary executable, causing OllyDbg to crash when loading the executable. The result is an error of “Bad or Unknown 32-bit Executable File,” yet the program usually runs fine outside the debugger.

This issue is due to the fact that OllyDbg follows the Microsoft specifications regarding the PE header too strictly. In the PE header, there is typically a structure known as the IMAGE_OPTIONAL_HEADER. Figure 16-5 shows a subset of this structure.

Figure 16-5. PE IMAGE_OPTIONAL_HEADER and NumberOfRvaAndSizes vulnerability

The last several elements in this structure are of particular interest. The NumberOfRvaAndSizes field identifies the number of entries in the DataDirectory array that follows. The DataDirectory array indicates where to find other important executable components in the file; it is little more than an array of IMAGE_DATA_DIRECTORY structures at the end of the optional header structure. Each data directory structure specifies the size and relative virtual address of the directory.

The size of the array is set to IMAGE_NUMBEROF_DIRECTORY_ENTRIES, which is equal to 0x10. The Windows loader ignores any NumberOfRvaAndSizes greater than 0x10, because anything larger will not fit in the DataDirectory array. OllyDbg follows the standard and uses NumberOfRvaAndSizes no matter what. As a consequence, setting the size of the array to a value greater than 0x10 (like 0x99) will cause OllyDbg to generate a pop-up window to the user before exiting the program.

The easiest way to overcome this technique is to manually modify the PE header and set the NumberOfRvaAndSizes to 0x10 using a hex editor or PE Explorer. Or, of course, you can use a debugger that is not vulnerable to this technique, such as WinDbg or OllyDbg 2.0.

Another PE header trick involves section headers, causing OllyDbg to crash during loading with the error “File contains too much data.” (WinDbg and OllyDbg 2.0 are not vulnerable to this technique.) Sections contain the content of the file, including code, data, resources, and other information. Each section has a header in the form of an IMAGE_SECTION_HEADER structure. Figure 16-6 shows a subset of this structure.

Figure 16-6. PE IMAGE_SECTION_HEADER structure

The elements of interest are VirtualSize and the SizeOfRawData. According to the Windows PE specification, VirtualSize should contain the total size of the section when loaded into memory, and SizeOfRawData should contain the size of data on disk. The Windows loader uses the smaller of VirtualSize and SizeOfRawData to map the section data into memory. If the SizeOfRawData is larger than VirtualSize, only VirtualSize data is copied into memory; the rest is ignored. Because OllyDbg uses only the SizeOfRawData, setting the SizeofRawData to something large like 0x77777777, will cause OllyDbg to crash.

The easiest way to overcome this anti-debugging technique is to manually modify the PE header and set the SizeOfRawData using a hex editor to change the value to be close to VirtualSize. (Note that, according to the specification, this value must be a multiple of the FileAlignment value from the IMAGE_OPTIONAL_HEADER). PE Explorer is a great program to use for this purpose because it is not fooled by a large value for SizeofRawData.

Malware often attempts to exploit a format string vulnerability in version 1.1 of OllyDbg, by providing a string of %s as a parameter to OutputDebugString to cause OllyDbg to crash. Beware of suspicious calls like OutputDebugString ("%s%s%s%s%s%s%s%s%s%s%s%s%s%s"). If this call executes, your debugger will crash.