Basic static analysis consists of examining the executable file without viewing the actual instructions. Basic static analysis can confirm whether a file is malicious, provide information about its functionality, and sometimes provide information that will allow you to produce simple network signatures. Basic static analysis is straightforward and can be quick, but it’s largely ineffective against sophisticated malware, and it can miss important behaviors.

Basic dynamic analysis techniques involve running the malware and observing its behavior on the system in order to remove the infection, produce effective signatures, or both. However, before you can run malware safely, you must set up an environment that will allow you to study the running malware without risk of damage to your system or network. Like basic static analysis techniques, basic dynamic analysis techniques can be used by most people without deep programming knowledge, but they won’t be effective with all malware and can miss important functionality.

Advanced static analysis consists of reverse-engineering the malware’s internals by loading the executable into a disassembler and looking at the program instructions in order to discover what the program does. The instructions are executed by the CPU, so advanced static analysis tells you exactly what the program does. However, advanced static analysis has a steeper learning curve than basic static analysis and requires specialized knowledge of disassembly, code constructs, and Windows operating system concepts, all of which you’ll learn in this book.

Advanced dynamic analysis uses a debugger to examine the internal state of a running malicious executable. Advanced dynamic analysis techniques provide another way to extract detailed information from an executable. These techniques are most useful when you’re trying to obtain information that is difficult to gather with the other techniques. In this book, we’ll show you how to use advanced dynamic analysis together with advanced static analysis in order to completely analyze suspected malware.

When the packed program is run, a small wrapper program also runs to decompress the packed file and then run the unpacked file, as shown in Figure 1-4. When a packed program is analyzed statically, only the small wrapper program can be dissected. (Chapter 18 discusses packing and unpacking in more detail.)

Figure 1-4. The file on the left is the original executable, with all strings, imports, and other information visible. On the right is a packed executable. All of the packed file’s strings, imports, and other information are compressed and invisible to most static analysis tools.

One way to detect packed files is with the PEiD program. You can use PEiD to detect the type of packer or compiler employed to build an application, which makes analyzing the packed file much easier. Figure 1-5 shows information about the orig_af2.ex_ file as reported by PEiD.

Figure 1-5. The PEiD program

As you can see, PEiD has identified the file as being packed with UPX version 0.89.6-1.02 or 1.05-2.90. (Just ignore the other information shown here for now. We’ll examine this program in more detail in Chapter 18.)

When a program is packed, you must unpack it in order to be able to perform any analysis. The unpacking process is often complex and is covered in detail in Chapter 18, but the UPX packing program is so popular and easy to use for unpacking that it deserves special mention here. For example, to unpack malware packed with UPX, you would simply download UPX (http://upx.sourceforge.net/) and run it like so, using the packed program as input:

upx -d PackedProgram.exe

Static linking is the least commonly used method of linking libraries, although it is common in UNIX and Linux programs. When a library is statically linked to an executable, all code from that library is copied into the executable, which makes the executable grow in size. When analyzing code, it’s difficult to differentiate between statically linked code and the executable’s own code, because nothing in the PE file header indicates that the file contains linked code.

While unpopular in friendly programs, runtime linking is commonly used in malware, especially when it’s packed or obfuscated. Executables that use runtime linking connect to libraries only when that function is needed, not at program start, as with dynamically linked programs.

Several Microsoft Windows functions allow programmers to import linked functions not listed in a program’s file header. Of these, the two most commonly used are LoadLibrary and GetProcAddress. LdrGetProcAddress and LdrLoadDll are also used. LoadLibrary and GetProcAddress allow a program to access any function in any library on the system, which means that when these functions are used, you can’t tell statically which functions are being linked to by the suspect program.

Of all linking methods, dynamic linking is the most common and the most interesting for malware analysts. When libraries are dynamically linked, the host OS searches for the necessary libraries when the program is loaded. When the program calls the linked library function, that function executes within the library.

The PE file header stores information about every library that will be loaded and every function that will be used by the program. The libraries used and functions called are often the most important parts of a program, and identifying them is particularly important, because it allows us to guess at what the program does. For example, if a program imports the function URLDownloadToFile, you might guess that it connects to the Internet to download some content that it then stores in a local file.

The Dependency Walker program (http://www.dependencywalker.com/), distributed with some versions of Microsoft Visual Studio and other Microsoft development packages, lists only dynamically linked functions in an executable.

Figure 1-6 shows the Dependency Walker’s analysis of SERVICES.EX_ ❶. The far left pane at ❷ shows the program as well as the DLLs being imported, namely KERNEL32.DLL and WS2_32.DLL.

Figure 1-6. The Dependency Walker program

Clicking KERNEL32.DLL shows its imported functions in the upper-right pane at ❸. We see several functions, but the most interesting is CreateProcessA, which tells us that the program will probably create another process, and suggests that when running the program, we should watch for the launch of additional programs.

The middle right pane at ❹ lists all functions in KERNEL32.DLL that can be imported—information that is not particularly useful to us. Notice the column in panes ❸ and ❹ labeled Ordinal. Executables can import functions by ordinal instead of name. When importing a function by ordinal, the name of the function never appears in the original executable, and it can be harder for an analyst to figure out which function is being used. When malware imports a function by ordinal, you can find out which function is being imported by looking up the ordinal value in the pane at ❹.

The bottom two panes (❺ and ❻) list additional information about the versions of DLLs that would be loaded if you ran the program and any reported errors, respectively.

A program’s DLLs can tell you a lot about its functionality. For example, Table 1-1 lists common DLLs and what they tell you about an application.

The PE file header also includes information about specific functions used by an executable. The names of these Windows functions can give you a good idea about what the executable does. Microsoft does an excellent job of documenting the Windows API through the Microsoft Developer Network (MSDN) library. (You’ll also find a list of functions commonly used by malware in Appendix A.)

Like imports, DLLs and EXEs export functions to interact with other programs and code. Typically, a DLL implements one or more functions and exports them for use by an executable that can then import and use them.

The PE file contains information about which functions a file exports. Because DLLs are specifically implemented to provide functionality used by EXEs, exported functions are most common in DLLs. EXEs are not designed to provide functionality for other EXEs, and exported functions are rare. If you discover exports in an executable, they often will provide useful information.

In many cases, software authors name their exported functions in a way that provides useful information. One common convention is to use the name used in the Microsoft documentation. For example, in order to run a program as a service, you must first define a ServiceMain function. The presence of an exported function called ServiceMain tells you that the malware runs as part of a service.

Unfortunately, while the Microsoft documentation calls this function ServiceMain, and it’s common for programmers to do the same, the function can have any name. Therefore, the names of exported functions are actually of limited use against sophisticated malware. If malware uses exports, it will often either omit names entirely or use unclear or misleading names.

You can view export information using the Dependency Walker program discussed in Exploring Dynamically Linked Functions with Dependency Walker. For a list of exported functions, click the name of the file you want to examine. Referring back to Figure 1-6, window ❹ shows all of a file’s exported functions.

Table 1-2 shows an abridged list of functions imported by PotentialKeylogger.exe, as collected using Dependency Walker. Because we see so many imports, we can immediately conclude that this file is not packed.

Like most average-sized programs, this executable contains a large number of imported functions. Unfortunately, only a small minority of those functions are particularly interesting for malware analysis. Throughout this book, we will cover the imports for malicious software, focusing on the most interesting functions from a malware analysis standpoint.

When you are not sure what a function does, you will need to look it up. To help guide your analysis, Appendix A lists many of the functions of greatest interest to malware analysts. If a function is not listed in Appendix A, search for it on MSDN online.

As a new analyst, you will spend time looking up many functions that aren’t very interesting, but you’ll quickly start to learn which functions could be important and which ones are not. For the purposes of this example, we will show you a large number of imports that are uninteresting, so you can become familiar with looking at a lot of data and focusing on some key nuggets of information.

Normally, we wouldn’t know that this malware is a potential keylogger, and we would need to look for functions that provide the clues. We will be focusing on only the functions that provide hints to the functionality of the program.

The imports from Kernel32.dll in Table 1-2 tell us that this software can open and manipulate processes (such as OpenProcess, GetCurrentProcess, and GetProcessHeap) and files (such as ReadFile, CreateFile, and WriteFile). The functions FindFirstFile and FindNextFile are particularly interesting ones that we can use to search through directories.

The imports from User32.dll are even more interesting. The large number of GUI manipulation functions (such as RegisterClassEx, SetWindowText, and ShowWindow) indicates a high likelihood that this program has a GUI (though the GUI is not necessarily displayed to the user).

The function SetWindowsHookEx is commonly used in spyware and is the most popular way that keyloggers receive keyboard inputs. This function has some legitimate uses, but if you suspect malware and you see this function, you are probably looking at keylogging functionality.

The function RegisterHotKey is also interesting. It registers a hotkey (such as CTRL-SHIFT-P) so that whenever the user presses that hotkey combination, the application is notified. No matter which application is currently active, a hotkey will bring the user to this application.

The imports from GDI32.dll are graphics-related and simply confirm that the program probably has a GUI. The imports from Shell32.dll tell us that this program can launch other programs—a feature common to both malware and legitimate programs.

The imports from Advapi32.dll tell us that this program uses the registry, which in turn tells us that we should search for strings that look like registry keys. Registry strings look a lot like directories. In this case, we found the string Software\Microsoft\Windows\CurrentVersion\Run, which is a registry key (commonly used by malware) that controls which programs are automatically run when Windows starts up.

This executable also has several exports: LowLevelKeyboardProc and Low-LevelMouseProc. Microsoft’s documentation says, “The LowLevelKeyboardProc hook procedure is an application-defined or library-defined callback function used with the SetWindowsHookEx function.” In other words, this function is used with SetWindowsHookEx to specify which function will be called when a specified event occurs—in this case, the low-level keyboard event. The documentation for SetWindowsHookEx further explains that this function will be called when certain low-level keyboard events occur.

The Microsoft documentation uses the name LowLevelKeyboardProc, and the programmer in this case did as well. We were able to get valuable information because the programmer didn’t obscure the name of an export.

Using the information gleaned from a static analysis of these imports and exports, we can draw some significant conclusions or formulate some hypotheses about this malware. For one, it seems likely that this is a local keylogger that uses SetWindowsHookEx to record keystrokes. We can also surmise that it has a GUI that is displayed only to a specific user, and that the hotkey registered with RegisterHotKey specifies the hotkey that the malicious user enters to see the keylogger GUI and access recorded keystrokes. We can further speculate from the registry function and the existence of Software\Microsoft\Windows\CurrentVersion\Run that this program sets itself to load at system startup.

Table 1-3 shows a complete list of the functions imported by a second piece of unknown malware. The brevity of this list tells us that this program is packed or obfuscated, which is further confirmed by the fact that this program has no readable strings. A Windows compiler would not create a program that imports such a small number of functions; even a Hello, World program would have more.

The fact that this program is packed is a valuable piece of information, but its packed nature also prevents us from learning anything more about the program using basic static analysis. We’ll need to try more advanced analysis techniques such as dynamic analysis (covered in Chapter 3) or unpacking (covered in Chapter 18).

The PE file format stores interesting information within its header. We can use the PEview tool to browse through the information, as shown in Figure 1-7.

In the figure, the left pane at ❶ displays the main parts of a PE header. The IMAGE_FILE_HEADER entry is highlighted because it is currently selected.

The first two parts of the PE header—the IMAGE_DOS_HEADER and MS-DOS Stub Program—are historical and offer no information of particular interest to us.

The next section of the PE header, IMAGE_NT_HEADERS, shows the NT headers. The signature is always the same and can be ignored.

The IMAGE_FILE_HEADER entry, highlighted and displayed in the right panel at ❷, contains basic information about the file. The Time Date Stamp description at ❸ tells us when this executable was compiled, which can be very useful in malware analysis and incident response. For example, an old compile time suggests that this is an older attack, and antivirus programs might contain signatures for the malware. A new compile time suggests the reverse.

Figure 1-7. Viewing the IMAGE_FILE_HEADER in the PEview program

That said, the compile time is a bit problematic. All Delphi programs use a compile time of June 19, 1992. If you see that compile time, you’re probably looking at a Delphi program, and you won’t really know when it was compiled. In addition, a competent malware writer can easily fake the compile time. If you see a compile time that makes no sense, it probably was faked.

The IMAGE_OPTIONAL_HEADER section includes several important pieces of information. The Subsystem description indicates whether this is a console or GUI program. Console programs have the value IMAGE_SUBSYSTEM_WINDOWS_CUI and run inside a command window. GUI programs have the value IMAGE_SUBSYSTEM_WINDOWS_GUI and run within the Windows system. Less common subsystems such as Native or Xbox also are used.

The most interesting information comes from the section headers, which are in IMAGE_SECTION_HEADER, as shown in Figure 1-8. These headers are used to describe each section of a PE file. The compiler generally creates and names the sections of an executable, and the user has little control over these names. As a result, the sections are usually consistent from executable to executable (see Table 1-4), and any deviations may be suspicious.

For example, in Figure 1-8, Virtual Size at ❶ tells us how much space is allocated for a section during the loading process. The Size of Raw Data at ❷ shows how big the section is on disk. These two values should usually be equal, because data should take up just as much space on the disk as it does in memory. Small differences are normal, and are due to differences between alignment in memory and on disk.

The section sizes can be useful in detecting packed executables. For example, if the Virtual Size is much larger than the Size of Raw Data, you know that the section takes up more space in memory than it does on disk. This is often indicative of packed code, particularly if the .text section is larger in memory than on disk.

Figure 1-8. Viewing the IMAGE_SECTION_HEADER .text section in the PEview program

Table 1-5 shows the sections from PotentialKeylogger.exe. As you can see, the .text, .rdata, and .rsrc sections each has a Virtual Size and Size of Raw Data value of about the same size. The .data section may seem suspicious because it has a much larger virtual size than raw data size, but this is normal for the .data section in Windows programs. But note that this information alone does not tell us that the program is not malicious; it simply shows that it is likely not packed and that the PE file header was generated by a compiler.

Table 1-6 shows the sections from PackedProgram.exe. The sections in this file have a number of anomalies: The sections named Dijfpds, .sdfuok, and Kijijl are unusual, and the .text, .data, and .rdata sections are suspicious. The .text section has a Size of Raw Data value of 0, meaning that it takes up no space on disk, and its Virtual Size value is A000, which means that space will be allocated for the .text segment. This tells us that a packer will unpack the executable code to the allocated .text section.

Now that we’re finished looking at the header for the PE file, we can look at some of the sections. The only section we can examine without additional knowledge from later chapters is the resource section. You can use the free Resource Hacker tool found at http://www.angusj.com/ to browse the .rsrc section. When you click through the items in Resource Hacker, you’ll see the strings, icons, and menus. The menus displayed are identical to what the program uses. Figure 1-9 shows the Resource Hacker display for the Windows Calculator program, calc.exe.

Figure 1-9. The Resource Hacker tool display for calc.exe

The panel on the left shows all resources included in this executable. Each root folder shown in the left pane at ❶ stores a different type of resource. The informative sections for malware analysis include:

The .rsrc section shown in Figure 1-9 is typical of Windows applications and can include whatever a programmer requires.

Many other tools are available for browsing a PE header. Two of the most useful tools are PEBrowse Professional and PE Explorer.

PEBrowse Professional (http://www.smidgeonsoft.prohosting.com/pebrowse-profile-viewer.html) is similar to PEview. It allows you to look at the bytes from each section and shows the parsed data. PEBrowse Professional does the better job of presenting information from the resource (.rsrc) section.

PE Explorer (http://www.heaventools.com/) has a rich GUI that allows you to navigate through the various parts of the PE file. You can edit certain parts of the PE file, and its included resource editor is great for browsing and editing the file’s resources. The tool’s main drawback is that it is not free.

The PE header contains useful information for the malware analyst, and we will continue to examine it in subsequent chapters. Table 1-7 reviews the key information that can be obtained from a PE header.

This lab uses the files Lab01-01.exe and Lab01-01.dll. Use the tools and techniques described in the chapter to gain information about the files and answer the questions below.

Analyze the file Lab01-02.exe.

Analyze the file Lab01-03.exe.

Analyze the file Lab01-04.exe.

Most malware includes network functionality. For example, a worm will perform network attacks against other machines in an effort to spread itself. But you would not want to allow a worm access to your own network, because it could to spread to other computers.

When analyzing malware, you will probably want to observe the malware’s network activity to help you understand the author’s intention, to create signatures, or to exercise the program fully. VMware offers several networking options for virtual networking, as shown in Figure 2-2 and discussed in the following sections.

Figure 2-2. Virtual network configuration options for a network adapter

Setting Up Host-Only Networking

Host-only networking, a feature that creates a separate private LAN between the host OS and the guest OS, is commonly used for malware analysis. A host-only LAN is not connected to the Internet, which means that the malware is contained within your virtual machine but allowed some network connectivity.

Note

When configuring your host computer, ensure that it is fully patched, as protection in case the malware you’re testing tries to spread. It’s a good idea to configure a restrictive firewall to the host from the virtual machine to help prevent the malware from spreading to your host. The Microsoft firewall that comes with Windows XP Service Pack 2 and later is well documented and provides sufficient protection. Even if patches are up to date, however, the malware could spread by using a zero-day exploit against the host OS.

Figure 2-3 illustrates the network configuration for host-only networking. When host-only networking is enabled, VMware creates a virtual network adapter in the host and virtual machines, and connects the two without touching the host’s physical network adapter. The host’s physical network adapter is still connected to the Internet or other external network.

Figure 2-3. Host-only networking in VMware

Using Multiple Virtual Machines

One last configuration combines the best of all options. It requires multiple virtual machines linked by a LAN but disconnected from the Internet and host machine, so that the malware is connected to a network, but the network isn’t connected to anything important.

Figure 2-4 shows a custom configuration with two virtual machines connected to each other. In this configuration, one virtual machine is set up to analyze malware, and the second machine provides services. The two virtual machines are connected to the same VMNet virtual switch. In this case, the host machine is still connected to the external network, but not to the machine running the malware.

Figure 2-4. Custom networking in VMware

When using more than one virtual machine for analysis, you’ll find it useful to combine the machines as a virtual machine team. When your machines are joined as part of a virtual machine team, you will be able to manage their power and network settings together. To create a new virtual machine team, choose File ▶ New ▶ Team.

Sometimes you’ll want to connect your malware-running machine to the Internet to provide a more realistic analysis environment, despite the obvious risks. The biggest risk, of course, is that your computer will perform malicious activity, such as spreading malware to additional hosts, becoming a node in a distributed denial-of-service attack, or simply spamming. Another risk is that the malware writer could notice that you are connecting to the malware server and trying to analyze the malware.

You should never connect malware to the Internet without first performing some analysis to determine what the malware might do when connected. Then connect only if you are comfortable with the risks.

The most common way to connect a virtual machine to the Internet using VMware is with a bridged network adapter, which allows the virtual machine to be connected to the same network interface as the physical machine. Another way to connect malware running on a virtual machine to the Internet is to use VMware’s Network Address Translation (NAT) mode.

NAT mode shares the host’s IP connection to the Internet. The host acts like a router and translates all requests from the virtual machine so that they come from the host’s IP address. This mode is useful when the host is connected to the network, but the network configuration makes it difficult, if not impossible, to connect the virtual machine’s adapter to the same network.

For example, if the host is using a wireless adapter, NAT mode can be easily used to connect the virtual machine to the network, even if the wireless network has Wi-Fi Protected Access (WPA) or Wired Equivalent Privacy (WEP) enabled. Or, if the host adapter is connected to a network that allows only certain network adapters to connect, NAT mode allows the virtual machine to connect through the host, thereby avoiding the network’s access control settings.

Peripheral devices, such as CD-ROMs and external USB storage drives, pose a particular problem for virtual machines. Most devices can be connected either to the physical machine or the virtual machine, but not both.

The VMware interface allows you to connect and disconnect external devices to virtual machines. If you connect a USB device to a machine while the virtual machine window is active, VMware will connect the USB device to the guest and not the host, which may be undesirable, considering the growing popularity of worms that spread via USB storage devices. To modify this setting, choose VM ▶ Settings ▶ USB Controller and uncheck the Automatically connect new USB devices checkbox to prevent USB devices from being connected to the virtual machine.

Taking snapshots is a concept unique to virtual machines. VMware’s virtual machine snapshots allow you save a computer’s current state and return to that point later, similar to a Windows restore point.

The timeline in Figure 2-5 illustrates how taking snapshots works. At 8:00 you take a snapshot of the computer. Shortly after that, you run the malware sample. At 10:00, you revert to the snapshot. The OS, software, and other components of the machine return to the same state they were in at 8:00, and everything that occurred between 8:00 and 10:00 is erased as though it never happened. As you can see, taking snapshots is an extremely powerful tool. It’s like a built-in undo feature that saves you the hassle of needing to reinstall your OS.

Figure 2-5. Snapshot timeline

After you’ve installed your OS and malware analysis tools, and you have configured the network, take a snapshot. Use that snapshot as your base, clean-slate snapshot. Next, run your malware, complete your analysis, and then save your data and revert to the base snapshot, so that you can do it all over again.

But what if you’re in the middle of analyzing malware and you want to do something different with your virtual machine without erasing all of your progress? VMware’s Snapshot Manager allows you to return to any snapshot at any time, no matter which additional snapshots have been taken since then or what has happened to the machine. In addition, you can branch your snapshots so that they follow different paths. Take a look at the following example workflow:

When you return to your virtual machine, you can access either snapshot at any time, as shown in Figure 2-6. The two machine states are completely independent, and you can save as many snapshots as you have disk space.

Figure 2-6. VMware Snapshot Manager

One drawback of using snapshots is that any work undertaken on the virtual machine is lost when you revert to an earlier snapshot. You can, however, save your work before loading the earlier snapshot by transferring any files that you want to keep to the host OS using VMware’s drag-and-drop feature. As long as VMware Tools is installed in the guest OS and both systems are running Windows, you should be able to drag and drop a file directly from the guest OS to the host OS. This is the simplest and easiest way to transfer files.

Another way to transfer your data is with VMware’s shared folders. A shared folder is accessible from both the host and the guest OS, similar to a shared Windows folder.

Many malware sandboxes—such as Norman SandBox, GFI Sandbox, Anubis, Joe Sandbox, ThreatExpert, BitBlaze, and Comodo Instant Malware Analysis—will analyze malware for free. Currently, Norman SandBox and GFI Sandbox (formerly CWSandbox) are the most popular among computer-security professionals.

These sandboxes provide easy-to-understand output and are great for initial triage, as long as you are willing to submit your malware to the sandbox websites. Even though the sandboxes are automated, you might choose not to submit malware that contains company information to a public website.

Most sandboxes work similarly, so we’ll focus on one example, GFI Sandbox. Figure 3-1 shows the table of contents for a PDF report generated by running a file through GFI Sandbox’s automated analysis. The malware report includes a variety of details on the malware, such as the network activity it performs, the files it creates, the results of scanning with VirusTotal, and so on.

Figure 3-1. GFI Sandbox sample results for win32XYZ.exe

Reports generated by GFI Sandbox vary in the number of sections they contain, based on what the analysis finds. The GFI Sandbox report has six sections in Figure 3-1, as follows:

Malware sandboxes do have a few major drawbacks. For example, the sandbox simply runs the executable, without command-line options. If the malware executable requires command-line options, it will not execute any code that runs only when an option is provided. In addition, if your subject malware is waiting for a command-and-control packet to be returned before launching a backdoor, the backdoor will not be launched in the sandbox.

The sandbox also may not record all events, because neither you nor the sandbox may wait long enough. For example, if the malware is set to sleep for a day before it performs malicious activity, you may miss that event. (Most sandboxes hook the Sleep function and set it to sleep only briefly, but there is more than one way to sleep, and the sandboxes cannot account for all of these.)

Other potential drawbacks include the following:

Procmon displays configurable columns containing information about individual events, including the event’s sequence number, timestamp, name of the process causing the event, event operation, path used by the event, and result of the event. This detailed information can be too long to fit on the screen, or it can be otherwise difficult to read. If you find either to be the case, you can view the full details of a particular event by double-clicking its row.

Figure 3-2 shows a collection of procmon events that occurred on a machine running a piece of malware named mm32.exe. Reading the Operation column will quickly tell you which operations mm32.exe performed on this system, including registry and file system accesses. One entry of note is the creation of a file C:\Documents and Settings\All Users\Application Data\mw2mmgr.txt at sequence number 212 using CreateFile. The word SUCCESS in the Result column tells you that this operation was successful.

Figure 3-2. Procmon mm32.exe example

It’s not always easy to find information in procmon when you are looking through thousands of events, one by one. That’s where procmon’s filtering capability is key.

You can set procmon to filter on one executable running on the system. This feature is particularly useful for malware analysis, because you can set a filter on the piece of malware you are running. You can also filter on individual system calls such as RegSetValue, CreateFile, WriteFile, or other suspicious or destructive calls.

When procmon filtering is turned on, it filters through recorded events only. All recorded events are still available even though the filter shows only a limited display. Setting a filter is not a way to prevent procmon from consuming too much memory.

To set a filter, choose Filter ▶ Filter to open the Filter menu, as shown in the top image of Figure 3-3. When setting a filter, first select a column to filter on using the drop-down box at the upper left, above the Reset button. The most important filters for malware analysis are Process Name, Operation, and Detail. Next, select a comparator, choosing from options such as Is, Contains, and Less Than. Finally, choose whether this is a filter to include or exclude from display. Because, by default, the display will show all system calls, it is important to reduce the amount displayed.

Figure 3-3. Setting a procmon filter

As you can see in the first two rows of Figure 3-3, we’re filtering on Process Name and Operation. We’ve added a filter on Process Name equal to mm32.exe that’s active when the Operation is set to RegSetValue.

After you’ve chosen a filter, click Add for each, and then click Apply. As a result of applying our filters, the display window shown in the lower image displays only 11 of the 39,351 events, making it easier for us to see that mm32.exe performed a RegSetValue of registry key HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run\Sys32V2Controller (sequence number 3 using RegSetValue). Double-clicking this RegSetValue event will reveal the data written to this location, which is the current path to the malware.

If the malware extracted another executable and ran it, don’t worry, because that information is still there. Remember that the filter controls only the display. All of the system calls that occurred when you ran the malware are captured, including system calls from malware that was extracted by the original executable. If you see any malware extracted, change the filter to display the extracted name, and then click Apply. The events related to the extracted malware will be displayed.

Procmon provides helpful automatic filters on its toolbar. The four filters circled in Figure 3-4 filter by the following categories:

All four filters are selected by default. To turn off a filter, simply click the icon in the toolbar corresponding to the category.

Figure 3-4. Filter buttons for procmon

Analysis of procmon’s recorded events takes practice and patience, since many events are simply part of the standard way that executables start up. The more you use procmon, the easier you will find it to quickly review the event listing.

Process Explorer monitors the processes running on a system and shows them in a tree structure that displays child and parent relationships. For example, in Figure 3-5 you can see that services.exe is a child process of winlogon.exe, as indicated by the left curly bracket.

Figure 3-5. Process Explorer examining svchost.exe malware

Process Explorer shows five columns: Process (the process name), PID (the process identifier), CPU (CPU usage), Description, and Company Name. The view updates every second. By default, services are highlighted in pink, processes in blue, new processes in green, and terminated processes in red. Green and red highlights are temporary, and are removed after the process has started or terminated. When analyzing malware, watch the Process Explorer window for changes or new processes, and be sure to investigate them thoroughly.

Process Explorer can display quite a bit of information for each process. For example, when the DLL information display window is active, you can click a process to see all DLLs it loaded into memory. You can change the DLL display window to the Handles window, which shows all handles held by the process, including file handles, mutexes, events, and so on.

The Properties window shown in Figure 3-6 opens when you double-click a process name. This window can provide some particularly useful information about your subject malware. The Threads tab shows all active threads, the TCP/IP tab displays active connections or ports on which the process is listening, and the Image tab (opened in the figure) shows the path on disk to the executable.

Figure 3-6. The Properties window, Image tab

One particularly useful Process Explorer feature is the Verify button on the Image tab. Click this button to verify that the image on disk is, in fact, the Microsoft signed binary. Because Microsoft uses digital signatures for most of its core executables, when Process Explorer verifies that a signature is valid, you can be sure that the file is actually the executable from Microsoft. This feature is particularly useful for verifying that the Windows file on disk has not been corrupted; malware often replaces authentic Windows files with its own in an attempt to hide.

The Verify button verifies the image on disk rather than in memory, and it is useless if an attacker uses process replacement, which involves running a process on the system and overwriting its memory space with a malicious executable. Process replacement provides the malware with the same privileges as the process it is replacing, so that the malware appears to be executing as a legitimate process, but it leaves a fingerprint: The image in memory will differ from the image on disk. For example, in Figure 3-6, the svchost.exe process is verified, yet it is actually malware. We’ll discuss process replacement in more detail in Chapter 12.

One way to recognize process replacement is to use the Strings tab in the Process Properties window to compare the strings contained in the disk executable (image) against the strings in memory for that same executable running in memory. You can toggle between these string views using the buttons at the bottom-left corner, as shown in Figure 3-7. If the two string listings are drastically different, process replacement may have occurred. This string discrepancy is displayed in Figure 3-7. For example, the string FAVORITES.DAT appears multiple times in the right half of the figure (svchost.exe in memory), but it cannot be found in the left half of the figure (svchost.exe on disk).

Figure 3-7. The Process Explorer Strings tab shows strings on disk (left) versus strings in memory (right) for active svchost.exe.

Process Explorer allows you to launch depends.exe (Dependency Walker) on a running process by right-clicking a process name and selecting Launch Depends. It also lets you search for a handle or DLL by choosing Find ▶ Find Handle or DLL.

The Find DLL option is particularly useful when you find a malicious DLL on disk and want to know if any running processes use that DLL. The Verify button verifies the EXE file on disk, but not every DLL loaded during runtime. To determine whether a DLL is loaded into a process after load time, you can compare the DLL list in Process Explorer to the imports shown in Dependency Walker.

You can also use Process Explorer to analyze malicious documents, such as PDFs and Word documents. A quick way to determine whether a document is malicious is to open Process Explorer and then open the suspected malicious document. If the document launches any processes, you should see them in Process Explorer, and be able to locate the malware on disk via the Image tab of the Properties window.

ApateDNS, a free tool from Mandiant (www.mandiant.com/products/research/mandiant_apatedns/download), is the quickest way to see DNS requests made by malware. ApateDNS spoofs DNS responses to a user-specified IP address by listening on UDP port 53 on the local machine. It responds to DNS requests with the DNS response set to an IP address you specify. ApateDNS can display the hexadecimal and ASCII results of all requests it receives.

To use ApateDNS, set the IP address you want sent in DNS responses at ❷ and select the interface at ❹. Next, press the Start Server button; this will automatically start the DNS server and change the DNS settings to localhost. Next, run your malware and watch as DNS requests appear in the ApateDNS window. For example, in Figure 3-9, we redirect the DNS requests made by malware known as RShell. We see that the DNS information is requested for evil.malwar3.com and that request was made at 13:22:08 ❶.

Figure 3-9. ApateDNS responding to a request for evil.malwar3.com

In the example shown in the figure, we redirect DNS requests to 127.0.0.1 (localhost), but you may want to change this address to point to something external, such as a fake web server running on a Linux virtual machine. Because the IP address will differ from that of your Windows malware analysis virtual machine, be sure to enter the appropriate IP address before starting the server. By default ApateDNS will use the current gateway or current DNS settings to insert into DNS responses.

You can catch additional domains used by a malware sample through the use of the nonexistent domain (NXDOMAIN) option at ❸. Malware will often loop through the different domains it has stored if the first or second domains are not found. Using this NXDOMAIN option can trick malware into giving you additional domains it has in its configuration.

Netcat, the “TCP/IP Swiss Army knife,” can be used over both inbound and outbound connections for port scanning, tunneling, proxying, port forwarding, and much more. In listen mode, Netcat acts as a server, while in connect mode it acts as a client. Netcat takes data from standard input for transmission over the network. All the data it receives is output to the screen via standard output.

Let’s look at how you can use Netcat to analyze the malware RShell from Figure 3-9. Using ApateDNS, we redirect the DNS request for evil.malwar3.com to our local host. Assuming that the malware is going out over port 80 (a common choice), we can use Netcat to listen for connections before executing the malware.

Malware frequently uses port 80 or 443 (HTTP or HTTPS traffic, respectively), because these ports are typically not blocked or monitored as outbound connections. Example 3-2 shows an example.

C:\> nc –l –p 80 ❶
POST /cq/frame.htm HTTP/1.1
Host: www.google.com ❷
User-Agent: Mozilla/5.0 (Windows; Windows NT 5.1; TWFsd2FyZUh1bnRlcg==;
rv:1.38)
Accept: text/html, application
Accept-Language: en-US, en:q=
Accept-Encoding: gzip, deflate
Keep-Alive: 300
Content-Type: application/x-form-urlencoded
Content-Length

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

Z:\Malware> ❸

The Netcat (nc) command ❶ shows the options required to listen on a port. The –l flag means listen, and –p (with a port number) specifies the port on which to listen. The malware connects to our Netcat listener because we’re using ApateDNS for redirection. As you can see, RShell is a reverse shell ❸, but it does not immediately provide the shell. The network connection first appears as an HTTP POST request to www.google.com ❷, fake POST data that RShell probably inserts to obfuscate its reverse shell, because network analysts frequently look only at the start of a session.

Analyze the malware found in the file Lab03-01.exe using basic dynamic analysis tools.

Analyze the malware found in the file Lab03-02.dll using basic dynamic analysis tools.

Execute the malware found in the file Lab03-03.exe while monitoring it using basic dynamic analysis tools in a safe environment.

Analyze the malware found in the file Lab03-04.exe using basic dynamic analysis tools. (This program is analyzed further in the Chapter 9 labs.)

The main memory (RAM) for a single program can be divided into the following four major sections, as shown in Figure 4-3.

Figure 4-3. Basic memory layout for a program

Although the diagram in Figure 4-3 shows the four major sections of memory in a particular order, these pieces may be located throughout memory. For example, there is no guarantee that the stack will be lower than the code or vice versa.

Instructions are the building blocks of assembly programs. In x86 assembly, an instruction is made of a mnemonic and zero or more operands. As shown in Table 4-1, the mnemonic is a word that identifies the instruction to execute, such as mov, which moves data. Operands are typically used to identify information used by the instruction, such as registers or data.

Each instruction corresponds to opcodes (operation codes) that tell the CPU which operation the program wants to perform. This book and other sources use the term opcode for the entire machine instruction, although Intel technically defines it much more narrowly.

Disassemblers translate opcodes into human-readable instructions. For example, in Table 4-2, you can see that the opcodes are B9 42 00 00 00 for the instruction mov ecx, 0x42. The value 0xB9 corresponds to mov ecx, and 0x42000000 corresponds to the value 0x42.

0x42000000 is treated as the value 0x42 because the x86 architecture uses the little-endian format. The endianness of data describes whether the most significant (big-endian) or least significant (little-endian) byte is ordered first (at the smallest address) within a larger data item. Changing between endianness is something malware must do during network communication, because network data uses big-endian and an x86 program uses little-endian. Therefore, the IP address 127.0.0.1 will be represented as 0x7F000001 in big-endian format (over the network) and 0x0100007F in little-endian format (locally in memory). As a malware analyst, you must be cognizant of endianness to ensure you don’t accidentally reverse the order of important indicators like an IP address.

Operands are used to identify the data used by an instruction. Three types of operands can be used:

A register is a small amount of data storage available to the CPU, whose contents can be accessed more quickly than storage available elsewhere. x86 processors have a collection of registers available for use as temporary storage or workspace. Table 4-3 shows the most common x86 registers, which fall into the following four categories:

You can use Table 4-3 as a reference throughout this chapter to see how a register is categorized and broken down. The sections that follow discuss each of these register categories in depth.

All general registers are 32 bits in size and can be referenced as either 32 or 16 bits in assembly code. For example, EDX is used to reference the full 32-bit register, and DX is used to reference the lower 16 bits of the EDX register.

Four registers (EAX, EBX, ECX, and EDX) can also be referenced as 8-bit values using the lowest 8 bits or the second set of 8 bits. For example, AL is used to reference the lowest 8 bits of the EAX register, and AH is used to reference the second set of 8 bits.

Table 4-3 lists the possible references for each general register. The EAX register breakdown is illustrated in Figure 4-4. In this example, the 32-bit (4-byte) register EAX contains the value 0xA9DC81F5, and code can reference the data inside EAX in three additional ways: AX (2 bytes) is 0x81F5, AL (1 byte) is 0xF5, and AH (1 byte) is 0x81.

General Registers

The general registers typically store data or memory addresses, and are often used interchangeably to get things accomplished within the program. However, despite being called general registers, they aren’t always used that way.

Figure 4-4. x86 EAX register breakdown

Some x86 instructions use specific registers by definition. For example, the multiplication and division instructions always use EAX and EDX.

In addition to instruction definitions, general registers can be used in a consistent fashion throughout a program. The use of registers in a consistent fashion across compiled code is known as a convention. Knowledge of the conventions used by compilers allows a malware analyst to examine the code more quickly, because time isn’t wasted figuring out the context of how a register is being used. For example, the EAX generally contains the return value for function calls. Therefore, if you see the EAX register used immediately after a function call, you are probably seeing the code manipulate the return value.

The simplest and most common instruction is mov, which is used to move data from one location to another. In other words, it’s the instruction for reading and writing to memory. The mov instruction can move data into registers or RAM. The format is mov destination, source. (We use Intel syntax throughout the book, which lists the destination operand first.)

Table 4-4 contains examples of the mov instruction. Operands surrounded by brackets are treated as memory references to data. For example, [ebx] references the data at the memory address EBX. The final example in Table 4-4 uses an equation to calculate a memory address. This saves space, because it does not require separate instructions to perform the calculation contained within the brackets. Performing calculations such as this within an instruction is not possible unless you are calculating a memory address. For example, mov eax, ebx+esi*4 (without the brackets) is an invalid instruction.

Another instruction similar to mov is lea, which means “load effective address.” The format of the instruction is lea destination, source. The lea instruction is used to put a memory address into the destination. For example, lea eax, [ebx+8] will put EBX+8 into EAX. In contrast, mov eax, [ebx+8] loads the data at the memory address specified by EBX+8. Therefore, lea eax, [ebx+8] would be the same as mov eax, ebx+8; however, a mov instruction like that is invalid.

Figure 4-5 shows values for registers EAX and EBX on the left and the information contained in memory on the right. EBX is set to 0xB30040. At address 0xB30048 is the value 0x20. The instruction mov eax, [ebx+8] places the value 0x20 (obtained from memory) into EAX, and the instruction lea eax, [ebx+8] places the value 0xB30048 into EAX.

Figure 4-5. EBX register used to access memory

The lea instruction is not used exclusively to refer to memory addresses. It is useful when calculating values, because it requires fewer instructions. For example, it is common to see an instruction such as lea ebx, [eax*4+4], where eax is a number, rather than a memory address. This instruction is the functional equivalent of ebx = (eax+1)*5, but the former is shorter or more efficient for the compiler to use instead of a total of four instructions (for example inc eax; mov ecx, 5; mul ecx; mov ebx, eax).

Arithmetic

x86 assembly includes many instructions for arithmetic, ranging from basic addition and subtraction to logical operators. We’ll cover the most commonly used instructions in this section.

Addition or subtraction adds or subtracts a value from a destination operand. The format of the addition instruction is add destination, value. The format of the subtraction instruction is sub destination, value. The sub instruction modifies two important flags: the zero flag (ZF) and carry flag (CF). The ZF is set if the result is zero, and CF is set if the destination is less than the value subtracted. The inc and dec instructions increment or decrement a register by one. Table 4-5 shows examples of the addition and subtraction instructions.

Table 4-5. Addition and Subtraction Instruction Examples

Instruction	Description
sub eax, 0x10	Subtracts 0x10 from EAX
add eax, ebx	Adds EBX to EAX and stores the result in EAX
inc edx	Increments EDX by 1
dec ecx	Decrements ECX by 1

Multiplication and division both act on a predefined register, so the command is simply the instruction, plus the value that the register will be multiplied or divided by. The format of the mul instruction is mul value. Similarly, the format of div instruction is div value. The assignment of the register on which a mul or div instruction acts can occur many instructions earlier, so you might need to search through a program to find it.

The mul value instruction always multiplies eax by value. Therefore, EAX must be set up appropriately before the multiplication occurs. The result is stored as a 64-bit value across two registers: EDX and EAX. EDX stores the most significant 32 bits of the operations, and EAX stores the least significant 32 bits. Figure 4-6 depicts the values in EDX and EAX when the decimal result of multiplication is 5,000,000,000 and is too large to fit in a single register.

The div value instruction does the same thing as mul, except in the opposite direction: It divides the 64 bits across EDX and EAX by value. Therefore, the EDX and EAX registers must be set up appropriately before the division occurs. The result of the division operation is stored in EAX, and the remainder is stored in EDX.

Figure 4-6. Multiplication result stored across EDX and EAX registers

A programmer obtains the remainder of a division operation by using an operation known as modulo, which will be compiled into assembly through the use of the EDX register after the div instruction (since it contains the remainder). Table 4-6 shows examples of the mul and div instructions. The instructions imul and idiv are the signed versions of the mul and div instructions.

Table 4-6. Multiplication and Division Instruction Examples

Instruction	Description
mul 0x50	Multiplies EAX by 0x50 and stores the result in EDX:EAX
div 0x75	Divides EDX:EAX by 0x75 and stores the result in EAX and the remainder in EDX

Logical operators such as OR, AND, and XOR are used in x86 architecture. The corresponding instructions operate similar to how add and sub operate. They perform the specified operation between the source and destination operands and store the result in the destination. The xor instruction is frequently encountered in disassembly. For example, xor eax, eax is a quick way to set the EAX register to zero. This is done for optimization, because this instruction requires only 2 bytes, whereas mov eax, 0 requires 5 bytes.

The shr and shl instructions are used to shift registers. The format of the shr instruction is shr destination, count, and the shl instruction has the same format. The shr and shl instructions shift the bits in the destination operand to the right and left, respectively, by the number of bits specified in the count operand. Bits shifted beyond the destination boundary are first shifted into the CF flag. Zero bits are filled in during the shift. For example, if you have the binary value 1000 and shift it right by 1, the result is 0100. At the end of the shift instruction, the CF flag contains the last bit shifted out of the destination operand.

The rotation instructions, ror and rol, are similar to the shift instructions, except the shifted bits that “fall off” with the shift operation are rotated to the other end. In other words, during a right rotation (ror) the least significant bits are rotated to the most significant position. Left rotation (rol) is the exact opposite. Table 4-7 displays examples of these instructions.

Table 4-7. Common Logical and Shifting Arithmetic Instructions

Instruction	Description
xor eax, eax	Clears the EAX register
or eax, 0x7575	Performs the logical or operation on EAX with 0x7575
mov eax, 0xA shl eax, 2	Shifts the EAX register to the left 2 bits; these two instructions result in EAX = 0x28, because 1010 (0xA in binary) shifted 2 bits left is 101000 (0x28)
mov bl, 0xA ror bl, 2	Rotates the BL register to the right 2 bits; these two instructions result in BL = 10000010, because 1010 rotated 2 bits right is 10000010

Shifting is often used in place of multiplication as an optimization. Shifting is simpler and faster than multiplication, because you don’t need to set up registers and move data around, as you do for multiplication. The shl eax, 1 instruction computes the same result as multiplying EAX by two. Shifting to the left two bit positions multiplies the operand by four, and shifting to the left three bit positions multiplies the operand by eight. Shifting an operand to the left n bits multiplies it by 2ⁿ.

During malware analysis, if you encounter a function containing only the instructions xor, or, and, shl, ror, shr, or rol repeatedly and seemingly randomly, you have probably encountered an encryption or compression function. Don’t get bogged down trying to analyze each instruction unless you really need to do so. Instead, your best bet in most cases is to mark this as an encryption routine and move on.

Memory for functions, local variables, and flow control is stored in a stack, which is a data structure characterized by pushing and popping. You push items onto the stack, and then pop those items off. A stack is a last in, first out (LIFO) structure. For example, if you push the numbers 1, 2, and then 3 (in order), the first item to pop off will be 3, because it was the last item pushed onto the stack.

The x86 architecture has built-in support for a stack mechanism. The register support includes the ESP and EBP registers. ESP is the stack pointer and typically contains a memory address that points to the top of stack. The value of this register changes as items are pushed on and popped off the stack. EBP is the base pointer that stays consistent within a given function, so that the program can use it as a placeholder to keep track of the location of local variables and parameters.

The stack instructions include push, pop, call, leave, enter, and ret. The stack is allocated in a top-down format in memory, and the highest addresses are allocated and used first. As values are pushed onto the stack, smaller addresses are used (this is illustrated a bit later in Figure 4-7).

The stack is used for short-term storage only. It frequently stores local variables, parameters, and the return address. Its primary usage is for the management of data exchanged between function calls. The implementation of this management varies among compilers, but the most common convention is for local variables and parameters to be referenced relative to EBP.

Stack Layout

As discussed, the stack is allocated in a top-down fashion, with the higher memory addresses used first. Figure 4-7 shows how the stack is laid out in memory. Each time a call is performed, a new stack frame is generated. A function maintains its own stack frame until it returns, at which time the caller’s stack frame is restored and execution is transferred back to the calling function.

Figure 4-7. x86 stack layout

Figure 4-8 shows a dissection of one of the individual stack frames from Figure 4-7. The memory locations of individual items are also displayed. In this diagram, ESP would point to the top of the stack, which is the memory address 0x12F02C. EBP would be set to 0x12F03C throughout the duration of the function, so that the local variables and arguments can be referenced using EBP. The arguments that are pushed onto the stack before the call are shown at the bottom of the stack frame. Next, it contains the return address that is put on the stack automatically by the call instruction. The old EBP is next on the stack; this is the EBP from the caller’s stack frame.

When information is pushed onto the stack, ESP will be decreased. In the example in Figure 4-8, if the instruction push eax were executed, ESP would be decremented by four and would contain 0x12F028, and the data contained in EAX would be copied to 0x12F028. If the instruction pop ebx were executed, the data at 0x12F028 would be moved into the EBX register, and then ESP would be incremented by four.

Figure 4-8. Individual stack frame

It is possible to read data from the stack without using the push or pop instructions. For example, the instruction mov eax, ss:[esp] will directly access the top of the stack. This is identical to pop eax, except the ESP register is not impacted. The convention used depends on the compiler and how the compiler is configured. (We discuss this in more detail in Chapter 6.)

The x86 architecture provides additional instructions for popping and pushing, the most popular of which are pusha and pushad. These instructions push all the registers onto the stack and are commonly used with popa and popad, which pop all the registers off the stack. The pusha and pushad functions operate as follows:

• pusha pushes the 16-bit registers on the stack in the following order: AX, CX, DX, BX, SP, BP, SI, DI.

• pushad pushes the 32-bit registers on the stack in the following order: EAX, ECX, EDX, EBX, ESP, EBP, ESI, EDI.

These instructions are typically encountered in shellcode when someone wants to save the current state of the registers to the stack so that they can be restored at a later time. Compilers rarely use these instructions, so seeing them often indicates someone hand-coded assembly and/or shellcode.

All programming languages have the ability to make comparisons and make decisions based on those comparisons. Conditionals are instructions that perform the comparison.

The two most popular conditional instructions are test and cmp. The test instruction is identical to the and instruction; however, the operands involved are not modified by the instruction. The test instruction only sets the flags. The zero flag (ZF) is typically the flag of interest after the test instruction. A test of something against itself is often used to check for NULL values. An example of this is test eax, eax. You could also compare EAX to zero, but test eax, eax uses fewer bytes and fewer CPU cycles.

The cmp instruction is identical to the sub instruction; however, the operands are not affected. The cmp instruction is used only to set the flags. The zero flag and carry flag (CF) may be changed as a result of the cmp instruction. Table 4-8 shows how the cmp instruction impacts the flags.

A branch is a sequence of code that is conditionally executed depending on the flow of the program. The term branching is used to describe the control flow through the branches of a program.

The most popular way branching occurs is with jump instructions. An extensive set of jump instructions is used, of which the jmp instruction is the simplest. The format jmp location causes the next instruction executed to be the one specified by the jmp. This is known as an unconditional jump, because execution will always transfer to the target location. This simple jump will not satisfy all of your branching needs. For example, the logical equivalent to an if statement isn’t possible with a jmp. There is no if statement in assembly code. This is where conditional jumps come in.

Conditional jumps use the flags to determine whether to jump or to proceed to the next instruction. More than 30 different types of conditional jumps can be used, but only a small set of them is commonly encountered. Table 4-9 shows the most common conditional jump instructions and details of how they operate. Jcc is the shorthand for generally describing conditional jumps.

Rep instructions are a set of instructions for manipulating data buffers. They are usually in the form of an array of bytes, but they can also be single or double words. We will focus on arrays of bytes in this section. (Intel refers to these instructions as string instructions, but we won’t use this term to avoid confusion with the strings we discussed in Chapter 1.)

The most common data buffer manipulation instructions are movsx, cmpsx, stosx, and scasx, where x = b, w, or d for byte, word, or double word, respectively. These instructions work with any type of data, but our focus in this section will be bytes, so we will use movsb, cmpsb, and so on.

The ESI and EDI registers are used in these operations. ESI is the source index register, and EDI is the destination index register. ECX is used as the counting variable.

These instructions require a prefix to operate on data lengths greater than 1. The movsb instruction will move only a single byte and does not utilize the ECX register.

In x86, the repeat prefixes are used for multibyte operations. The rep instruction increments the ESI and EDI offsets, and decrements the ECX register. The rep prefix will continue until ECX = 0. The repe/repz and repne/repnz prefixes will continue until ECX = 0 or until the ZF = 1 or 0. This is illustrated in Table 4-10. Therefore, in most data buffer manipulation instructions, ESI, EDI, and ECX must be properly initialized for the rep instruction to be useful.

The movsb instruction is used to move a sequence of bytes from one location to another. The rep prefix is commonly used with movsb to copy a sequence of bytes, with size defined by ECX. The rep movsb instruction is the logical equivalent of the C memcpy function. The movsb instruction grabs the byte at address ESI, stores it at address EDI, and then increments or decrements the ESI and EDI registers by one according to the setting of the direction flag (DF). If DF = 0, they are incremented; otherwise, they are decremented.

You rarely see this in compiled C code, but in shellcode, people will sometimes flip the direction flag so they can store data in the reverse direction. If the rep prefix is present, the ECX is checked to see if it contains zero. If not, then the instruction moves the byte from ESI to EDI and decrements the ECX register. This process repeats until ECX = 0.

The cmpsb instruction is used to compare two sequences of bytes to determine whether they contain the same data. The cmpsb instruction subtracts the value at location EDI from the value at ESI and updates the flags. It is typically used with the repe prefix. When coupled with the repe prefix, the cmpsb instruction compares each byte of the two sequences until it finds a difference between the sequences or reaches the end of the comparison. The cmpsb instruction obtains the byte at address ESI, compares the value at location EDI to set the flags, and then increments the ESI and EDI registers by one. If the repe prefix is present, ECX is checked and the flags are also checked, but if ECX = 0 or ZF = 0, the operation will stop repeating. This is equivalent to the C function memcmp.

The scasb instruction is used to search for a single value in a sequence of bytes. The value is defined by the AL register. This works in the same way as cmpsb, but it compares the byte located at address ESI to AL, rather than to EDI. The repe operation will continue until the byte is found or ECX = 0. If the value is found in the sequence of bytes, ESI stores the location of that value.

The stosb instruction is used to store values in a location specified by EDI. This is identical to scasb, but instead of being searched for, the specified byte is placed in the location specified by EDI. The rep prefix is used with scasb to initialize a buffer of memory, wherein every byte contains the same value. This is equivalent to the C function memset. Table 4-11 displays some common rep instructions and describes their operation.

Because malware is often written in C, it’s important that you know how the main method of a C program translates to assembly. This knowledge will also help you understand how offsets differ when you go from C code to assembly.

A standard C program has two arguments for the main method, typically in this form:

int main(int argc, char ** argv)

The parameters argc and argv are determined at runtime. The argc parameter is an integer that contains the number of arguments on the command line, including the program name. The argv parameter is a pointer to an array of strings that contain the command-line arguments. The following example shows a command-line program and the results of argc and argv when the program is run.

filetestprogram.exe -r filename.txt

argc = 3
argv[0] = filetestprogram.exe
argv[1] = -r
argv[2] = filename.txt

Example 4-1 shows the C code for a simple program.

int main(int argc, char* argv[])
{
      if (argc != 3) {return 0;}

      if (strncmp(argv[1], "-r", 2) == 0){

            DeleteFileA(argv[2]);

      }
      return 0;
}

Example 4-2 shows the C code from Example 4-1 in compiled form. This example will help you understand how the parameters listed in Table 4-12 are accessed in assembly code. argc is compared to 3 at ❶, and argv[1] is compared to -r at ❷ through the use of a strncmp. Notice how argv[1] is accessed: First the location of the beginning of the array is loaded into eax, and then 4 (the offset) is added to eax to get argv[1]. The number 4 is used because each entry in the argv array is an address to a string, and each address is 4 bytes in size on a 32-bit system. If -r is provided on the command line, the code starting at ❸ will be executed, which is when we see argv[2] accessed at offset 8 relative to argv and provided as an argument to the DeleteFileA function.

004113CE                 cmp     [ebp+argc], 3 ❶
004113D2                 jz      short loc_4113D8
004113D4                 xor     eax, eax
004113D6                 jmp     short loc_411414
004113D8                 mov     esi, esp
004113DA                 push    2               ; MaxCount
004113DC                 push    offset Str2     ; "-r"
004113E1                 mov     eax, [ebp+argv]
004113E4                 mov     ecx, [eax+4]
004113E7                 push    ecx             ; Str1
004113E8                 call    strncmp ❷
004113F8                 test    eax, eax
004113FA                 jnz     short loc_411412
004113FC                 mov     esi, esp ❸
004113FE                 mov     eax, [ebp+argv]
00411401                 mov     ecx, [eax+8]
00411404                 push    ecx             ; lpFileName
00411405                 call    DeleteFileA

What if you encounter an instruction you have never seen before? If you can’t find your answer with a Google search, you can download the complete x86 architecture manuals from Intel at http://www.intel.com/products/processor/manuals/index.htm. This set includes the following:

Volume 1: Basic Architecture

Volume 2A: Instruction Set Reference, A–M, and Volume 2B: Instruction Set Reference, N–Z

Volume 3A: System Programming Guide, Part I, and Volume 3B: System Programming Guide, Part II

Optimization Reference Manual

You can display the disassembly window in one of two modes: graph (the default, shown in Figure 5-2) and text. To switch between modes, press the spacebar.

Graph Mode

In graph mode, IDA Pro excludes certain information that we recommend you display, such as line numbers and operation codes. To change these options, select Options ▶ General, and then select Line prefixes and set the Number of Opcode Bytes to 6. Because most instructions contain 6 or fewer bytes, this setting will allow you to see the memory locations and opcode values for each instruction in the code listing. (If these settings make everything scroll off the screen to the right, try setting the Instruction Indentation to 8.)

Figure 5-2. Graph mode of the IDA Pro disassembly window

In graph mode, the color and direction of the arrows help show the program’s flow during analysis. The arrow’s color tells you whether the path is based on a particular decision having been made: red if a conditional jump is not taken, green if the jump is taken, and blue for an unconditional jump. The arrow direction shows the program’s flow; upward arrows typically denote a loop situation. Highlighting text in graph mode highlights every instance of that text in the disassembly window.

Text Mode

The text mode of the disassembly window is a more traditional view, and you must use it to view data regions of a binary. Figure 5-3 displays the text mode view of a disassembled function. It displays the memory address (0040105B) and section name (.text) in which the opcodes (83EC18) will reside in memory ❶.

The left portion of the text-mode display is known as the arrows window and shows the program’s nonlinear flow. Solid lines mark unconditional jumps, and dashed lines mark conditional jumps. Arrows facing up indicate a loop. The example includes the stack layout for the function at ❷ and a comment (beginning with a semicolon) that was automatically added by IDA Pro ❸.

Note

If you are still learning assembly code, you should find the auto comments feature of IDA Pro useful. To turn on this feature, select Options ▶ General, and then check the Auto comments checkbox. This adds additional comments throughout the disassembly window to aid your analysis.

Figure 5-3. Text mode of IDA Pro’s disassembly window

Several other IDA Pro windows highlight particular items in an executable. The following are the most significant for our purposes.

These windows also offer a cross-reference feature that is particularly useful in locating interesting code. For example, to find all code locations that call an imported function, you could use the import window, double-click the imported function of interest, and then use the cross-reference feature to locate the import call in the code listing.

The IDA Pro interface is so rich that, after pressing a few keys or clicking something, you may find it impossible to navigate. To return to the default view, choose Windows ▶ Reset Desktop. Choosing this option won’t undo any labeling or disassembly you’ve done; it will simply restore any windows and GUI elements to their defaults.

By the same token, if you’ve modified the window and you like what you see, you can save the new view by selecting Windows ▶ Save desktop.

As we just noted, IDA Pro can be tricky to navigate. Many windows are linked to the disassembly window. For example, double-clicking an entry within the Imports window or Strings window will take you directly to that entry.

00401075        jnz     short ❶ loc_40107E
00401077        mov     [ebp+var_10], 1
0040107E loc_40107E:                  ; CODE XREF: ❶ ❷ sub_401040+35j
0040107E        cmp     [ebp+var_C], 0
00401082        jnz     short ❶ loc_401097
00401084        mov     eax, [ebp+var_4]
00401087        mov     [esp+18h+var_14], eax
0040108B        mov     [esp+18h+var_18], offset ❶ aPrintNumberD ; "Print Number= %d\n"
00401092        call   ❶printf
00401097        call   ❶sub_4010A0

Exploring Your History

IDA Pro’s forward and back buttons, shown in Figure 5-4, make it easy to move through your history, just as you would move through a history of web pages in a browser. Each time you navigate to a new location within the disassembly window, that location is added to your history.

Figure 5-4. Navigational buttons

Selecting Search from the top menu will display many options for moving the cursor in the disassembly window:

The following example displays the command-line analysis of the password.exe binary. This malware requires a password to continue running, and you can see that it prints the string Bad key after we enter an invalid password (test).

C:\>password.exe
Enter password for this Malware: test
Bad key

We then pull this binary into IDA Pro and see how we can use the search feature and links to unlock the program. We begin by searching for all occurrences of the Bad key string, as shown in Figure 5-5. We notice that Bad key is used at 0x401104 ❶, so we jump to that location in the disassembly window by double-clicking the entry in the search window.

Figure 5-5. Searching example

The disassembly listing around the location of 0x401104 is shown next. Looking through the listing, before "Bad key\n", we see a comparison at 0x4010F1, which tests the result of a strcmp. One of the parameters to the strcmp is the string, and likely password, $mab.

004010E0        push    offset aMab     ; "$mab"
004010E5        lea     ecx, [ebp+var_1C]
004010E8        push    ecx
004010E9        call    strcmp
004010EE        add     esp, 8
004010F1        test    eax, eax
004010F3        jnz     short loc_401104
004010F5        push    offset aKeyAccepted ; "Key Accepted!\n"
004010FA        call    printf
004010FF        add     esp, 4
00401102        jmp     short loc_401118
00401104 loc_401104                    ; CODE XREF: _main+53j
00401104        push    offset aBadKey  ; "Bad key\n"
00401109        call    printf

The next example shows the result of entering the password we discovered, $mab, and the program prints a different result.

C:\>password.exe
Enter password for this Malware: $mab
Key Accepted!
The malware has been unlocked

This example demonstrates how quickly you can use the search feature and links to get information about a binary.

Example 5-2 shows a code cross-reference at ❶ that tells us that this function (sub_401000) is called from inside the main function at offset 0x3 into the main function. The code cross-reference for the jump at ❷ tells us which jump takes us to this location, which in this example corresponds to the location marked at ❸. We know this because at offset 0x19 into sub_401000 is the jmp at memory address 0x401019.

00401000        sub_401000      proc near      ; ❶CODE XREF: _main+3p
00401000        push    ebp
00401001        mov     ebp, esp
00401003   loc_401003:                         ; ❷CODE XREF: sub_401000+19j
00401003        mov     eax, 1
00401008        test    eax, eax
0040100A        jz      short loc_40101B
0040100C        push    offset aLoop    ; "Loop\n"
00401011        call    printf
00401016        add     esp, 4
00401019        jmp     short loc_401003 ❸

By default, IDA Pro shows only a couple of cross-references for any given function, even though many may occur when a function is called. To view all the cross-references for a function, click the function name and press X on your keyboard. The window that pops up should list all locations where this function is called. At the bottom of the Xrefs window in Figure 5-6, which shows a list of cross-references for sub_408980, you can see that this function is called 64 times (“Line 1 of 64”).

Figure 5-6. Xrefs window

Double-click any entry in the Xrefs window to go to the corresponding reference in the disassembly window.

Data cross-references are used to track the way data is accessed within a binary. Data references can be associated with any byte of data that is referenced in code via a memory reference, as shown in Example 5-3. For example, you can see the data cross-reference to the DWORD 0x7F000001 at ❶. The corresponding cross-reference tells us that this data is used in the function located at 0x401020. The following line shows a data cross-reference for the string <Hostname> <Port>.

0040C000 dword_40C000    dd 7F000001h        ; ❶DATA XREF: sub_401020+14r
0040C004 aHostnamePort   db '<Hostname> <Port>',0Ah,0  ; DATA XREF: sub_401000+3o

Recall from Chapter 1 that the static analysis of strings can often be used as a starting point for your analysis. If you see an interesting string, use IDA Pro’s cross-reference feature to see exactly where and how that string is used within the code.

IDA Pro does a good job of automatically naming virtual address and stack variables, but you can also modify these names to make them more meaningful. Auto-generated names (also known as dummy names) such as sub_401000 don’t tell you much; a function named ReverseBackdoorThread would be a lot more useful. You should rename these dummy names to something more meaningful. This will also help ensure that you reverse-engineer a function only once. When renaming dummy names, you need to do so in only one place. IDA Pro will propagate the new name wherever that item is referenced.

After you’ve renamed a dummy name to something more meaningful, cross-references will become much easier to parse. For example, if a function sub_401200 is called many times throughout a program and you rename it to DNSrequest, it will be renamed DNSrequest throughout the program. Imagine how much time this will save you during analysis, when you can read the meaningful name instead of needing to reverse the function again or to remember what sub_401200 does.

Table 5-2 shows an example of how we might rename local variables and arguments. The left column contains an assembly listing with no arguments renamed, and the right column shows the listing with the arguments renamed. We can actually glean some information from the column on the right. Here, we have renamed arg_4 to port_str and var_598 to port. You can see that these renamed elements are much more meaningful than their dummy names.

IDA Pro lets you embed comments throughout your disassembly and adds many comments automatically.

To add your own comments, place the cursor on a line of disassembly and press the colon (:) key on your keyboard to bring up a comment window. To insert a repeatable comment to be echoed across the disassembly window whenever there is a cross-reference to the address in which you added the comment, press the semicolon (;) key.

When disassembling, IDA Pro makes decisions regarding how to format operands for each instruction that it disassembles. Unless there is context, the data displayed is typically formatted as hex values. IDA Pro allows you to change this data if needed to make it more understandable.

004013C8  mov   eax, [ebp+arg_4]
004013CB  push  eax
004013CC  call  _atoi
004013D1  add   esp, 4
004013D4  mov [ebp+var_598], ax
004013DB  movzx ecx, [ebp+var_598]
004013E2  test  ecx, ecx
004013E4  jnz   short loc_4013F8
004013E6  push  offset aError
004013EB  call  printf
004013F0  add   esp, 4
004013F3  jmp   loc_4016FB
004013F8 ; ----------------------
004013F8
004013F8 loc_4013F8:
004013F8  movzx edx, [ebp+var_598]
004013FF  push  edx
00401400  call  ds:htons

004013C8  mov   eax, [ebp+port_str]
004013CB  push  eax
004013CC  call  _atoi
004013D1  add   esp, 4
004013D4  mov   [ebp+port], ax
004013DB  movzx ecx, [ebp+port]
004013E2  test  ecx, ecx
004013E4  jnz   short loc_4013F8
004013E6  push  offset aError
004013EB  call  printf
004013F0  add   esp, 4
004013F3  jmp   loc_4016FB
004013F8 ; --------------------
004013F8
004013F8 loc_4013F8:
004013F8  movzx edx, [ebp+port]
004013FF  push  edx
00401400  call  ds:htons

Figure 5-10 shows an example of modifying operands in an instruction, where 62h is compared to the local variable var_4. If you were to right-click 62h, you would be presented with options to change the 62h into 98 in decimal, 142o in octal, 1100010b in binary, or the character b in ASCII—whatever suits your needs and your situation.

Figure 5-10. Function operand manipulation

To change whether an operand references memory or stays as data, press the O key on your keyboard. For example, suppose when you’re analyzing disassembly with a link to loc_410000, you trace the link back and see the following instructions:

mov eax, loc_410000
add ebx, eax
mul ebx

At the assembly level, everything is a number, but IDA Pro has mislabeled the number 4259840 (0x410000 in hex) as a reference to the address 410000. To correct this mistake, press the O key to change this address to the number 410000h and remove the offending cross-reference from the disassembly window.

Malware authors (and programmers in general) often use named constants such as GENERIC_READ in their source code. Named constants provide an easily remembered name for the programmer, but they are implemented as an integer in the binary. Unfortunately, once the compiler is done with the source code, it is no longer possible to determine whether the source used a symbolic constant or a literal.

Fortunately, IDA Pro provides a large catalog of named constants for the Windows API and the C standard library, and you can use the Use Standard Symbolic Constant option (shown in Figure 5-10) on an operand in your disassembly. Figure 5-11 shows the window that appears when you select Use Standard Symbolic Constant on the value 0x800000000.

Figure 5-11. Standard symbolic constant window

The code snippets in Table 5-3 show the effect of applying the standard symbolic constants for a Windows API call to CreateFileA. Note how much more meaningful the code is on the right.

Sometimes a particular standard symbolic constant that you want will not appear, and you will need to load the relevant type library manually. To do so, select View ▶ Open Subviews ▶ Type Libraries to view the currently loaded libraries. Normally, mssdk and vc6win will automatically be loaded, but if not, you can load them manually (as is often necessary with malware that uses the Native API, the Windows NT family API). To get the symbolic constants for the Native API, load ntapi (the Microsoft Windows NT 4.0 Native API). In the same vein, when analyzing a Linux binary, you may need to manually load the gnuunx (GNU C++ UNIX) libraries.

mov     esi, [esp+1Ch+argv]
mov     edx, [esi+4]
mov     edi, ds:CreateFileA
push    0    ; hTemplateFile
push    80h  ; dwFlagsAndAttributes
push    3    ; dwCreationDisposition
push    0    ; lpSecurityAttributes
push    1    ; dwShareMode
push    80000000h ; dwDesiredAccess
push    edx ;  lpFileName
call    edi ; CreateFileA

mov     esi, [esp+1Ch+argv]
mov     edx, [esi+4]
mov     edi, ds:CreateFileA
push    NULL  ; hTemplateFile
push    FILE_ATTRIBUTE_NORMAL ; dwFlagsAndAttributes
push    OPEN_EXISTING         ; dwCreationDisposition
push    NULL                  ; lpSecurityAttributes
push    FILE_SHARE_READ       ; dwShareMode
push    GENERIC_READ          ; dwDesiredAccess
push    edx ; lpFileName
call    edi ; CreateFileA

When IDA Pro performs its initial disassembly of a program, bytes are occasionally categorized incorrectly; code may be defined as data, data defined as code, and so on. The most common way to redefine code in the disassembly window is to press the U key to undefine functions, code, or data. When you undefine code, the underlying bytes will be reformatted as a list of raw bytes.

To define the raw bytes as code, press C. For example, Table 5-4 shows a malicious PDF document named paycuts.pdf. At offset 0x8387 into the file, we discover shellcode (defined as raw bytes) at ❶, so we press C at that location. This disassembles the shellcode and allows us to discover that it contains an XOR decoding loop with 0x97 at ❷.

Depending on your goals, you can similarly define raw bytes as data or ASCII strings by pressing D or A, respectively.

IDA Pro has had a built-in scripting language known as IDC that predates the widespread popularity of scripting languages such as Python and Ruby. The IDC subdirectory within the IDA installation directory contains several sample IDC scripts that IDA Pro uses to analyze disassembled texts. Refer to these programs if you want to learn IDC.

IDC scripts are programs made up of functions, with all functions declared as static. Arguments don’t need the type specified, and auto is used to define local variables. IDC has many built-in functions, as described in the IDA Pro help index or the idc.idc file typically included with scripts that use the built-in functions.

In Chapter 1, we discussed the PEiD tool and its plug-in Krypto ANALyzer (KANAL), which can export an IDC script. The IDC script sets bookmarks and comments in the IDA Pro database for a given binary, as shown in Example 5-5.

#include <idc.idc>
static main(void){
      auto slotidx;
      slotidx = 1;
      MarkPosition(0x00403108, 0, 0, 0, slotidx + 0, "RIJNDAEL [S] [char]");
      MakeComm(PrevNotTail(0x00403109), "RIJNDAEL [S] [char]\nRIJNDAEL (AES):
               SBOX (also used in other ciphers).");

      MarkPosition(0x00403208, 0, 0, 0, slotidx + 1, "RIJNDAEL [S-inv] [char]");
      MakeComm(PrevNotTail(0x00403209), "RIJNDAEL [S-inv] [char]\nRIJNDAEL (AES):
               inverse SBOX (for decryption)");
}

To load an IDC script, select File ▶ Script File. The IDC script should be executed immediately, and a toolbar window should open with one button for editing and another for re-executing the script if needed.

IDAPython is fully integrated into the current version of IDA Pro, bringing the power and convenience of Python scripting to binary analysis. IDAPython exposes a significant portion of IDA Pro’s SDK functionality, allowing for far more powerful scripting than offered with IDC. IDAPython has three modules that provide access to the IDA API (idaapi), IDC interface (idc), and IDAPython utility functions (idautils).

IDAPython scripts are programs that use an effective address (EA) to perform the primary method of referencing. There are no abstract data types, and most calls take either an EA or a symbol name string. IDAPython has many wrapper functions around the core IDC functions.

Example 5-6 shows a sample IDAPython script. The goal of this script is to color-code all call instructions in an idb to make them stand out more to the analyst. For example, ScreenEA is a common function that gets the location of the cursor. Heads is a function that will be used to walk through the defined elements, which is each instruction in this case. Once we’ve collected all of the function calls in functionCalls, we iterate through those instructions and use SetColor to set the color.

from idautils import *
from idc import *

heads = Heads(SegStart(ScreenEA()), SegEnd(ScreenEA()))

functionCalls = []

for i in heads:
  if GetMnem(i) == "call":
    functionCalls.append(i)

print "Number of calls found: %d" % (len(functionCalls))

for i in functionCalls:
  SetColor(i, CIC_ITEM, 0xc7fdff)

After you have gained solid experience with IDA Pro, you should consider purchasing a few commercial plug-ins, such as the Hex-Rays Decompiler and zynamics BinDiff. The Hex-Rays Decompiler is a useful plug-in that converts IDA Pro disassembly into a human-readable, C-like pseudocode text. Reading C-like code instead of disassembly can often speed up your analysis because it gets you closer to the original source code the malware author wrote.

zynamics BinDiff is a useful tool for comparing two IDA Pro databases. It allows you to pinpoint differences between malware variants, including new functions and differences between similar functions. One of its features is the ability to provide a similarity rating when you’re comparing two pieces of malware. We describe these IDA Pro extensions more extensively in Appendix B.

Analyze the malware found in the file Lab05-01.dll using only IDA Pro. The goal of this lab is to give you hands-on experience with IDA Pro. If you’ve already worked with IDA Pro, you may choose to ignore these questions and focus on reverse-engineering the malware.

IDA Pro has a graphing tool that is useful in recognizing constructs, as shown in Figure 6-1. This feature is the default view for analyzing functions.

Figure 6-1 shows a graph of the assembly code example in Example 6-9. As you can see, two different paths (❶ and ❷) of code execution lead to the end of the function, and each path prints a different string. Code path ❶ will print "x equals y.", and ❷ will print "x is not equal to y."

IDA Pro adds false ❶ and true ❷ labels at the decision points at the bottom of the upper code box. As you can imagine, graphing a function can greatly speed up the reverse-engineering process.

Example 6-10 shows C code for a nested if statement that is similar to Example 6-8, except that two additional if statements have been added within the original if statement. These additional statements test to determine whether z is equal to 0.

int x = 0;
int y = 1;
int z = 2;

if(x == y){
     if(z==0){
          printf("z is zero and x = y.\n");
     }else{
          printf("z is non-zero and x = y.\n");
     }
}else{
     if(z==0){
          printf("z zero and x != y.\n");
     }else{
          printf("z non-zero and x != y.\n");
     }
}

Figure 6-1. Disassembly graph for the if statement example in Example 6-9

Despite this minor change to the C code, the assembly code is more complicated, as shown in Example 6-11.

00401006        mov     [ebp+var_8], 0
0040100D        mov     [ebp+var_4], 1
00401014        mov     [ebp+var_C], 2
0040101B        mov     eax, [ebp+var_8]
0040101E        cmp     eax, [ebp+var_4]
00401021        jnz     short loc_401047 ❶
00401023        cmp     [ebp+var_C], 0
00401027        jnz     short loc_401038 ❷
00401029        push    offset aZIsZeroAndXY_ ; "z is zero and x = y.\n"
0040102E        call    printf
00401033        add     esp, 4
00401036        jmp     short loc_401045
00401038 loc_401038:
00401038        push    offset aZIsNonZeroAndX ; "z is non-zero and x = y.\n"
0040103D        call    printf
00401042        add     esp, 4
00401045 loc_401045:
00401045        jmp     short loc_401069
00401047 loc_401047:
00401047        cmp     [ebp+var_C], 0
0040104B        jnz     short loc_40105C ❸
0040104D        push    offset aZZeroAndXY_ ; "z zero and x != y.\n"
00401052        call    printf
00401057        add     esp, 4
0040105A        jmp     short loc_401069
0040105C loc_40105C:
0040105C        push    offset aZNonZeroAndXY_ ; "z non-zero and x != y.\n"
00401061        call    printf00401061

As you can see, three different conditional jumps occur. The first occurs if var_4 does not equal var_8 at ❶. The other two occur if var_C is not equal to zero at ❷ and ❸.

The for loop is a basic looping mechanism used in C programming. for loops always have four components: initialization, comparison, execution instructions, and the increment or decrement.

Example 6-12 shows an example of a for loop.

int i;

for(i=0; i<100; i++)
{
   printf("i equals %d\n", i);
}

In this example, the initialization sets i to 0 (zero), and the comparison checks to see if i is less than 100. If i is less than 100, the printf instruction will execute, the increment will add 1 to i, and the process will check to see if i is less than 100. These steps will repeat until i is greater than or equal to 100.

In assembly, the for loop can be recognized by locating the four components—initialization, comparison, execution instructions, and increment/decrement. For example, in Example 6-13, ❶ corresponds to the initialization step. The code between ❸ and ❹ corresponds to the increment that is initially jumped over at ❷ with a jump instruction. The comparison occurs at ❺, and at ❻, the decision is made by the conditional jump. If the jump is not taken, the printf instruction will execute, and an unconditional jump occurs at ❼, which causes the increment to occur.

00401004        mov     [ebp+var_4], 0 ❶
0040100B        jmp     short loc_401016 ❷
0040100D loc_40100D:
0040100D        mov     eax, [ebp+var_4] ❸
00401010        add     eax, 1
00401013        mov     [ebp+var_4], eax ❹
00401016 loc_401016:
00401016        cmp     [ebp+var_4], 64h ❺
0040101A        jge     short loc_40102F ❻
0040101C        mov     ecx, [ebp+var_4]
0040101F        push    ecx
00401020        push    offset aID  ; "i equals %d\n"
00401025        call    printf
0040102A        add     esp, 8
0040102D        jmp     short loc_40100D ❼

A for loop can be recognized using IDA Pro’s graphing mode, as shown in Figure 6-2.

Figure 6-2. Disassembly graph for the for loop example in Example 6-13

In the figure, the upward pointing arrow after the increment code indicates a loop. These arrows make loops easier to recognize in the graph view than in the standard disassembly view. The graph displays five boxes: The top four are the components of the for loop (initialization, comparison, execution, and increment, in that order). The box on the bottom right is the function epilogue, which we described in Chapter 4 as the portion of a function responsible for cleaning up the stack and returning.

The while loop is frequently used by malware authors to loop until a condition is met, such as receiving a packet or command. while loops look similar to for loops in assembly, but they are easier to understand. The while loop in Example 6-14 will continue to loop until the status returned from checkResult is 0.

int status=0;
int result = 0;

while(status == 0){
     result = performAction();
     status = checkResult(result);
}

The assembly code in Example 6-15 looks similar to the for loop, except that it lacks an increment section. A conditional jump occurs at ❶ and an unconditional jump at ❷, but the only way for this code to stop executing repeatedly is for that conditional jump to occur.

00401036        mov     [ebp+var_4], 0
0040103D        mov     [ebp+var_8], 0
00401044 loc_401044:
00401044        cmp     [ebp+var_4], 0
00401048        jnz     short loc_401063 ❶
0040104A        call    performAction
0040104F        mov     [ebp+var_8], eax
00401052        mov     eax, [ebp+var_8]
00401055        push    eax
00401056        call    checkResult
0040105B        add     esp, 4
0040105E        mov     [ebp+var_4], eax
00401061        jmp     short loc_401044 ❷

cdecl is one of the most popular conventions and was described in Chapter 4 when we introduced the stack and function calls. In cdecl, parameters are pushed onto the stack from right to left, the caller cleans up the stack when the function is complete, and the return value is stored in EAX. Example 6-17 shows an example of what the disassembly would look like if the code in Example 6-16 were compiled to use cdecl.

push c
push b
push a
call test
add esp, 12
mov ret, eax

Notice in the highlighted portion that the stack is cleaned up by the caller. In this example, the parameters are pushed onto the stack from right to left, beginning with c.

The popular stdcall convention is similar to cdecl, except stdcall requires the callee to clean up the stack when the function is complete. Therefore, the add instruction highlighted in Example 6-17 would not be needed if the stdcall convention were used, since the function called would be responsible for cleaning up the stack.

The test function in Example 6-16 would be compiled differently under stdcall, because it must be concerned with cleaning up the stack. Its epilogue would need to take care of the cleanup.

stdcall is the standard calling convention for the Windows API. Any code calling these API functions will not need to clean up the stack, since that’s the responsibility of the DLLs that implement the code for the API function.

The fastcall calling convention varies the most across compilers, but it generally works similarly in all cases. In fastcall, the first few arguments (typically two) are passed in registers, with the most commonly used registers being EDX and ECX (the Microsoft fastcall convention). Additional arguments are loaded from right to left, and the calling function is usually responsible for cleaning up the stack, if necessary. It is often more efficient to use fastcall than other conventions, because the code doesn’t need to involve the stack as much.

In addition to using the different calling conventions described so far, compilers may also choose to use different instructions to perform the same operation, usually when the compiler decides to move rather than push things onto the stack. Example 6-18 shows a C code example of a function call. The function adder adds two arguments and returns the result. The main function calls adder and prints the result using printf.

int adder(int a, int b)
{
   return a+b;
}

void main()
{
   int x = 1;
   int y = 2;

   printf("the function returned the number %d\n", adder(x,y));
}

The assembly code for the adder function is consistent across compilers and is displayed in Example 6-19. As you can see, this code adds arg_0 to arg_4 and stores the result in EAX. (As discussed in Chapter 4, EAX stores the return value.)

00401730        push    ebp
00401731        mov     ebp, esp
00401733        mov     eax, [ebp+arg_0]
00401736        add     eax, [ebp+arg_4]
00401739        pop     ebp
0040173A        retn

Table 6-1 displays different calling conventions used by two different compilers: Microsoft Visual Studio and GNU Compiler Collection (GCC). On the left, the parameters for adder and printf are pushed onto the stack before the call. On the right, the parameters are moved onto the stack before the call. You should be prepared for both types of calling conventions, because as an analyst, you won’t have control over the compiler. For example, one instruction on the left does not correspond to any instruction on the right. This instruction restores the stack pointer, which is not necessary on the right because the stack pointer is never altered.

00401746   mov     [ebp+var_4], 1
0040174D   mov     [ebp+var_8], 2
00401754   mov     eax, [ebp+var_8]
00401757   push    eax
00401758   mov     ecx, [ebp+var_4]
0040175B   push    ecx
0040175C   call    adder
00401761   add     esp, 8
00401764   push    eax
00401765   push    offset TheFunctionRet
0040176A   call    ds:printf

00401085    mov     [ebp+var_4], 1
0040108C    mov     [ebp+var_8], 2
00401093    mov     eax, [ebp+var_8]
00401096    mov     [esp+4], eax
0040109A    mov     eax, [ebp+var_4]
0040109D    mov     [esp], eax
004010A0    call    adder

004010A5    mov     [esp+4], eax
004010A9    mov     [esp], offset TheFunctionRet
004010B0    call    printf

Example 6-20 shows a simple switch statement that uses the variable i. Depending on the value of i, the code under the corresponding case value will be executed.

switch(i)
{
   case 1:
      printf("i = %d", i+1);
      break;
   case 2:
      printf("i = %d", i+2);
      break;
   case 3:
      printf("i = %d", i+3);
      break;
   default:
      break;
}

This switch statement has been compiled into the assembly code shown in Example 6-21. It contains a series of conditional jumps between ❶ and ❷. The conditional jump determination is made by the comparison that occurs directly before each jump.

The switch statement has three options, shown at ❸, ❹, and ❺. These code sections are independent of each other because of the unconditional jumps to the end of the listing. (You’ll probably find that switch statements are easier to understand using the graph shown in Figure 6-3.)

00401013        cmp     [ebp+var_8], 1
00401017        jz      short loc_401027 ❶
00401019        cmp     [ebp+var_8], 2
0040101D        jz      short loc_40103D
0040101F        cmp     [ebp+var_8], 3
00401023        jz      short loc_401053
00401025        jmp     short loc_401067 ❷
00401027 loc_401027:
00401027        mov     ecx, [ebp+var_4] ❸
0040102A        add     ecx, 1
0040102D        push    ecx
0040102E        push    offset unk_40C000 ; i = %d
00401033        call    printf
00401038        add     esp, 8
0040103B        jmp     short loc_401067
0040103D loc_40103D:
0040103D        mov     edx, [ebp+var_4] ❹
00401040        add     edx, 2
00401043        push    edx
00401044        push    offset unk_40C004 ; i = %d
00401049        call    printf
0040104E        add     esp, 8
00401051        jmp     short loc_401067
00401053 loc_401053:
00401053        mov     eax, [ebp+var_4] ❺
00401056        add     eax, 3
00401059        push    eax
0040105A        push    offset unk_40C008 ; i = %d
0040105F        call    printf
00401064        add     esp, 8

Figure 6-3 breaks down each of the switch options by splitting up the code to be executed from the next decision to be made. Three of the boxes in the figure, labeled ❶, ❷, and ❸, correspond directly to the case statement’s three different options. Notice that all of these boxes terminate at the bottom box, which is the end of the function. You should be able to use this graph to see the three checks the code must go through when var_8 is greater than 3.

From this disassembly, it is difficult, if not impossible, to know whether the original code was a switch statement or a sequence of if statements, because a compiled switch statement looks like a group of if statements—both can contain a bunch of cmp and Jcc instructions. When performing your disassembly, you may not always be able to get back to the original source code, because there may be multiple ways to represent the same code constructs in assembly, all of which are valid and equivalent.

The next disassembly example is commonly found with large, contiguous switch statements. The compiler optimizes the code to avoid needing to make so many comparisons. For example, if in Example 6-20 the value of i were 3, three different comparisons would take place before the third case was executed. In Example 6-22, we add one case to Example 6-20 (as you can see by comparing the listings), but the assembly code generated is drastically different.

switch(i)
{
   case 1:
      printf("i = %d", i+1);
      break;
   case 2:
      printf("i = %d", i+2);
      break;
   case 3:
      printf("i = %d", i+3);
      break;
   case 4:
      printf("i = %d", i+3);
      break;
   default:
      break;
}

Figure 6-3. Disassembly graph of the if style switch statement example in Example 6-21

The more efficient assembly code in Example 6-23 uses a jump table, shown at ❷, which defines offsets to additional memory locations. The switch variable is used as an index into the jump table.

In this example, ecx contains the switch variable, and 1 is subtracted from it in the first line. In the C code, the switch table range is 1 through 4, and the assembly code must adjust it to 0 through 3 so that the jump table can be properly indexed. The jump instruction at ❶ is where the target is based on the jump table.

In this jump instruction, edx is multiplied by 4 and added to the base of the jump table (0x401088) to determine which case code block to jump to. It is multiplied by 4 because each entry in the jump table is an address that is 4 bytes in size.

00401016        sub     ecx, 1
00401019        mov     [ebp+var_8], ecx
0040101C        cmp     [ebp+var_8], 3
00401020        ja      short loc_401082
00401022        mov     edx, [ebp+var_8]
00401025        jmp     ds:off_401088[edx*4] ❶
0040102C   loc_40102C:
              ...
00401040        jmp     short loc_401082
00401042   loc_401042:
              ...
00401056        jmp     short loc_401082
00401058   loc_401058:
              ...
0040106C        jmp     short loc_401082
0040106E   loc_40106E:
              ...
00401082   loc_401082:
00401082        xor     eax, eax
00401084        mov     esp, ebp
00401086        pop     ebp
00401087        retn
00401087   _main   endp
00401088  ❷off_401088  dd offset loc_40102C
0040108C               dd offset loc_401042
00401090               dd offset loc_401058
00401094               dd offset loc_40106E

The graph in Figure 6-4 for this type of switch statement is clearer than the standard disassembly view.

Figure 6-4. Disassembly graph of jump table switch statement example

As you can see, each of the four cases is broken down clearly into separate assembly code chunks. These chunks appear one after another in a column after the jump table determines which one to use. Notice that all of these boxes and the initial box terminate at the right box, which is the end of the function.

In this lab, you will analyze the malware found in the file Lab06-01.exe.

Analyze the malware found in the file Lab06-02.exe.

In this lab, we’ll analyze the malware found in the file Lab06-03.exe.

In this lab, we’ll analyze the malware found in the file Lab06-04.exe.