First, we run PEiD on the Lab18-03.exe file, and it tells us that the packer is PECompact 1.68 - 1.84 -> Jeremy Collake. We load the program into OllyDbg and see several warnings that the file may be packed. We can ignore these warnings.

The program starts at address 0x00405130. We try the Find OEP by Section Hop (Trace Into) option in the OllyDump plug-in. We see the code shown in Example C-176 as OllyDump’s guess at the OEP. However, there are several reasons this doesn’t look like the OEP. The most obvious is that it accesses values above the base pointer at ❶. If this were the file’s entry point, any data above the base pointer would not have been initialized.

0040A110   ENTER 0,0
0040A114   PUSH EBP
0040A115  ❶MOV ESI,DWORD PTR SS:[EBP+8]
0040A118   MOV EDI,DWORD PTR SS:[EBP+C]
0040A11B   CLD
0040A11C   MOV DL,80
0040A11E   MOV AL,BYTE PTR DS:[ESI]
0040A120   INC ESI
0040A121   MOV BYTE PTR DS:[EDI],AL

Next, we try the Find OEP by Section Hop (Trace Over) option and we see that the code stops on a ret instruction at the end of a function in ntdll, which is clearly not the OEP.

Since the OllyDump plug-in didn’t work, we examine the code to see if the tail jump is easy to spot. As shown in Example C-177, we eventually come to some code that looks like a tail jump. This code is a retn instruction followed by a bunch of zero bytes. We know that the code can’t go past this point.

00405622   SCAS DWORD PTR ES:[EDI]
00405623   ADD BH,CH
00405625   STC
00405626  ❶RETN 0EC3F
00405629   ADD BYTE PTR DS:[EAX],AL
0040562B   ADD BYTE PTR DS:[EAX],AL
0040562D   ADD BYTE PTR DS:[EAX],AL

Now, we set a breakpoint on the retn instruction at ❶ and start our program. First, we set a regular breakpoint (INT 3). OllyDbg displays a warning, because the breakpoint is outside the code section and may cause problems. When we run our program, we eventually get an exception that the program can’t handle, and we see that the code at our breakpoint has been changed. Now we know that the code is self-modifying and that our breakpoint has not worked properly.

When dealing with self-modifying code, it’s often useful to use a hardware breakpoint instead of a software breakpoint because the self-modifying code will overwrite the INT 3 (0xcc) instruction used to implement software breakpoints. Starting over with a hardware breakpoint, we run the program and see that it starts to run without ever hitting our breakpoint. This tells us that we probably haven’t found the tail jump and we need to try another strategy.

Looking at the entry point of the packed program, we see the instructions shown in Example C-178.

00405130  ❶JMP SHORT Lab09-02.00405138
00405132   PUSH 1577
00405137   RETN
00405138  ❷PUSHFD
00405139  ❸PUSHAD
0040513A  ❹CALL Lab09-02.00405141
0040513F   XOR EAX,EAX

The first instruction at ❶ is an unconditional jump that skips the next two instructions. The first two instructions that affect memory are pushfd at ❷ and pushad at ❸. These instructions save all of the registers and flags. It’s likely that the packing program will restore all the registers and flags immediately before it jumps to the OEP, so we can try to find the OEP by setting an access breakpoint on the stack. Presumably, there will be a popad or popfd instruction right before the tail jump, which will lead us to the OEP.

We restart the program and step-over the first three instructions. The program should be stopped at the call instruction at ❹ in Example C-178. Now we need to find the value of the stack pointer to set a breakpoint. To do so, we examine the registers window, as shown on the top right of Figure C-67.

Figure C-67. Setting a hardware breakpoint on the stack to help find OEP

The stack is at address 0x12FFA0, as shown at ❶ in Figure C-67. To set a breakpoint, we first load that address in the memory dump by right-clicking ❶ and selecting Follow in Dump. This will make the memory dump window at ❷ appear as it does in Figure C-67.

To set a breakpoint on the last piece of data pushed onto the stack, we right-click the first data element on the stack at ❸ in Figure C-67 and select Breakpoint ▸ Memory on Access. We then run our program. Unfortunately, it reaches an unhandled exception similar to when we set a breakpoint before. Next, we set the breakpoint with Breakpoint ▸ Hardware, on Access ▸ Dword. When we start our program, our breakpoint is triggered. The program will break at the instructions shown in Example C-179.

0040754F   POPFD
00407550   PUSH EAX
00407551   PUSH Lab18-03.00401577
00407556   RETN 4

A few instructions into our code, we see a retn instruction that transfers execution to another location. This is probably the tail jump. We step to that instruction to determine where it goes and see the code in Example C-180. This looks like the original code; the call to GetVersion at ❷ is a dead giveaway.

00401577  ❶PUSH EBP
00401578   MOV EBP,ESP
0040157A   PUSH -1
0040157C   PUSH Lab18-03.004040C0
00401581   PUSH Lab18-03.0040203C         ;  SE handler installation
00401586   MOV EAX,DWORD PTR FS:[0]
0040158C   PUSH EAX
0040158D   MOV DWORD PTR FS:[0],ESP
00401594   SUB ESP,10
00401597   PUSH EBX
00401598   PUSH ESI
00401599   PUSH EDI
0040159A   MOV DWORD PTR SS:[EBP-18],ESP
0040159D  ❷CALL DWORD PTR DS:[404030]     ;  kernel32.GetVersion

Now, with EIP pointing to the first instruction at ❶, we select Plugins ▸ OllyDump ▸ Dump Debugged Process. We click the Get EIP as OEP button, leaving all the other options with their default settings, and then click Dump. In the dialog, we enter a filename to save a copy of our unpacked program.

When we’re finished, we run the program and open it in IDA Pro to verify that it has been unpacked successfully. A brief analysis of the program reveals that the functionality is the same as Lab09-02.exe.

This packer uses a variety of techniques to make it difficult to unpack and recognize the tail jump. Several of the usual strategies were ineffective because the packer takes explicit steps to thwart them. If using a particular technique seems difficult on a packed program, try different approaches until one works. In rare cases, none of the techniques will work easily.