Perhaps you are considering developing a processor module, but you notice that an existing processor module does almost everything that you need. If you have the source code for the processor module, then you might easily modify it to suit your needs. On the other hand, if you don’t have the source code, you might feel that you are out of luck. Fortunately, IDA offers a mechanism for customizing existing processors through the use of plug-ins. By hooking the appropriate processor notifications, a plug-in module can intercept calls to one or more of an existing processor’s analyzer, emulator, and outputter stages. Potential applications for customizing a processor include the following:

The following notification codes, declared in processor_t and discussed in idp.hpp, may be hooked by plug-ins that want to intercept calls to various stages of a processor:

The following code excerpts are from a plug-in that modifies the output of the x86 processor module to replace the leave instruction with a cya instruction and to swap the display order for instructions that have two operands (similar to the AT&T-style syntax):

int idaapi init(void) {
    if (ph.id != PLFM_386) return PLUGIN_SKIP;
    hook_to_notification_point(HT_IDP, hook, NULL);
     return PLUGIN_KEEP;
  }

  int idaapi hook(void *user_data, int notification_code, va_list va) {
     switch (notification_code) {
        case processor_t::custom_out: {
         if (cmd.itype == NN_leave) {  //intercept the leave instruction
            MakeLine(SCOLOR_ON SCOLOR_INSN "cya" SCOLOR_OFF);
              return 2;
           }
           else if (cmd.Op2.type != o_void) {
              //intercept 2 operand instructions
              op_t op1 = cmd.Op1;
              op_t op2 = cmd.Op2;
              cmd.Op1 = op2;
              cmd.Op2 = op1;
               (*ph.u_out)();
              cmd.Op1 = op1;
              cmd.Op2 = op2;
              return 2;
           }
        }
     }
     return 0;
  }
    plugin_t PLUGIN = {
    IDP_INTERFACE_VERSION,
   PLUGIN_PROC | PLUGIN_HIDE | PLUGIN_MOD,  // plugin flags
    init,                 // initialize
    term,                 // terminate. this pointer may be NULL.
    run,                  // invoke plugin
    comment,              // long comment about the plugin
    help,                 // multiline help about the plugin
    wanted_name,          // the preferred short name of the plugin
    wanted_hotkey         // the preferred hotkey to run the plugin
  };

The plug-in’s init function verifies that the current processor is the x86 processor and then hooks processor notifications . In the callback hook function, the plug-in processes the custom_out notification to recognize the leave instruction and generates an alternative output line . For two operand instructions, the hook function temporarily saves the operands associated with the current command, before swapping them within the command just prior to invoking the x86 processor’s u_out function to handle all of the details of printing the line. Upon return, the command’s operands are swapped back to their original order. Finally, the plug-in’s flags specify that the plug-in should be loaded when a processor is loaded, should not be listed on the Edit ▸ Plugins menu, and modifies the database. The following output shows the effects of the customizations performed by the plug-in:

.text:00401350            push    ebp
 .text:00401351            mov     400000h, edx
  .text:00401356            mov     esp, ebp
 .text:00401358            mov     offset unk_402060, eax
 .text:0040135D            sub     0Ch, esp
  .text:00401360            mov     edx, [esp+8]
  .text:00401364            mov     eax, [esp+4]
 .text:00401368            mov     offset unk_402060, [esp]
  .text:0040136F            call    sub_401320
 .text:00401374            cya
  .text:00401375            retn

You can observe the plug-in’s effects by noting that constants appear as the first operand in four instructions and that the cya instruction is used in place of the leave instruction .

In Chapter 21, we will look at using a custom processor plug-in to aid in the analysis of certain types of obfuscated binaries.