The first variety, ptrace based attacks, also falls under the category of debugging a process. As already discussed, a binary protector wants to make ptrace based debugging very difficult for a reverse engineer. Aside from debugging, however, there are many other attacks that could potentially help break a protected binary, and it is important to know and understand what some of these are in order to give further clarification as to why a binary protector wants to protect a running process from ptrace.

If a protector has gone so far that it is able to detect breakpoint instructions (and therefore make debugging more difficult) but is not able to protect itself from being traced by ptrace, then it is possible that it is still very vulnerable to ptrace based attacks, such as function hijacking and shared library injection. An attacker may not want to simply unpack a protected binary, but may aim to only change the binary's behavior. A good binary protector should try to protect the integrity of its control flow.

Imagine that an attacker is aware that a protected binary is calling the dlopen() function to load a specific shared library, and the attacker wants the process to load a trojaned shared library instead. The following steps could lead to an attacker compromising a protected binary by changing its control flow:

At this point, the attacker has just forced a protected binary to load a malicious shared library and has therefore completely compromised the security of the protected binary.

The Maya protector, as discussed earlier, is armed against such vulnerabilities thanks to a runtime engine that works as an active debugger, preventing any other process from attaching. If a protector can disable ptrace from attaching to the protected process, then that process is at much less risk of this type of runtime attack.

A vulnerability-based attack is a type of attack in which an attacker may be able to exploit an inherent weakness in the protected program, such as a stack-based buffer overflow, and subsequently change the execution flow to something of their choice.

This type of attack is often more difficult to carry out on a protected program, since it yields much less information about itself, and using a debugger to narrow down on the locations used in the memory by the exploit is potentially much more difficult to gain insight into. Nevertheless, this type of attack is very possible, and this is why the Maya protector enforces control flow integrity and read-only relocations to protect specifically against vulnerability exploitation attacks.

I am not aware whether any other protectors out there right now are using similar anti-exploitation techniques, but I can only surmise that they are out there.