The ptrace command can be used as an anti-debugging technique. Often when a hacker doesn't want their program to be easily debugged, they include certain anti-debugging techniques. One popular way in Linux is to use ptrace with the PTRACE_TRACEME request so that it traces the process of itself.
Remember that a process can only have one tracer at a time, so if a process is already being traced and a debugger tries to attach using ptrace, it says Operation not permitted. PTRACE_TRACEME can also be used to check whether your program is already being debugged. You can use the code in the following section to check this.
Let's take a look at a code snippet that will use ptrace to find out whether your program is already being traced:
if (ptrace(PTRACE_TRACEME, 0) < 0) {
printf("This process is being debugged!!!\n");
exit(1);
}The preceding code works because it should only fail if the program is already being traced. So, if ptrace returns an error value (less than 0) with PTRACE_TRACEME, you can be certain that a debugger is present and then exit the program.
As shown in Chapter 1, The Linux Environment and Its Tools, the LD_PRELOAD environment variable may be used to bypass this anti-debug measure by tricking the program into loading a fake ptrace command that does nothing but return 0, and will therefore not have any effect against debuggers. On the contrary, if a program uses the ptrace anti-debugging trick without using the libc ptrace wrapper—and instead creates its own wrapper—then the LD_PRELOAD trick will not work. This is because the program is not relying on any library for access to ptrace.
Here is an alternative way to use ptrace by writing your own wrapper for it. We will be using the x86_64 ptrace wrapper in this example:
#define SYS_PTRACE 101
long my_ptrace(long request, long pid, void *addr, void *data)
{
long ret;
__asm__ volatile(
"mov %0, %%rdi\n"
"mov %1, %%rsi\n"
"mov %2, %%rdx\n"
"mov %3, %%r10\n"
"mov $SYS_PTRACE, %%rax\n"
"syscall" : : "g"(request), "g"(pid),
"g"(addr), "g"(data));
__asm__ volatile("mov %%rax, %0" : "=r"(ret));
return ret;
}