Controlling and Examining Processes

In addition to ps and kill, Unix supports a large number of lesser known tools for examining and controlling running processes. These commands can be useful for programmers and system administrators; they are also very helpful in analyzing the processes of an attacker during and after a break-in. Some of the ways you can examine or control processes include the following:

gdb: You can attach to the running process with a debugger such as gdb.
gcore: You can use the gcore command to dump the process memory map.
lsof: You can use the lsof program to list the open files in use by the program.
/proc: You can examine the process directly using the /proc process filesystem.
pstree: You can see a tree of all processes with the pstree command.

Not all of these tools are available on every version of Unix.

Strictly speaking, many of these tools will work with processes that are either running or stopped. However, if you have a rogue process on your system, you may wish to stop it with the SIGSTOP signal before examining it.

Note

One reason to be familiar with these tools is that many attackers will modify a penetrated system in such a way that the system ps command will no longer display processes belonging to the attacker. These modifications are most often done with programs that are collectively known as rootkits.

Once a system has been modified with a rootkit, it can be very difficult to detect the continued presence of an attacker. However, few rootkits modify such programs as lsof or pstree. Thus, if these tools show that a process is present on your system but the ps command does not, that is a good indication that your system has been compromised.

gdb: Controlling a Process

On many systems, you can use the gdb command to “attach” to a running process. If the process is running an executable that was linked with a full symbol table, you will be able to use the debugger to examine the process’s variables and detailed call stack. Even if you do not have an executable that was linked for debugging, you may be able to use the debugger to determine what the process is doing.

adb and dbx are additional debuggers you can use to control processes.

gcore: Dumping Core

Under many versions of Unix, you can use the gcore program to generate a core file of a running process. A core file is a specially formatted image of the memory being used by the process at the time the signal was caught.

Tip

Some versions of Unix name core files core.####, in which #### is the PID of the process that generated the core file, or name.core, in which name is the name of the program’s executable. Others simply use the name core.

Once you have a core file, you can examine it with adb (a debugger), dbx (another debugger), or gdb (yet another debugger). By examining the core file with a debugger, you can see which routines were executed, register values, and more. If you simply want to get an idea of what the process was doing, you can run strings (a program that finds printable strings in a binary file) over the core image to see which files it was referencing. If the process was running a shell such as sh or csh, the strings command will display the shell’s history.

Programs that you run may also dump core if they receive one of the signals that causes a core dump. On systems without a gcore program, you can send a SIGEMT or SIGSYS signal to cause the program to dump core. This method will work only if the process is currently in a directory where it can write, it has not redefined the action to receive the signal, and the core will not be larger than the core file limits imposed for the process’s UID. If you use this approach, you will also be faced with the problem of finding where the process left the core file!

Warning

Core files are big! You can fill your disk with a core file—be sure to look at the memory size of a process via the ps command before you try to get its core image.

lsof: Examining a Process

The List of Open Files (lsof) command is now provided as a standard part of many Unix systems; it is available as a free download for still more systems.^[373]

As the name implies, this command examines the kernel’s table of file descriptors associated with each process and displays the name of each file that is currently opened. In addition to giving the name of each file being currently referenced, lsof reveals the name of the executable currently being run by the process and the filenames of all mapped-in shared libraries. Besides this information, current versions of lsof can report open TCP/IP connections, as well as TCP and UDP sockets that are being listened to.

When lsof is run by a user, the program restricts its output to processes that are owned by that user. When lsof is run by the superuser, the program displays output for all processes on the system.

Here is an example of the output from the lsof command:

[simsong@r2 ~] 304 % lsof
COMMAND   PID    USER   FD   TYPE     DEVICE   SIZE/OFF     NODE NAME
tcsh    81776 simsong  cwd   VDIR       13,2      15360 12657945 /usr/home/simsong
tcsh    81776 simsong  rtd   VDIR 116,131072       1024        2 /
tcsh    81776 simsong  txt   VREG 116,131072     638988     6355 /bin/tcsh
tcsh    81776 simsong   15u  VCHR        5,3   0t138415     7898 /dev/ttyp3
tcsh    81776 simsong   16u  VCHR        5,3   0t138415     7898 /dev/ttyp3
tcsh    81776 simsong   17u  VCHR        5,3   0t138415     7898 /dev/ttyp3
tcsh    81776 simsong   18u  VCHR        5,3   0t138415     7898 /dev/ttyp3
tcsh    81776 simsong   19u  VCHR        5,3   0t138415     7898 /dev/ttyp3
lsof    81991 simsong  cwd   VDIR       13,2      15360 12657945 /usr/home/simsong
lsof    81991 simsong  rtd   VDIR 116,131072       1024        2 /
lsof    81991 simsong  txt   VREG       13,2     106848  7618686 /usr/local/sbin/lsof
lsof    81991 simsong  txt   VREG       13,2      76752  1984040 /usr/libexec/ld-elf.
so.1
lsof    81991 simsong  txt   VREG       13,2      19232   634990 /usr/lib/libkvm.so.2
lsof    81991 simsong  txt   VREG       13,2     573888   634976 /usr/lib/libc.so.4
lsof    81991 simsong    0u  VCHR        5,3   0t138415     7898 /dev/ttyp3
lsof    81991 simsong    1u  VCHR        5,3   0t138415     7898 /dev/ttyp3
lsof    81991 simsong    2u  VCHR        5,3   0t138415     7898 /dev/ttyp3
lsof    81991 simsong    3r  VCHR        2,0        0t0     6880 /dev/mem
lsof    81991 simsong    4r  VCHR        2,1 0xc28649c0     6872 /dev/kmem
[simsong@r2 ~] 305 %

The lsof program has too many options to list here. There are significant security issues that arise from its installation and use—specifically, lsof lists the names of files throughout the filesystem, and this information is cached in a file that is located in the home directory of the person who runs the lsof command. The lsof command can be compiled and installed with various options that minimize the privacy exposure that can result from these cache files. For details, consult the lsof documentation.

/proc: Examining a Process Directly

/proc is the process filesystem. It allows user programs to access aspects of a process through the filesystem interface in a relatively transparent and straightforward fashion, without having to open up the kernel’s memory and wade through memory structures. It also allows direct access to the memory space of other processes, which is otherwise impossible or very difficult.

pstree: Viewing the Process Tree

Every Unix process has an associated parent process. Normally, this information is displayed as a form similar to the display of the PPID field output by the ps command. The pstree command uses this information to draw a graph of all of the processes currently running.

During a break-in, the process tree can be very useful for understanding which processes were launched by the attacker and which are innocent processes that happen to be running on the same system.

With the -u option, the pstree command will show UID transitions—that is, when one process has a child that is executing under a separate UID. Another useful option is -a, which shows the entire command line that was executed. For a list of all the options, see the documentation.

Here is an example of the output of the pstree program:

% pstree -u
init-+-arpwatch
     |-cron
     |-dhcpd
     |-gdomap(nobody)
     |-8*[getty]
     |-httpd---11*[httpd(http)]
     |-inetd-+-imapd(simsong)
     |       `-sslwrap
     |-lpd
     |-master-+-2*[bounce(postfix)]
     |        |-cleanup(postfix)
     |        |-flush(postfix)
     |        |-local(postfix)
     |        |-pickup(postfix)
     |        |-qmgr(postfix)
     |        |-2*[smtp(postfix)]
     |        |-smtpd(postfix)
     |        |-tlsmgr(postfix)
     |        `-trivial-rewrite(postfix)
     |-mountd
     |-moused
     |-named
     |-nfsd---4*[nfsd]
     |-4*[nfsiod]
     |-nmbd---nmbd
     |-ntpd
     |-portmap(daemon)
     |-pwcheck
     |-rpc.statd
     |-rwhod(daemon)
     |-setiathome(nobody)
     |-slapd
     |-smbd---smbd(beth)
     |-snmpd
     |-sshd-+-sshd---tcsh(simsong)---pstree
     |      |-sshd---tcsh(simsong)---tcsh(root)
     |      `-sshd---tcsh(simsong)
     |-syslogd
     `-usbd

The boldfaced line near the end of this output shows that init (executing as root) spawned an sshd process (executing as root). This process forks a child (still root-owned in this case) for each incoming connection. When simsong logged into this sshd connection, it started up a shell (tcsh) owned by simsong, and simsong has apparently managed to start a root-privileged tcsh shell (perhaps with /bin/su, but if you don’t expect simsong to have the root password, this is cause for concern)!

^[373] lsof Version 4.64 is available from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/ for at least the following Unix systems: AIX 4.3.[23], 5L, and 5.1; Apple Darwin 1.[23] and 1.4 for Power Macintosh systems; BSDI BSD/OS 4.1 for Intel-based systems; DEC OSF/1, Digital UNIX, Tru64 UNIX 4.0, and 5.[01]; FreeBSD 4.[23456] and 5.0 for Intel-based systems; HP-UX 11.00 and 11.11; Linux 2.1.72 and above for Intel-based systems; NetBSD 1.[456] for Alpha-, Intel-, and SPARC-based systems; NEXTSTEP 3.[13] for NEXTSTEP architectures; OpenBSD 2.[89] and 3.[01] for Intel-based systems; OPENSTEP 4.x; Caldera OpenUNIX 8; SCO OpenServer Release 5.0.[46] for Intel-based systems; SCO UnixWare 7.1.1 for Intel-based systems; Solaris 2.6, 7, 8, and 9 BETA-Refresh.

Previous Chapter

Signals

Next Chapter

Starting Up Unix and Logging In