Many of the standard user command shells, including bash, csh, tcsh, and ksh, can keep a history file . When the user issues commands, the text of each command and its arguments are stored in the history file for later re-execution. If you are trying to recreate activity performed on an account, possibly by some intruder, the contents of this file can be quite helpful when coupled with system log information. You must check the modification time on the file to be sure that it was in use during the time the suspicious activity occurred. If it was created and modified during the intruder’s activities, you should be able to determine the commands run, the programs compiled, and sometimes even the names of remote accounts or machines that might also be involved in the incident. Be sure of your target, however, because this is potentially a violation of privacy for the real user of this account.

Obviously, an aware intruder will delete the file before logging out. Thus, this mechanism may be of limited utility. However, there are two ways to increase your opportunity to get a useful file. The first way is to force the logout of the suspected intruder, perhaps by using a signal or shutting down the system. If a history file is being kept, this will leave the file on disk where it can be read. The second way to increase your chances of getting a usable file is to make a hard link to the existing history file and locate the link in a directory on the same disk that is normally inaccessible to the user (e.g., in a root-owned directory). Even if the intruder unlinks the file from the user’s directory, it can still be accessed through the extra link.

This technique can also come in handy if you suspect that an account is being used inappropriately. You can alter the system profile to create and keep a history file, if none was kept before. On some systems, you can even designate a named pipe (FIFO) as the history file, thus transmitting the material to a logging process in a manner that cannot be truncated or deleted.

Even if you were unable to preserve a copy of the history file, but one was created and then unlinked by the intruder, you can still gain some useful information if you act quickly enough. The first thing you must do is either take the system to single-user mode, or umount the disk with the suspect account (we recommend going to single-user mode). Then, you can use disk-examination tools to look at the records on the free list. When a file is deleted, the contents are not immediately overwritten. Instead, the data records are added back into the freelist on disk. If they are not reused yet (which is why you umount the disk or shut down the system), you can still read the contents.

Some user accounts are configured to make a copy of all outgoing mail in a file. If an intruder sends mail from a user account where this feature is set (or where you set it), this feature can provide you with potentially useful information. In at least one case we know of, a person stealing confidential information by using a coworker’s pirated password was exposed because of recorded email to his colleagues that he signed with his own name!

Some systems also record a log file of mail sent and received. This file may be kept per-user, or it may be part of the system-wide syslog audit trail. The contents of this log can be used to track the mail that has come in and left the system. If nothing else, we have found this information to be useful when a disk error (or human error) wipes out a whole set of mailboxes—the people listed in the mail log file can be contacted to resend their mail.

Each user account usually has an associated crontab, a file that contains commands to be run at specified intervals. The cron daemon processes these files and runs the commands at the appropriate times. It also generates logging messages that can reveal commands that may have been installed or exploited by the attacker.

Each user account can have several network configuration files that can be edited to provide shortcuts for commands or assert access rights. Sometimes, the information in these files will provide clues as to the activities of a malefactor. Examples include the .rhosts, .ssh/known_hosts, and .ssh/authorized_keys files for remote logins, and the .netrc file for FTP. Examine these files carefully for clues, but remember: the presence of information in one of these files may have been there prior to the incident, or it may have been planted to throw you off.