The remainder of this chapter describes several different integrity management systems. All of these systems perform more or less the same function: they examine files on a computer’s disk drive to determine whether the files have been changed in any significant way.

Although there are many reasons that you might want to examine the integrity of your system’s files, one of the most common is to determine what has changed after a computer has been attacked, broken into, and compromised.

If you suspect that a system has been compromised, there are many ways that you can examine its files for evidence of this fact:

Clearly, the most thorough way to examine the suspect system is the first technique. In practice, the third and fourth techniques are the most common. And to all of the people who have simply treated the symptoms of a compromised system, rather than taken a more thorough approach, we have one question:

If an attacker truly compromises your computer system, all bets are off. Nothing should be trusted. It is possible that the attacker has done nothing to affect the integrity of critical system programs such as login, ps, ls, and netstat. On the other hand, it is possible that the attacker has replaced all of these programs with modified programs that contain Trojan horses and back doors, and then modified your computer’s kernel so that integrity-checking tools cannot tell the difference!^[303]

Sadly, it takes a lot of extra time to do things the right way. It’s much easier to log into a suspect computer and run a copy of Tripwire or AIDE to check for modifications—rather than going to the trouble of booting a kernel from CD-ROM that is known to be good. That’s why many people—the authors included—will occasionally run automated tools on possibly compromised machines before breaking out the CD-ROMs and the screwdrivers. But beware: if it looks like nothing is wrong, everything could be wrong.

The most direct and assured method of detecting changes to data is to keep a copy of the unaltered data and do a byte-by-byte comparison when needed. If there is a difference, this indicates not only that a change occurred, but explains what that change involved. There is no more reliable and complete method of detecting changes.

Comparison copies, however, are unwieldy. They require that you keep copies of every file of interest. Not only does such a method require twice as much storage as the original files, it also may involve a violation of the licenses or copyrights of the files. (Copyright law allows one copy for archival purposes, and your distribution media is that one copy.)^[304] To use a comparison copy means that both the original and the copy must be read through, byte by byte, each time a check is made. And, of course, the comparison copy needs to be saved in a protected location.

Even with these drawbacks, comparison copies have a particular benefit: if you discover an unauthorized change, you can simply replace the altered version with the saved comparison copy, thus restoring the system to normal. These copies can be made locally, at remote sites, or over the network, as we describe in the following sections.

Saving an extra copy of each critical file and performing a byte-by-byte comparison can be unduly expensive. It requires substantial disk space to store the copies. Furthermore, if the comparison is performed over the network, either via rdist or NFS, it will involve substantial disk and network overhead each time the comparisons are made.

A more efficient approach would be to store a summary of important characteristics of each file and directory. When the time comes to do a comparison, the characteristics are regenerated and compared with the saved information. If the characteristics are comprehensive and smaller than the file contents (on average), then this method is clearly a more efficient way of doing the comparison.

Furthermore, this approach can capture changes that a simple comparison copy cannot: comparison copies detect changes in the contents of files, but do little to detect changes in metadata such as file owners or protection modes. It is this data—the data normally kept in the inodes of files and directories—that is sometimes more important than the data within the files themselves. For instance, changes in owner or protection bits may result in disaster if they occur to the wrong file or directory.

Thus, we would like to compare the values in the inodes of critical files and directories with a database of comparison values. The values we wish to compare and monitor for critical changes are owner, group, and protection modes. We also wish to monitor the mtime (modification time) and the file size to determine if the file contents change in an unauthorized or unexpected manner. We may also wish to monitor the link count, inode number, and ctime as additional indicators of change. All of this material can be listed with the ls command.

#!/bin/sh

cat /usr/adm/filelist | xargs ls -ild > /tmp/now
diff -b /usr/adm/savelist /tmp/now

#!/bin/sh

find `cat /usr/adm/filelist` -ls > /tmp/now
diff -b /usr/adm/savelist /tmp/now

#!/bin/ksh

typeset pdir

function getdir      # Gets the real, physical pathname
{
   if [[ $1 != /* ]]
   then
      print -u2 "$1 is not an absolute pathname"
      return 1
   elif cd "${1%/*}"
   then
      pdir=$(pwd -P)
      cd ~-
   else
      print -u2 "Unable to attach to directory of $1"
      return 2
   fi
   return 0
}

cd /
print /     # Ensure we always have the root directory included

while read name
do
   getdir $name || continue
   while [[ -n $pdir ]]
   do
      print $pdir
      pdir=${pdir%/*}
   done
done | sort -u

Unfortunately, the approach we described for monitoring files can be defeated with a little effort. Files can be modified in such a way that the information we monitor will not disclose the change. For instance, a file might be modified by writing to the raw disk device after the appropriate block is known. As the modification did not go through the filesystem, none of the information in the inodes will be altered.

An attacker could also surreptitiously alter a file by setting the system clock back to the time of the last legitimate change, making the edits, and then setting the clock forward again. If this is done quickly enough, no one will notice the change. Furthermore, all the times on the file (including the ctime) will be set to the “correct” values. Several so-called “rootkits” in widespread use on the Internet actually take this approach. It is easier and safer than writing to the raw device. It is also more portable.

Thus, we need to have some stronger approach in place to check the contents of files against a known good value. Obviously, we could use comparison copies, but we have already noted that they are expensive. A second approach would be to create a signature of the file’s contents to determine if a change occurred.

The first, naive approach using such a signature might involve the use of a standard CRC checksum, as implemented by the sum command. CRC polynomials are often used to detect changes in message transmissions, so they could logically be applied here. However, this application would be a mistake.

CRC checksums are designed to detect random bit changes, not purposeful alterations. As such, CRC checksums are good at finding a few bits changed at random. However, because they are generated with well-known polynomials, an attacker can alter the input file to generate an arbitrary CRC polynomial after an edit. In fact, some of the same attacker toolkits that allow files to be changed without altering the time also contain code to set the file contents to generate the same sum outputs for the altered file as for the original. These tools have been generally available since at least 1992.

To generate a checksum that cannot be easily spoofed, we need to use a stronger mechanism, such as the message digests described in Section 7.4. These are also dependent on the contents of the file, but they are too difficult to spoof after changes have been made.

If we had a program to generate the MD5 checksum of a file, we might alter our checklist script to be:

#!/bin/sh

find 'cat /usr/adm/filelist' -ls -type f -exec md5 {} \; > /tmp/now
diff -b /usr/adm/savelist /tmp/now

Both the mtree command and the Tripwire system (discussed later in this chapter) employ cryptographic checksums for this purpose.