One of the best examples of how a single line of code in a program can result in the compromise of thousands of machines dates back to the pre-dawn of the commercial Internet. The year was 1988, and a graduate student at Cornell University had discovered several significant security flaws in versions of Unix that were widely used on the Internet. Using his knowledge, the student created a program (known as a worm) that would find vulnerable computers, exploit one of these flaws, transfer a copy of itself to the compromised system, and then repeat the process. The program infected between 2,000 and 6,000 computers within hours of being released. While that does not seem like a lot of machines today, in 1988 it represented a substantial percentage of the academic and commercial mail servers on the Internet. The Internet was effectively shut down for two days following the worm’s release.

Although the worm used several techniques for compromising systems, the most effective attack in its arsenal was a buffer overflow attack directed against the Unix fingerd daemon.

The original fingerd program contained these lines of code:

                char line[512];
...
                line[0] = '\0';
                gets(line);

Because the gets( ) function does not check the length of the line read, a program that supplied more than 512 bytes of valid data would overrun the memory allocated to the line[] array and, ultimately, corrupt the program’s stack frame. The worm contained code that used the stack overflow to cause the fingerd program to execute a shell; because at the time it was standard practice to run fingerd as the superuser, this shell inherited superuser access to the server computer. fingerd didn’t need to run as superuser, but it was spawned as a root process during the system startup and never switched to a different user ID.^[236] Because fingerd’s standard input and standard output file descriptors were connected to the TCP socket, the remote process that caused the overflow was given complete, interactive control of the system.

The fix for the fingerd program was simple: replace the gets( ) function with the fgets( ) function. Whereas gets( ) takes one parameter, the buffer, the fgets( ) function takes three arguments: the buffer, the size of the buffer, and the file handle from which to fetch the data:

fgets(line,sizeof(line),stdin);

When the original fingerd program was written, it was common practice among many programers to use gets( ) instead of fgets( )—probably because using gets( ) required typing fewer characters each time. Nevertheless, because of the way that the C programming language and the Standard IO library were designed, any program that used gets( ) to fill a buffer on the stack potentially had—and still has—this vulnerability.

Although it seems like ancient history now, this story continues to illustrate many important lessons:

Interestingly enough, the fallible human component of secure programming is illustrated by the same example. Shortly after the problem with the gets( ) subroutine was exposed, the Berkeley programming group went through all of its code and eliminated every similar use of the gets( ) call in a network server. Most vendors did the same with their code. Several people, including Spafford in his paper analyzing the operations and effects of the worm, publicly warned that uses of other library calls that wrote to buffers without bounds checks also needed to be examined. These included calls to the sprintf( ) routine, and byte-copy routines such as strcpy ( ) . However, those admonitions were not heeded.

In late 1995, as we were finishing the second edition of this book, a new security vulnerability in several versions of Unix was widely publicized. It was based on buffer overruns in the syslog library routine. An attacker could carefully craft an argument to a network daemon such that, when an attempt was made to log it using syslog, the message overran the buffer and compromised the system in a manner hauntingly similar to the fingerd problem. After seven years, a close cousin to the fingerd bug was discovered. What underlying library calls contribute to the problem? The sprintf( ) library call does, and so do byte-copy routines such as strcpy( ).

In the summer of 2002, as we were working on the third edition of this book, not one but four separate overflow vulnerabilities were found in the popular OpenSSL security library, based on effectively the same vulnerability. In use on more than a million Internet servers, this SSL library is the basis of the SSL offering used by the Apache web server and all Unix SSL-wrapped mail services.

While many Unix security bugs are the result of poor programming tools and methods, even more regrettable is the failure to learn from old mistakes, and the failure to redesign the underlying operating system or programming languages so that this broad class of attacks will no longer be effective.^[238]

In December 1990, the Communications of the ACM published an article by Miller, Fredrickson, and So entitled “An Empirical Study of the Reliability of Unix Utilities” (Volume 33, issue 12, pp. 32-44). The paper started almost as a joke: a researcher was logged into a Unix computer from home, and the programs he was running kept crashing because of line noise from a poor modem connection. Eventually, Barton Miller, a professor at the University of Wisconsin, decided to subject the Unix utility programs from a variety of different vendors to a selection of random inputs and monitor the results.^[239]