Table of Contents for

sed & awk, 2nd Edition

Before you begin debugging a program, make a copy of it. This is extremely important. To debug an awk script, you have to change it. These modifications may point you to the error but many changes will have no effect or may introduce new problems. It’s good to be able to restore changes that you make. However, it is bothersome to restore each change that you make, so I like to continue making changes until I have found the problem. When I know what it is, I go back to the original and make the change. In effect, that restores all the other inconsequential changes that were made in the copy.

It is also helpful to view the process of creating a program as a series of stages. Look at a core set of features as a single stage. Once you have implemented these features and tested them, make a copy of the program before going to the next stage to develop new features. That way, you can always return to the previous stage if you have problems with the code that you add.

We would recommend that you formalize this process, and go so far as to use a source code management system, such as SCCS (Source Code Control System), RCS (Revision Control System), or CVS (Concurrent Versioning System, which is compatible with RCS). The latter two are freely available from any GNU FTP mirror site.

What is difficult in debugging awk is that you don’t always know what is happening during the course of the program. You can inspect the input and the output, but there is no way to stop the program in mid-course and examine its state. Thus, it is difficult to know which part of the program is causing a problem.

A common problem is determining when or where in the program the assignment of a variable takes place. The first method of attack is to use the print statement to print the value of the variable at various points in the program. For instance, it is common to use a variable as a flag to determine that a certain condition has occurred. At the beginning of the program, the flag might be set to 0. At one or more points in the program, the value of this flag might be set to 1. The problem is to find where the change actually occurs. If you want to check the flag at a particular part of the program, use print statements before and after the assignment. For instance:

print flag, "before"
if (! $1) {
	.
	.
	.
	flag = 1
}
print flag, "after"

If you are unsure about the result of a substitution command or any function, print the string before and after the function is called:

print $2
sub(/ *\(/, "(", $2)
print $2

The value of printing the value before the substitution command is to make sure that the command sees the value that you think should be there. A previous command might have changed that variable. The problem may turn out to be that the format of the input record is not as you thought. Checking the input carefully is a very important step in debugging. In particular, use print statements to verify that the sequence of fields is as you expect. When you find that input is causing the problem, you can either fix the input or write new code to accommodate it.

The more modular a script is—that is, the more it can be broken down into separate parts—the easier it is to test and debug the program. One of the advantages of writing functions is that you can isolate what is going on inside the function and test it without affecting other parts of the program. You can omit an entire action and see what happens.

If a program has a number of branching constructs, you might find that an input line falls through one of branches. Test that the input reaches part of a program. For instance, when debugging the masterindex program, described in Chapter 12, we wanted to know if an entry containing the word “retrieving” was being handled in a particular part of the program. We inserted the following line in the part of the program where we thought it should be encountered:

if ($0 ~ /retrieving/) print ">> retrieving" > "/dev/tty"

When the program runs, if it encounters the string “retrieving,” it will print the message. (“>>” is used as a pair of characters that will instantly call attention to the output; “!!” is also a good one.)

Sometimes you might not be sure which of several print statements are causing a problem. Insert identifiers into the print statement that will alert you to the print statement being executed. In the following example, we simply use the variable name to identify what is printed with a label:

if (PRIMARY)
	print (">>PRIMARY:", PRIMARY)
else
	if (SECONDARY)
		print (">>SECONDARY:", SECONDARY)
	else
		print (">>TERTIARY:", TERTIARY)

This technique is also useful for investigating whether or not parts of the program are executed at all. Some programs get to be like remodeled homes: a room is added here, a wall is taken down there. Trying to understand the basic structure can be difficult. You might wonder if each of the parts is truly needed or indeed if it is ever executed at all.

If an awk program is part of a pipeline of several programs, even other awk programs, you can use the tee command to redirect output to a file, while also piping the output to the next command. For instance, look at the shell script for running the masterindex program, as shown in Chapter 12:

$INDEXDIR/input.idx $FILES |
sort -bdf -t:  +0 -1 +1 -2 +3 -4 +2n -3n | uniq |
$INDEXDIR/pagenums.idx | tee page.tmp |
$INDEXDIR/combine.idx |
$INDEXDIR/format.idx

By adding “tee page.tmp”, we are able to capture the output of the pagenums.idx program in a file named page.tmp. The same output is also piped to combine.idx.

Another technique is simply commenting out a series of lines that may be causing problems to see whether they really are. We recommend developing a consistent two-character symbol such as “#%” to comment out lines temporarily. Then you will notice them on subsequent editing and remember to deal with them. It also becomes easier to remove the symbols and restore the lines with a single editing command that does not affect program comments:

#% if ( thisFails )
	print "I give up"

Using the comment here eliminates the conditional, so the print statement is executed unconditionally.

When all else fails, arm yourself with your editor’s delete command and begin deleting portions of the program until the error disappears. Of course, make a copy of the program and delete lines from the temporary copy. This is a very crude technique, but an effective one to use before giving up altogether or starting over from scratch. It is sometimes the only way to discover what is wrong when the only result you get is that the program dumps core. The idea is the same as above, to isolate the problem code. Remove a function, for instance, or a for loop to see if it is the cause of the problem. Be sure to cut out complete units: for instance, all the statements within braces and the matching braces. If the problem persists—the program continues to break—then cut out another large section of the program. Sooner or later, you will find the part that is causing the problem.

You can use “slash and burn” to learn how a program works. First, run the original program on sample input, saving the output. Begin by removing a part of the program that you don’t understand. Then run the modified program on sample input and compare the output to the original. Look to see what changed.

There are all types of input errors and inconsistencies that will turn up bugs in your script. You probably didn’t consider that user errors will be pointed to as problems with your program. Therefore, it is a good idea to surround your core program with “defensive” procedures designed to trap inconsistent input records and prevent the program from failing unexpectedly. For instance, you might want to verify each input record before processing it, making sure that the proper number of fields exist or that the kind of data that you expect is found in a particular field.

Another aspect of incorporating defensive techniques is error handling. In other words, what do you want to have happen once the program detects an error? While in some cases you can have the program continue, in other cases it may be preferable that the program print an error message and/or halt.

It is also appropriate to recognize that awk scripts are typically confined to the realm of quick fixes, programs that solve a particular problem rather than solving a class of problems encountered by many different users. Because of the nature of these programs, it is not really necessary that they be professional quality. Thus, it is not necessary to write 100% user-proof programs. For one thing, defensive programming is quite time-consuming and frequently tedious. Secondly, as amateurs, we are at liberty to write programs that perform the way we expect them to; a professional has to write for an audience and must account for their expectations. In brief, if you are writing the script for others to use, consider how it may be used and what problems its users may encounter before considering the program complete. If not, maybe the fact that the script works—even for a very narrow set of circumstances—is good enough and all there is time for.