Table of Contents for

sed & awk, 2nd Edition

Contributed by Martin Weitzel

I am often preparing technical documents, especially for courses and training. In these documents, I often need to print source files of different kinds (C programs, awk programs, shell scripts, makefiles). The problem is that the sources often change with time and I want the most recent version when I print. I also want to avoid typos in print.

As I’m using troff for text processing, it should be easy to include the original sources into the text. But there are some characters (especially “” and “.” and “,” at the beginning of a line) that I must escape to prevent interpretation by troff.

I often want excerpts from sources rather than a complete file. I also need a mechanism for setting page breaks. Well, perhaps I’m being a perfectionist, but I don’t want to see a C function printed nearly complete on one page, but only the two last lines appear on the next. As I frequently change the documents, I cannot hunt for “nice” page breaks—this must be done automatically.

To solve these set of problems, I wrote a filter that preprocesses any source for inclusion as text in troff. This is the awk program I send with this letter. [He didn’t offer a name for it so it is here named readsource.]

The whole process can be further automated through makefiles. I include a preprocessed version of the sources into my troff documents, and I make the formatting dependent on these preprocessed files. These files again are dependent on their originals, so if I “make” the document to print it, the preprocessed sources will be checked to see if they are still current; otherwise they will be generated new from their originals.

My program contains a complete description in the form of comments. But as the description is more for me than for others, I’ll give you some more hints. Basically, the program simply guards some characters, e.g., “\” is turned into “\e” and “\&” is written before every line. Tabs may be expanded to spaces (there’s a switch for it), and you may even generate line numbers in front of every line (switch selectable). The format of these line numbers can be set through an environmental variable.

If you want only parts of a file to be processed, you can select these parts with two regular expressions (with another switch). You must specify the first line to be included and the first line not to be. I’ve found that this is often practical: If you want to show only a certain function of a C program, you can give the first line of the function definition and the first line of the next function definition. If the source is changed such that new functions are inserted between the two or the order is changed, the pattern matching will not work correctly. But this will accommodate the more frequently made, smaller changes in a program.

The final feature, getting the page breaks right, is a bit tricky. Here a technique has evolved that I call “here-you-may-break.” Those points are marked by a special kind of line (I use “/*!” in C programs and “#!” in awk, shell, makefiles, etc.). How the points are marked doesn’t matter too much, you may have your own conventions, but it must be possible to give a regular expression that matches exactly this kind of line and no others (e.g., if your sources are written so that a page break is acceptable wherever you have an empty line, you can specify this very easily, as all you need is the regular expression for empty lines).

Before all the marked lines, a special sequence will be inserted which again is given by an environmental variable. With troff, I use the technique of opening a “display” (.DS) before I include such preprocessed text, and inserting a close (.DE) and new open (.DS) display wherever I would accept a page break. After this, troff does the work of gathering as many lines as fit onto the current page. I suppose that suitable techniques for other text processors exist.

#! /bin/sh
# Copyright 1990 by EDV-Beratung Martin Weitzel, D-6100 Darmstadt
# ==================================================================
# PROJECT:	Printing Tools
# SH-SCRIPT:	Source to Troff Pre-Formatter
# ==================================================================

#!
# ------------------------------------------------------------------
# This programm is a tool to preformat source files, so that they
# can be included (.so) within nroff/troff-input. Problems when
# including arbitrary files within nroff/troff-input occur on lines,
# starting with dot (.) or an apostrophe ('), or with the respective
# chars, if these are changed, furthermore from embedded backslashes.
# While changing the source so that none of the above will cause
# any problems, some other useful things can be done, including
# line numbering and selecting interesting parts.
# ------------------------------------------------------------------
#!
  USAGE="$0 [-x d] [-n] [-b pat] [-e pat] [-p pat] [file ...]"
#
# SYNOPSIS:
# The following options are supported:
#	-x d	expand tabs to "d" spaces
#	-n 	number source lines (see also: NFMT)
#	-b pat	start output on a line containing "pat",
#		including this line (Default: from beginning)
#	-e pat	end output on a line containing "pat"
#		excluding this line (Default: upto end)
#	-p pat	before lines containing "pat", page breaks
#		may occur (Default: no page breaks)
# "pat" may be an "extended regular expression" as supported by awk.
# The following variables from the environment are used:
#	NFMT	specify format for line numbers (Default: see below)
#	PBRK	string, to mark page breaks. (Default: see below)
#!
# PREREQUISITES:
# Common UNIX-Environment, including awk.
#
# CAVEATS:
# "pat"s are not checked before they are used (processing may have
# started, before problems are detected).
# "NFMT" must contain exactly one %d-format specifier, if -n
# option is used.
# In "NFMT" and "PBRK", embedded double quotes must be guarded with
# a leading backslash.
# In "pat"s, "NFMT" and "PBRK" embedded TABs and NLs must be written
# as \t and \n. Backslashes that should "go thru" to the output as
# such, should be doubled. (The latter is only *required* in a few
# special cases, but it does no harm the other cases).
# 
#!
# BUGS:
# Slow - but may serve as prototype for a faster implementation.
# (Hint: Guarding backslashes the way it is done by now is very
# expensive and could also be done using sed 's/\\/\\e/g', but tab
# expansion would be much harder then, because I can't imagine how
# to do it with sed. If you have no need for tab expansion, you may
# change the program. Another option would be to use gsub( ), which
# would limit the program to environments with nawk.)
# 
# Others bugs may be, please mail me.
#!
# AUTHOR:	Martin Weitzel, D-6100 DA (martin@mwtech.UUCP)
#
# RELEASED: 	25. Nov 1989, Version 1.00
# ------------------------------------------------------------------

#! CSOPT
# ------------------------------------------------------------------
# 	check/set options
# ------------------------------------------------------------------

xtabs=0 nfmt= bpat= epat= ppat=
for p
do
case $sk in
1) shift; sk=0; continue
esac
case $p in
-x)	shift;
	case $1 in
	[1-9]|1[0-9]) xtabs=$1; sk=1;;
	*) { >&2 echo "$0: bad value for option -x: $1"; exit 1; }
	esac
	;;
-n)	nfmt="${NFMT:-<%03d>\•}"; shift ;;
-b)	shift; bpat=$1; sk=1 ;;
-e)	shift; epat=$1; sk=1 ;;
-p)	shift; ppat=$1; sk=1 ;;
--)	shift; break ;;
*)	break
esac
done

#! MPROC
# ------------------------------------------------------------------
# 	now the "real work"
# ------------------------------------------------------------------

awk '
#. prepare for tab-expansion, page-breaks and selection
BEGIN {
	if (xt = '$xtabs') while (length(sp) < xt) sp = sp " ";
	PBRK = "'"${PBRK-'.DE\n.DS\n'}"'"
	'${bpat:+' skip = 1; '}'
} #! limit selection range
{
	'${epat:+' if (!skip && $0 ~ /'"$epat"'/) skip = 1; '}'
	'${bpat:+' if (skip && $0 ~ /'"$bpat"'/) skip = 0; '}'
	if (skip) next;
}
#! process one line of input as required
{
	line = ""; ll = 0;
	for (i = 1; i <= length; i++) {
		c = substr($0, i, 1);
		if (xt && c == "\t") {
			# expand tabs
			nsp = 8 - ll % xt;
			line = line substr(sp, 1, nsp);
			ll += nsp;
		}
		else {
			if (c == "\\") c = "\\e";
			line = line c;
			ll++;
		}
	}
}
#! finally print this line
{
	'${ppat:+' if ($0 ~ /'"$ppat"'/) printf("%s", PBRK); '}'
	'${nfmt:+' printf("'"$nfmt"'", NR) '}'
	printf("\\&%s\n", line);
}
' $*

For an example of how it works, we ran readsource to extract a part of its own program.

$ readsource -x 3 -b "process one line" -e "finally print" readsource
\&#! process one line of input as required
\&{
\&   line = ""; ll = 0;
\&   for (i = 1; i <= length; i++) {
\&      c = substr($0, i, 1);
\&      if (xt && c == "\\et") {
\&         # expand tabs
\&         nsp = 8 - ll % xt;
\&         line = line substr(sp, 1, nsp);
\&         ll += nsp;
\&      }
\&      else {
\&         if (c == "\\e\\e") c = "\\e\\ee";
\&         line = line c;
\&         ll++;
\&      }
\&   }
\&}

$ timex readsource -x 3 readsource > /dev/null
real        1.56
user        1.22
sys         0.20

#! process one line of input as required
{
        if ( xt && $0 ~ "\t" )
                gsub(/\t/, sp)
        if ($0 ~ "\\")
                gsub(/\\/, "\\e")
}

$ timex readsource.2 -x 3 readsource > /dev/null
real        0.44
user        0.10
sys         0.22

#! process one line of input as required
{
        if ( xt && index($0, "\t") > 0 )
                gsub(/\t/, sp)
        if (index($0, "\\") > 0)
                gsub(/\\/, "\\e")
}