Table of Contents for

21st Century C

A shell largely behaves like a macro language, wherein certain blobs of text get replaced with other blobs of text. These are called expansions in the shell world, and there are many types: this section touches on variable substitution, command substitution, a smattering of history substitution, and will give examples touching on tilde expansion and arithmetic substitution for quick desk calculator math. I leave you to read your shell’s manual on alias expansion, brace expansion, parameter expansion, word splitting, pathname expansion, and glob expansion.

Variables are a simple expansion. If you set a variable like

onething="another thing"

on the command line, then when you later type:

echo $onething

then another thing will print to screen.

Your shell will require that there be no spaces on either side of the =, which will annoy you at some point.

When one program starts a new program (in POSIX C, when the fork() system call is used), a copy of all environment variables is sent to the child program. Of course, this is how your shell works: when you enter a command, the shell forks a new process and sends all the environment variables to the child.

Environment variables, however, are a subset of the shell variables. When you make an assignment like the previous one, you have set a variable for the shell to use; when you:

export onething="another thing"

then that variable is available for use in the shell, and its export attribute is set. Once the export attribute is set, you can still change the variable’s value.

For our next expansion, how about the backtick, `, which is not the more vertical-looking single tick '.

onething="another thing"
echo "$onething"
echo '$onething'

another thing
$onething

The backtick replaces the command you give with its output, doing so macro-style, where the command text is replaced in place with the output text.

Example 3-1 presents a script that counts lines of C code by how many lines have a ;, ), or } on them. Given that lines of source code is a lousy metric for most purposes anyway, this is as good a means as any, and has the bonus of being one line of shell code.

 # Count lines with a ;, ), or }, and let that count be named Lines.
Lines=`grep '[;)}]' *.c | wc -l`

 # Now count how many lines there are in a directory listing; name it Files.
Files=`ls *.c |wc -l`

echo files=$Files and lines=$Lines

 # Arithmetic expansion is a double-paren.
 # In bash, the remainder is truncated; more on this later.
echo lines/file = $(($Lines/$Files))

 # Or, use those variables in a here script.
 # By setting scale=3, answers are printed to 3 decimal places.
bc << ---
scale=3
$Lines/$Files
---

You can run the shell script via . linecount.sh. The dot is the POSIX-standard command to source a script. Your shell probably also lets you do this via the nonstandard but much more comprehensible source linecount.sh.

Let’s get to some proper programming, with if statements and for loops.

But first, some caveats and annoyances about shell scripting:

Some people don’t see these details as much of an issue, and ♥ the shell. Me, I write shell scripts to automate what I would type at the command line, and once things get complex enough that there are functions calling other functions, I take the time to switch to Perl, Python, awk, or whatever is appropriate.

My vote for greatest bang for the buck from having a programming language that you can type directly onto the command line goes to running the same command on several files. Let’s back up every .c file the old fashioned way, by copying it to a new file with a name ending in .bkup:

for file in *.c;
do
 cp $file ${file}.bkup;
done

You see where the semicolon is: at the end of the list of files the loop will use, on the same line as the for statement. I’m pointing this out because when cramming this onto one line, as in:

for file in *.c; do cp $file ${file}.bkup; done

I always forget that the order is ; do and not do ;.

The for loop is useful for dealing with a sequence of N runs of a program. By way of a simple example, benford.sh searches C code for numbers beginning with a certain digit (i.e., head of the line or a nondigit followed by the digit we are looking for), and writes each line that has the given number to a file, as shown in Example 3-2:

for i in 0 1 2 3 4 5 6 7 8 9; do grep -E '(^|[^0-9.])'$i *.c > lines_with_${i}; done
wc -l lines_with*         //A rough histogram of your digit usage.
        

Testing against Benford’s law is left as an exercise for the reader.

The curly braces in ${i} are there to distinguish what is the variable name and what is subsequent text; you don’t need it here, but you would if you wanted a filename like ${i}lines.

You probably have the seq command installed on your machine—it’s BSD/GNU standard but not POSIX standard. Then we can use backticks to generate a sequence:

for i in `seq 0 9`; do grep -E '(^|[^0-9.])'$i *.c > lines_with_${i}; done

Running your program a thousand times is now trivial:

for i in `seq 1 1000`; do ./run_program > ${i}.out; done

#or append all output to a single file:
for i in `seq 1 1000`; do
  echo output for run $i: >> run_outputs
  ./run_program >> run_outputs
done

Now let’s say that your program relies on a data set that has to be read in from a text file to a database. You only want to do the read-in once, or in pseudocode: if (database exists) then (do nothing), else (generate database from text).

On the command line, you would use test, a versatile command typically built into the shell. To try it, run a quick ls, get a filename you know is there, and use test to check that the file exists like this:

test -e a_file_i_know
echo $?

By itself, test outputs nothing, but because you’re a C programmer, you know that every program has a main function that returns an integer, and we will use only that return value here. Custom is to read the return value as a problem number, so 0==no problem, and in this case 1==file does not exist (which is why, as will be discussed in Don’t Bother Explicitly Returning from main, the default is that main returns zero). The shell doesn’t print the return value to the screen, but stores it in a variable, $?, which you can print via echo.

The echo command itself has a return value, and $? will be set to that value after you run echo $?. If you want to use the value of $? for a specific command more than once, assign it to a variable, such as returnval=$?.

Now let us use it in an if statement to act only if a file does not exist. As in C, ! means not.

if test ! -e a_test_file; then
    echo test file had not existed
    touch a_test_file
else
    echo test file existed
    rm a_test_file
fi

Notice that, as with the for loops from last time, the semicolon is in what I consider an awkward position, and we have the super-cute rule that we end if blocks with fi. By the way, else if is not valid syntax; use the elif keyword.

To make it easier for you to run this repeatedly, let’s cram it onto one margin-busting line. The keywords [ and ] are equivalent to test, so when you see this form in other people’s scripts and want to know what’s going on, the answer is in man test.

if [ ! -e a_test_file ]; then echo test file had not existed;  ↩
   touch a_test_file; else echo test file existed; rm a_test_file; fi

Because so many programs follow the custom that zero==OK and nonzero==problem, we can use if statements without test to express the clause if the program ran OK, then…. For example, it’s common enough to use tar to archive a directory into a single .tgz file, then delete the directory. It would be a disaster if the tar file somehow didn’t get created but the directory contents were deleted anyway, so we should have some sort of test that the tar command completed successfully before deleting everything:

#generate some test files
mkdir a_test_dir
echo testing ... testing > a_test_dir/tt


if tar cz a_test_dir > archived.tgz; then
    echo Compression went OK. Removing directory.
    rm -r a_test_dir
else
    echo Compression failed. Doing nothing.
fi

If you want to see this fail after running once, try chmod 000 archived.tgz to make the destination archive unwritable, then rerun.

unbind C-b
set -g prefix C-a
bind a send-prefix

fc is a (POSIX-standard) command for turning your noodling on the shell into a repeatable script. Try:

fc -l   # The l is for list and is important.

You now have on the screen a numbered list of your last few commands. Your shell might let you type history to get the same effect.

The -n flag suppresses the line numbers, so you can write history items 100 through 200 to a file via:

fc -l -n 100 200 > a_script

then remove all the lines that were experiments that didn’t work, and you’ve converted your futzing on the command line into a clean shell script.

If you omit the -l flag, then fc becomes a more immediate and volatile tool. It pulls up an editor (which means if you redirect with >, you’re basically hung), doesn’t display line numbers, and when you quit your editor, whatever is in that file gets executed immediately. This is great for a quick repetition of the last few lines, but can be disastrous if you’re not careful. If you realize that you forgot the -l or are otherwise surprised to see yourself in the editor, delete everything on the screen to prevent unintended lines from getting executed.

But to end on a positive note, fc stands for fix command, and that is its simplest usage. With no options, it edits the prior line only, so it’s nice for when you need to make elaborate corrections to a command.

setopt INTERACTIVE_COMMENTS
#now comments like this won’t give an error

# This line may create a lot of files all over your home directory.
for ff in ~/**/*.c; do cp $ff ${ff}.bkup; done

echo $((3/2.))      #works for zsh, syntax error for bash

#repeating the line-count example from earlier:
Files=`ls *.c |wc -l`
Lines=`grep '[)};]' *.c | wc -l`

echo lines/file = $(($Lines/($Files+0.0)))    #Cast to floating point by adding 0.0

# Generate two files, one of which has spaces in the name.
echo t1 > "test_file_1"
echo t2 > "test file 2"

# This fails in bash, is OK in Zsh.
for f in test* ; do cat $f; done

SHELL=command -v zsh

Example 3-5 presents a script that gets Autotools to take care of Hello, World. It is in the form of a shell script you can copy/paste onto your command line (as long as you make sure there are no spaces after the backslashes). Of course, it won’t run until you ask your package manager to install the Autotools: Autoconf, Automake, and Libtool.

if [ -e autodemo ]; then rm -r autodemo; fi
mkdir -p autodemo                      
cd autodemo
cat > hello.c <<\
"--------------"
#include <stdio.h>

int main(){ printf("Hi.\n"); }
--------------

cat > Makefile.am <<\                  
"--------------"
bin_PROGRAMS=hello
hello_SOURCES=hello.c
--------------

autoscan                               
sed -e 's/FULL-PACKAGE-NAME/hello/' \  
    -e 's/VERSION/1/'   \
    -e 's|BUG-REPORT-ADDRESS|/dev/null|' \
    -e '10i\
AM_INIT_AUTOMAKE' \
       < configure.scan > configure.ac

touch NEWS README AUTHORS ChangeLog    

autoreconf -iv                         
./configure
make distcheck

How much do all these macros do? The hello.c program itself is a leisurely three lines and Makefile.am is two lines, for five lines of user-written text. Your results may differ a little, but when I run wc -l * in the post-script directory, I find 11,000 lines of text, including a 4,700-line configure script.

It’s so bloated because it’s so portable: your recipients probably don’t have Autotools installed, and who knows what else they’re missing, so this script depends only on rudimentary POSIX-compliance.

I count 73 targets in the 600-line makefile.

Figure 3-1. An Autotools flowchart. You will only be writing two of these files (the shaded ones); everything else is autogenerated by the given command.

Figure 3-1 summarizes the story as a flow diagram.

You will only be writing two of these files (the shaded ones); everything else is autogenerated by the given command. Let’s start from the bottom portion: the user gets your package as a tarball, and untars it via tar xvzf your_pkg.tgz, which produces a directory with your code, Makefile.am, configure, and a host of other auxiliary files that aren’t worth discussing here. The user types ./configure, and that produces configure.h and the Makefile. Now everything is in place for the user to type make; sudo make install.

As an author, your goal is to produce that tarball, with a high-quality configure and Makefile.am, so the user can run his or her part without a hitch. Start by writing Makefile.am yourself. Run autoscan to get a preliminary configure.scan, which you will manually edit to configure.ac. (Not shown: the four files required by the GNU coding standards: NEWS, README, AUTHORS, and ChangeLog.) Then run autoreconf -iv to generate the configure script (plus many other auxiliary files). Given the configure script, you can now run it to produce the makefile; given the makefile, you can run make distcheck to generate the tarball to ship out.

Notice that there is some overlap: you will be using the same configure and Makefile as the user does, though your purpose is to produce a package and the user’s purpose is to install the package. That means you have the facilities to install and test the code without fully packaging it, and users have the facility to repackage the code if somehow so inclined.

A typical makefile is half about the structure of what parts of your project depend on what other parts, and half about the specific variables and procedures to execute. Your Makefile.am will focus on the structure of what needs to be compiled and what it depends on, and the specifics will be filled in by Autoconf and Automake’s built-in knowledge of compilation on different platforms.

Makefile.am will consist of two types of entry, which I will refer to as form variables and content variables.

PROGRAMS       
HEADERS    
LIBRARIES      static libraries
LTLIBRARIES    shared libraries generated via Libtool
DIST           items to be distributed with the package, such as data files that didn’t go elsewhere

bin_PROGRAMS       programs to build and install
check_PROGRAMS     programs to build for testing
include_HEADERS    headers to install in the system-wide include directory
lib_LTLIBRARIES    dynamic and shared libraries, via Libtool
noinst_LIBRARIES   static library (no Libtool), to keep on hand for later
noinst_DIST        distribute with the package, but that's all
python_PYTHON      Python code, to byte-compile and install wherever Python packages go

bin_PROGRAMS = hello

pkginclude_HEADERS = firstpart.h secondpart.h
noinst_DIST = sample1.csv sample2.csv \
             sample3.csv sample4.csv

bin_PROGRAMS= weather wxpredict
weather_SOURCES= temp.c barometer.c
wxpredict_SOURCES=rng.c tarotdeck.c

$(CC) $(LDFLAGS) temp.o barometer.o $(LDADD) -o weather

AM_CFLAGS=-g -Wall -O3

AM_CFLAGS=-g -Wall -O3
hello_CFLAGS = $(AM_CFLAGS) -O0

AM_CFLAGS=`pkg-config --cflags glib-2.0` -g -O3 -Wall 

lib_LTLIBRARIES=libdict.la                            
libdict_la_SOURCES=dict.c keyval.c                    

include_HEADERS=keyval.h dict.h

bin_PROGRAMS=dict_use
dict_use_SOURCES=dict_use.c
dict_use_LDADD=libdict.la                             

TESTS=$(check_PROGRAMS)                               
check_PROGRAMS=dict_test
dict_test_LDADD=libdict.la

target: deps
    script

TEMP=@autotemp@
HUMIDITY=@autohum@

#configure is a plain shell script; these are plain shell variables
autotemp=40
autohum=.8

AC_SUBST(autotemp)
AC_SUBST(autohum)

TEMP=40
HUMIDITY=.8

The configure.ac shell script produces two outputs: a makefile (with the help of Automake), and a header file named config.h.

If you’ve opened one of the sample configure.ac files produced so far, you might have noticed that it looks nothing at all like a shell script. This is because it makes heavy use of a set of macros (in the m4 macro language) that are predefined by Autoconf. Rest assured that every one of them will blow up into familiar-looking lines of shell script. That is, configure.ac isn’t a recipe or specification to generate the configure shell script, it is configure, just compressed by some very impressive macros.

The m4 language doesn’t have all that much syntax. Every macro is function-like, with parens after the macro name listing the comma-separated arguments (if any; else the parens are typically dropped). Where most languages write 'literal text', m4-via-Autoconf writes [literal text], and to prevent surprises where m4 parses your inputs a little too aggressively, wrap all of your macro inputs in those square brackets.

The first line that Autoscan generated is a good example:

AC_INIT([FULL-PACKAGE-NAME], [VERSION], [BUG-REPORT-ADDRESS])

We know that this is going to generate a few hundred lines of shell code, and somewhere in there, the given elements will be set. Change the values in square brackets to whatever is relevant. You can often omit elements, so something like:

AC_INIT([hello], [1.0])

is valid if you don’t want to hear from your users. At the extreme, one might give zero arguments to a macro like AC_OUTPUT, in which case you don’t need to bother with the parentheses.

What macros do we need for a functional Autoconf file? In order of appearance:

So we have the specification for a functional build package for any POSIX system anywhere in four or five lines, three of which Autoscan probably wrote for you.

But the real art that takes configure.ac from functional to intelligent is in predicting problems some users might have and finding the Autoconf macro that detects the problem (and, where possible, fixes it). You saw one example earlier: I recommended adding AC_PROG_CC_C99 to configure.ac to check for a C99 compiler. The POSIX standard requires that one be present via the command name c99, but just because POSIX says so doesn’t mean that every system will have one, so it is exactly the sort of thing that a good configure script checks for.

Having libraries on hand is the star example of a prerequisite that has to be checked. Getting back to Autoconf’s outputs for a moment, config.h is a standard C header consisting of a series of #define statements. For example, if Autoconf verified the presence of the GSL, you would find:

#define HAVE_LIBGSL 1

in config.h. You can then put #ifdefs into your C code to behave appropriately under appropriate circumstances.

Autoconf’s check doesn’t just find the library based on some naming scheme and hope that it actually works. It writes a do-nothing program using any one function somewhere in the library, then tries linking the program with the library. If the link step succeeds, then the linker was able to find and use the library as expected. So Autoscan can’t autogenerate a check for the library, because it doesn’t know what functions are to be found in it. The macro to check for a library is a one-liner, to which you provide the library name and a function that can be used for the check. For example:

AC_CHECK_LIB([glib-2.0],[g_free])
AC_CHECK_LIB([gsl],[gsl_blas_dgemm])

Add one line to configure.ac for every library you use that is not 100% guaranteed by the C standard, and those one-liners will blossom into the appropriate shell script snippets in configure.

You may recall how package managers always split libraries into the binary shared object package and the devel package with the headers. Users of your library might not remember (or even know) to install the header package, so check for it with, for example:

AC_CHECK_HEADER([gsl/gsl_matrix.h], , [AC_MSG_ERROR(
    [Couldn’t find the GSL header files (I searched for \
    <gsl/gsl_matrix.h> on the include path). If you are \
    using a package manager, don’t forget to install the \
    libgsl-devel package, as well as libgsl itself.])])

Notice the two commas: the arguments to the macro are (header to check, action if found, action if not found), and we are leaving the second blank.

What else could go wrong in a compilation? It’s hard to become an authority on all the glitches of all the world’s computers, given that we each have only a couple of machines at our disposal. Autoscan will give you some good suggestions, and you might find that running autoreconf also spits out some further warnings about elements to add to configure.ac. It gives good advice—follow its suggestions. But the best reference I have seen—a veritable litany of close readings of the POSIX standard, implementation failures, and practical advice—is the Autoconf manual itself. Some of it catalogs the glitches that Autoconf takes care of and are thus (thankfully) irrelevant nitpicking for the rest of us,^[7] some of it is good advice for your code-writing, and some of the descriptions of system quirks are followed by the name of an Autoconf macro to include in your project’s configure.ac should it be relevant to your situation.

echo \
"-----------------------------------------------------------

Thank you for installing ${PACKAGE_NAME} version ${PACKAGE_VERSION}.

Installation directory prefix: '${prefix}'.
Compilation command: '${CC} ${CFLAGS} ${CPPFLAGS}'

Now type 'make; sudo make install' to generate the program
and install it to your system.

------------------------------------------------------------"