Problem

You want some simple output from your shell commands.

Solution

Use the echo builtin command. All the parameters on the command line are printed to the screen. For example:

echo Please wait.

produces:

Please wait.

as we see in this simple session where we typed the command at the bash prompt (the $ character):

$ echo Please wait.
Please wait.
$

Discussion

The echo command is one of the simplest of all bash commands. It prints the arguments of the command line to the screen. But there are a few points to keep in mind. First, the shell is parsing the arguments on the echo command line (like it does for every other command line). This means that it does all its substitutions, wildcard matching, and other things before handing the arguments off to echo. Second, since they are parsed as arguments, the spacing between arguments is ignored. For example:

$ echo this    was     very    widely    spaced
this was very widely spaced
$

Normally the fact that the shell is very forgiving about whitespace between arguments is a helpful feature. Here, with echo, it’s a bit disconcerting (see Recipe 2.2 for tips on preserving whitespace in output and Recipe 13.15 for tips on trimming it from your data).

See Also

• help echo

• help printf

• Recipe 2.2, “Writing Output but Preserving Spacing”

• Recipe 2.3, “Writing Output with More Formatting Control”

• Recipe 13.15, “Trimming Whitespace”

• Recipe 15.6, “Using echo Portably”

• Recipe 19.1, “Forgetting to Set Execute Permissions”

• “echo Options and Escape Sequences” in Appendix A

• “printf” in Appendix A

Problem

You want the output to preserve your spacing.

Solution

Enclose the string in quotes. The previous example, but with quotes added, will preserve our spacing:

$ echo "this was    very    widely    spaced"
this    was    very    widely    spaced
$

or:

$ echo 'this  was  very   widely   spaced'
this    was   very  widely  spaced
$

Discussion

Since the words are enclosed in quotes, they form a single argument to the echo command. That argument is a string, and the shell doesn’t need to interfere with the contents of the string. In fact, by using single quotes ('') you explicitly tell the shell not to interfere with the string at all. If you use double quotes (""), some shell substitutions do take place (variable, arithmetic, and tilde expansions and command substitutions), but since we have none in this example, the shell has nothing to change. When in doubt, use the single quotes.

See Also

• help echo

• help printf

• Chapter 5 for more information about substitution

• Recipe 2.3, “Writing Output with More Formatting Control”

• Recipe 15.6, “Using echo Portably”

• Recipe 19.11, “Seeing Odd Behavior from printf”

• “echo Options and Escape Sequences” in Appendix A

Problem

You want more control over the formatting and placement of output.

Solution

Use the printf builtin command.

For example:

$ printf '%s = %d\n' Lines $LINES
Lines = 24
$

or:

$ printf '%-10.10s = %4.2f\n' 'Gigahertz' 1.92735
Gigahertz   = 1.93
$

Discussion

The printf builtin command behaves like the C language library call, where the first argument is the format control string and the successive arguments are formatted according to the format specifications (%).

The numbers between the % and the format type (s or f in our example) provide additional formatting details. For the floating-point type (f), the first number (4 in the 4.2 specifier) is the width of the entire field. The second number (2) is how many digits should be printed to the right of the decimal point. Note that it rounds the answer.

For a string, the first number is the maximum field width, and the second is the number of bytes to be printed. The string will be truncated (if longer than max) or blank padded (if less than min) as needed. When the max and min specifiers are the same, then the string is guaranteed to be that length. The negative sign on the specifier means to left-align the string (within its field width). Without the minus sign, the string would right-justify, thus:

$ printf '%10.10s = %4.2f\n' 'Gigahertz' 1.92735
  Gigahertz = 1.93
$

The string argument can either be quoted or unquoted. Use quotes if you need to preserve embedded spacing (there were no spaces needed in our one-word strings), or if you need to escape the special meaning of any special characters in the string (again, our example had none). It’s a good idea to be in the habit of quoting any string that you pass to printf, so that you don’t forget the quotes when you need them.

See Also

• help printf

• http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html

• Learning the bash Shell, 3rd Edition, by Cameron Newham (O’Reilly), page 171, or any C reference on its printf function

• Recipe 15.6, “Using echo Portably”

• Recipe 19.11, “Seeing Odd Behavior from printf”

• “printf” in Appendix A

Problem

You want to produce some output without the default newline that echo provides.

Solution

Using printf it’s easy—just leave off the ending \n in your format string:

$ printf "%s %s" next prompt
next prompt$

With echo, use the -n option:

$ echo -n prompt
prompt$

Discussion

Since there was no newline at the end of the printf format string (the first argument), the prompt character ($) appears right where the printf left off. This feature is much more useful in shell scripts where you may want to do partial output across several statements before completing the line, or where you want to display a prompt to the user before reading input.

With the echo command (see Recipe 15.6), there are two ways to eliminate the newline. First, the -n option suppresses the trailing newline. The echo command also has several escape sequences with special meanings similar to those in C language strings (e.g., \n for newline). To use these escape sequences, you must invoke echo with the -e option. One of echo’s escape sequences is \c, which doesn’t print a character, but rather inhibits printing the ending newline. Thus, here’s a third solution:

$ echo -e 'hi\c'
hi$

Because of the powerful and flexible formatting that printf provides, and because it is a builtin with very little overhead to invoke (unlike in other shells or older versions of bash, where printf was a standalone executable), we will use printf for many of our examples throughout the book.

See Also

• help echo

• help printf

• http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html

• Chapter 3, particularly Recipe 3.5, “Getting User Input”

• Recipe 2.3, “Writing Output with More Formatting Control”

• Recipe 15.6, “Using echo Portably”

• Recipe 19.11, “Seeing Odd Behavior from printf”

• “echo Options and Escape Sequences” in Appendix A

• “printf” in Appendix A

Problem

You want to keep the output from a command by putting it in a file.

Solution

Use the > symbol to tell the shell to redirect the output into a file. For example:

$ echo fill it up
fill it up
$ echo fill it up > file.txt
$

Just to be sure, let’s look at what is inside file.txt to see if it captured our output:

$ cat file.txt
fill it up
$

Discussion

The first line of the first part of the example shows an echo command with three arguments that are printed out. The second line uses the > to capture that output into a file named file.txt, which is why no output appears after that echo command.

The second part of the example uses cat to display the contents of the file. We can see that the file contains what echo would have otherwise sent as output.

The cat command gets its name from the longer word concatenation. The cat command concatenates the output from the files listed on its command line, so if you enter cat file1 filetwo anotherfile morefiles the contents of those files will be sent, one after another, to the terminal window. If a large file has been split in half, you can also use cat to glue it back together (i.e., concatenate the two halves) by capturing the output into a third file:

cat first.half second.half > whole.file

So our simple command, cat file.txt, is really just the trivial case of concatenating only one file, with the result sent to the screen. That is to say, while cat is capable of more, its primary use in this example is to dump the contents of a file to the screen.

See Also

• man cat

• Recipe 17.23, “Numbering Lines”

Problem

You want to save the output with a redirect to elsewhere in the filesystem, not in the current directory.

Solution

Use more of a pathname when you redirect the output:

echo some more data > /tmp/echo.out

or:

echo some more data > ../../over.here

Discussion

The filename that appears after the redirection character (the >) is actually a path-name. If it begins with no other qualifiers, the file will be placed in the current directory.

If that filename begins with a slash (/) then it is an absolute pathname, and output will be placed where it specifies in the filesystem hierarchy (i.e., tree), beginning at the root (provided all the intermediary directories exist and have permissions that allow you to traverse them). We used /tmp since it is a well-known, universally available scratch directory on virtually all Unix systems. The shell, in this example, will create the file named echo.out in the /tmp directory.

Our second example, placing the output into ../../over.here, uses a relative pathname, and the .. is the specially named directory inside every directory that refers to the parent directory. So, each reference to .. moves up a level in the filesystem tree (toward the root, not what we usually mean by “up” in a tree). The point here is that we can redirect our output, if we want, into a file that is far away from where we are running the command.

See Also

• Learning the bash Shell, 3rd Edition, by Cameron Newham (O’Reilly), pages 7–10, for an introduction to files, directories, and the dot notation (i.e., . and ..)

Problem

You tried to save output from the ls command with a redirect, but when you look at the resulting file, the format is not what you expected.

Solution

Use the -C option on ls when you redirect the output.

Here’s the ls command showing the contents of a directory:

$ ls
a.out cong.txt def.conf  file.txt  more.txt  zebra.list
$

But when we save the output with the > to redirect it to a file, and then show the file contents, we get one file per line, like this:

$ ls > /tmp/save.out
$ cat /tmp/save.out
a.out
cong.txt
def.conf
file.txt
more.txt
zebra.list
$

This time we’ll use the -C option:

$ ls -C > /tmp/save.out
$ cat /tmp/save.out
a.out cong.txt def.conf file.txt more.txt zebra.list
$

Alternatively, if we use the -1 option on ls when we don’t redirect, we get output like this:

$ ls -1
a.out
Cong.txt
def.conf
file.txt
more.txt
zebra.list
$

The original attempt at redirection matches this output.

Discussion

Just when you thought that you understood redirection and you tried it on a simple ls command, it didn’t quite work right. What’s going on here?

The shell’s redirection is meant to be transparent to all programs, so programs don’t need special code to make their output redirectable. The shell takes care of it when you use the > to send the output elsewhere. But it turns out that code can be added to a program to figure out when its output is a terminal (see man isatty). Then, the program can behave differently in those two cases—and that’s what ls is doing.

The authors of ls figured that if your output is going to the screen, then you probably want columnar output (the -C option), as screen real estate is limited. But they assumed if you’re redirecting it to a file, then you’ll want one file per line (the -1 option) since there are more interesting things you can do (i.e., other processing) that is easier if each filename is on a line by itself.

See Also

• man ls

• man isatty

• Recipe 2.6, “Saving Output to Other Files”

Problem

You are expecting output from a program, but you don’t want it to get littered with error messages. You’d like to save your error messages, but it’s harder to find them mixed among the expected output.

Solution

Redirect output and error messages to different files:

myprogram 1> messages.out 2> message.err

or more commonly:

myprogram > messages.out 2> message.err

Discussion

This example shows two different output files created by the shell. The first, messages.out, will get all the output from the hypothetical myprogram redirected into it. Any error messages from myprogram will be redirected into message.err.

In the constructs 1> and 2> the number is the file descriptor. 1 is standard output (STDOUT) and 2 is standard error (STDERR). Numbering starts at 0, for standard input (STDIN). When no number is specified, STDOUT is assumed. For more information on file descriptors and the difference between STDOUT and STDERR, see Recipe 2.19.

See Also

• Recipe 2.6, “Saving Output to Other Files”

• Recipe 2.13, “Throwing Output Away”

• Recipe 2.19, “Saving Output When Redirect Doesn’t Seem to Work”

Problem

Using redirection, you can redirect output or error messages to separate files, but how do you capture all the output and error messages to a single file?

Solution

Use the shell syntax to redirect standard error messages to the same place as standard output.

Preferred:

both >& outfile

or:

both &> outfile

or older and slightly more verbose (but also more portable):

both > outfile 2>&1

where both is just our (imaginary) program that is going to generate output to both STDERR and STDOUT.

Discussion

&> and >& are shortcuts that simply send both STDOUT and STDERR to the same place—exactly what we want to do.

In the third example, the 1 appears to be used as the target of the redirection, but the >& says to interpret the 1 as a file descriptor instead of a filename. In fact, the 2>&1 is a single entity—no spaces allowed—indicating that standard error (2) will be redirected (>) to a file descriptor (&) that follows (1). The 2>& all has to appear together without spaces; otherwise the 2 would look just like another argument, and the & actually means something completely different when it appears by itself. (It has to do with running the command in the background.)

It may help to think of all redirection operators as taking a leading number (e.g., 2>), but that the default number for > is 1, the standard output file descriptor.

You could also do the redirection in the other order, though it is slightly less read-able, and redirect standard output to the same place to which you have already redirected standard error (in fact you must do it this way if you are using a pipe; see Recipe 2.15):

both 2> outfile 1>&2

The 1 indicates standard output and the 2 standard error. We could have written just >&2 for that last redirection, since 1 is the default for >, but we find it more readable to write the number explicitly when redirecting file descriptors.

See Also

• Recipe 2.6, “Saving Output to Other Files”

• Recipe 2.13, “Throwing Output Away”

Problem

Each time you redirect your output, it creates that output file anew. What if you want to redirect output a second (or third, or…) time, and don’t want to clobber the previous output?

Solution

The double greater-than sign (>>) is a bash redirector that means append the output:

$ ls > /tmp/ls.out
$ cd ../elsewhere
$ ls >> /tmp/ls.out
$ cd ../anotherdir
$ ls >> /tmp/ls.out
$

Discussion

The first line includes a redirect that truncates the file if it exists and starts with a clean (empty) file, filling it with the output from the ls command.

The second and third invocations of ls use the double greater-than sign (>>) to indicate appending to, rather than replacing the contents of, the output file.

If you want to have error messages (i.e., STDERR) included in the redirection, specify that redirection after redirecting STDERR, like this:

ls >> /tmp/ls.out 2>&1

As of bash version 4 you can combine both of those redirections in one:

ls &>> /tmp/ls.out

which will redirect both STDERR and STDOUT and append them to the specified file. Just remember that the ampersand must come first and no spacing is allowed between the three characters.

See Also

• Recipe 2.6, “Saving Output to Other Files”

• Recipe 2.13, “Throwing Output Away”

Problem

You need to display or use just the beginning or end of a file.

Solution

Use the head or tail command. By default, head will output the first 10 lines and tail will output the last 10 lines of the given file. If more than one file is given, the appropriate lines from each of them are output. Use the -number switch (e.g., -5) to change the number of lines. tail also has the -f and -F switches, which follow the end of the file as it is written to, and it has an interesting + switch that we cover in Recipe 2.12.

Discussion

head and tail, along with cat, grep, sort, cut, and uniq, are some of the most commonly used Unix text processing tools out there. If you aren’t already familiar with them, you’ll soon wonder how you ever got along without them.

See Also

• Recipe 2.12, “Skipping a Header in a File”

• Recipe 7.1, “Sifting Through Files for a String”

• Recipe 8.1, “Sorting Your Output”

• Recipe 8.4, “Cutting Out Parts of Your Output”

• Recipe 8.5, “Removing Duplicate Lines”

• Recipe 17.23, “Numbering Lines”

Problem

You have a file with one or more header lines and you need to process just the data, and skip the header.

Solution

Use the tail command with a special argument. For example, to skip the first line of a file:

$ tail -n +2 lines
Line 2
Line 3
Line 4
Line 5
$

Discussion

An argument to tail, of the format -n number (or just -number), will specify a line offset relative to the end of the file. So, tail -n 10 file shows the last 10 lines of file, which also happens to be the default if you don’t specify anything. Specifying a number starting with a plus sign (+) indicates an offset relative to the top of the file. Thus, tail -n +1 file gives you the entire file, tail -n +2 skips the first line, and so on.

See Also

• man tail

• Recipe 13.12, “Setting Up a Database with MySQL”

Problem

Sometimes you don’t want to save the output into a file; in fact, sometimes you don’t even want to see it at all.

Solution

Redirect the output to /dev/null as shown in these examples:

find / -name myfile -print 2> /dev/null

or:

noisy > /dev/null 2>&1

Discussion

You could redirect the unwanted output into a file, then remove the file when you’re done. But there is an easier way. Unix and Linux systems have a special device that isn’t real hardware at all, just a bit bucket where we can dump unwanted data. It’s called /dev/null and is perfect for these situations. Any data written there is simply thrown away, so it takes up no disk space. Redirection makes it easy.

In the first example, only the output going to standard error is thrown away. In the second example, both standard output and standard error are discarded.

In rare cases, you may find yourself in a situation where /dev is on a read-only filesystem (for example, certain information security appliances), in which case you are stuck with the first suggestion of writing to a file and then removing it.

See Also

• Recipe 2.6, “Saving Output to Other Files”

Problem

You want to capture the output with a redirect, but you’re typing several commands on one line:

pwd; ls; cd ../elsewhere; pwd; ls > /tmp/all.out

The final redirect applies only to the last command, the last ls on that line. All the other output appears on the screen (i.e., does not get redirected).

Solution

Use braces ({ }) to group these commands together; then redirection applies to the output from all commands in the group. For example:

{ pwd; ls; cd ../elsewhere; pwd; ls; } > /tmp/all.out

Alternatively, you could use parentheses, (), to tell bash to run the commands in a subshell, then redirect the output of the entire subshell’s execution. For example:

(pwd; ls; cd ../elsewhere; pwd; ls) > /tmp/all.out

Discussion

While these two solutions look very similar, there are two important differences. The first difference is syntactic, the second semantic. Syntactically, the braces need to have whitespace around them, and the last command inside the list must terminate with a semicolon. That’s not required when you use parentheses. The bigger difference, though, is semantic—what these constructs mean. The braces are just a way to group several commands together, more like a shorthand for our redirecting, so that we don’t have to redirect each command separately. Commands enclosed in parentheses, however, run in another instance of the shell, a child of the current shell called a subshell.

The subshell is almost identical to the current shell’s environment—i.e., variables, including $PATH, are all the same, but traps are handled differently (for more on traps, see Recipe 10.6). Now here is the big difference in using the subshell approach: because a subshell is used to execute the cd commands, when the subshell exits, your main shell remains where it started. That is, its current directory hasn’t moved, and its variables haven’t changed.

With the braces used for grouping, you end up in the new directory (../elsewhere in our example). Any other changes that you make (variable assignments, for example) will be made to your current shell instance. While both approaches result in the same output, they leave you in very different places.

One interesting thing you can do with braces is form more concise branching blocks (Recipe 6.2). You can shorten this:

if [ $result = 1 ]; then
    echo "Result is 1; excellent."
    exit 0
else
    echo "Uh-oh, ummm, RUN AWAY! "
    exit 120
fi

into this:

[ $result = 1 ] \
  && { echo "Result is 1; excellent." ; exit 0; } \
  || { echo "Uh-oh, ummm, RUN AWAY! " ; exit 120; }

How you write it depends on your style and what you think is readable, but we recommend the first form because it is clearer to a wider audience.

Problem

You want to take the output from one program and use it as the input of another program.

Solution

You could redirect the output from the first program into a temporary file, then use that file as input to the second program. For example:

$ cat one.file another.file > /tmp/cat.out
$ sort < /tmp/cat.out
...
$ rm /tmp/cat.out
$

Or you could do all of that in one step, sending the output directly to the next program, by using the pipe symbol (|) to connect them. For example:

cat one.file another.file | sort

You can also link a sequence of several commands together by using multiple pipes:

cat my* | tr 'a-z' 'A-Z' | sort | uniq | awk -f transform.awk | wc

Discussion

Using the pipe symbol means we don’t have to invent a temporary filename, remember it, and remember to delete it.

Programs like sort can take input from standard input (redirected via the < symbol), but they can also take input as a filename. So, you can do this:

sort /tmp/cat.out

rather than redirecting the input into sort:

sort < /tmp/cat.out

That behavior (of using a filename if supplied, and if not, of using standard input) is a typical Unix/Linux characteristic, and a useful model to follow so that commands can be connected one to another via the pipe mechanism. Such programs are called filters, and if you write your programs and shell scripts that way, they will be more useful to you and to those with whom you share your work.

Feel free to be amazed at the powerful simplicity of the pipe mechanism. You can even think of the pipe as a rudimentary parallel processing mechanism. You have two commands (programs) running in parallel, sharing data—the output of one as the input to the next. They don’t have to run sequentially (where the first runs to completion before the second one starts); the second one can get started as soon as data is available from the first.

Be aware, however, that commands run this way (i.e., connected by pipes) are run in separate processes. While such a subtlety can often be ignored, there are a few times when the implications of this are important. We’ll discuss that in Recipe 19.8.

Also consider a command such as svn -v log | less. If less exits before Subversion has finished sending data, you’ll get an error like svn: Write error: Broken pipe. While it isn’t pretty, it also isn’t harmful. It happens all the time when you pipe a voluminous amount of data into a program like less—you often want to quit once you’ve found what you’re looking for, even if there is more data coming down the pipe.

See Also

• Recipe 3.1, “Getting Input from a File”

• Recipe 19.8, “Forgetting that Pipelines Make Subshells”

Problem

You want to debug a long sequence of piped I/O, such as:

cat my* | tr 'a-z' 'A-Z' | uniq | awk -f transform.awk | wc

How can you see what is happening between uniq and awk without disrupting the pipe?

Solution

The solution to these problems is to use what plumbers call a T-joint in the pipes. For bash, that means using the tee command to split the output into two identical streams, one that is written to a file and the other that is written to standard output, so as to continue the sending of data along the pipes.

For this example where we’d like to debug a long string of pipes, we insert the tee command between uniq and awk:

... uniq | tee /tmp/x.x | awk -f transform.awk ...

Discussion

The tee command writes the output to the filename(s) specified as its parameter and also writes that same output to standard out. In our example, it sends a copy to /tmp/x.x and also sends the same data to awk, the command to which the output of tee is connected via the pipe symbol.

Don’t worry about what each different piece of the command line is doing in these examples; we just want to illustrate how tee can be used in any sequence of commands.

Let’s back up just a bit and start with a simpler command line. Suppose you’d just like to save the output from a long-running command for later reference, while at the same time seeing it on the screen. After all, a command like:

find / -name '*.c' -print | less

could find a lot of C source files, so the output will likely scroll off the window. Using more or less will let you look at the output in manageable pieces, but once completed they don’t let you go back and look at that output without rerunning the command. Sure, you could run the command and save it to a file:

find / -name '*.c' -print > /tmp/all.my.sources

but then you have to wait for it to complete before you can see the contents of the file. (OK, we know about tail -f, but that’s just getting off-topic here.) The tee command can be used instead of the simple redirection of standard output:

find / -name '*.c' -print | tee /tmp/all.my.sources

In this example, since the output of tee isn’t redirected anywhere, it will print to the screen. But the copy that is diverted into a file will also be there for later use (e.g., cat /tmp/all.my.sources).

Notice, too, that in these examples we did not redirect standard error at all. This means that any errors, like you might expect from find, will be printed to the screen but won’t show up in the tee file. We could add a 2>&1 to the find command:

find / -name '*.c' -print 2>&1 | tee /tmp/all.my.sources

to include the error output in the tee file. It won’t be neatly separated, but it will be captured.

See Also

• man tee

• Recipe 18.5, “Reusing Arguments”

• Recipe 19.13, “Debugging Scripts”

Problem

What if one of the programs to which you would like to connect with a pipe doesn’t work that way? For example, you can remove files with the rm command, specifying the files to be removed as parameters to the command:

rm my.java your.c their.*

But rm doesn’t read from standard input, so you can’t do something like:

find . -name '*.c' | rm

Since rm only takes its filenames as arguments or parameters on the command line, how can we get the output of a previously run command (e.g., echo or ls) onto the command line?

Solution

Use the command substitution feature of bash:

rm $(find . -name '*.class')

You can also use the xarg command; see the discussion in Recipe 15.13.

Discussion

The $() encloses a command that is run in a subshell. The output from that command is substituted in place of the $() phrase. Newlines cause the output to become several parameters on the command line, which is often useful but may sometimes be surprising.

The earlier shell syntax was to use backquotes (``) instead of $() for enclosing the sub-command. The $() syntax is preferred over the older `` syntax because it is easier to nest and arguably easier to read. However, you may see `` more often than $(), especially in older scripts or from those who grew up with the original Bourne or C shells.

In our example, the output from find—typically a list of names—will become the arguments to the rm command.

$ find . -name '*.class'
First.class
Other.class
$ rm $(find . -name '*.class')
$

See Also

• Recipe 9.8, “Finding Files by Size”

• Recipe 18.2, “Repeating the Last Command”

• Recipe 15.13, “Working Around “Argument list too long” Errors”

Problem

You want to redirect output to several different places.

Solution

Use redirection with file numbers to open all the files that you want to use. For example:

divert 3> file.three 4> file.four 5> file.five 6> else.where

where divert might be a shell script with various commands whose output you want to send to different places. For example, you might write divert to contain lines like this: echo option $OPTSTR >&5. That is, your divert shell script could direct its output to various different descriptors, which the invoking program can send to different destinations.

Similarly, if divert was a C program executable, you could actually write to descriptors 3, 4, 5, and 6 without any need for open() calls.

Discussion

In Recipe 2.8 we explained that each file descriptor is indicated by a number, starting at zero: standard input is 0, standard output is 1, and standard error is 2. If no number is given, 1 is assumed. That means that you could redirect standard output with the slightly more verbose 1> (rather than a simple >) followed by a filename, but there’s no need; the shorthand > is fine. It also means that you can have the shell open up any number of arbitrary file descriptors and have them set to write various files so that the program that the shell then invokes from the command line can use these opened file descriptors without further ado.

While we don’t recommend this technique because it’s fragile and more complicated than it needs to be, it is intriguing.

See Also

• Recipe 2.6, “Saving Output to Other Files”

• Recipe 2.8, “Sending Output and Error Messages to Different Files”

• Recipe 2.13, “Throwing Output Away”

Problem

You tried using > but some (or all) of the output still appears on the screen.

For example, the compiler was producing these error messages:

$ gcc bad.c
bad.c: In function `main':
bad.c:3: error: `bad' undeclared (first use in this function)
bad.c:3: error: (Each undeclared identifier is reported only once
bad.c:3: error: for each function it appears in.)
bad.c:3: error: parse error before "c"
$

You wanted to capture those messages, so you tried redirecting the output:

$ gcc bad.c > save.it
bad.c: In function `main':
bad.c:3: error: `bad' undeclared (first use in this function)
bad.c:3: error: (Each undeclared identifier is reported only once
bad.c:3: error: for each function it appears in.)
bad.c:3: error: parse error before "c"
$

However, it doesn’t seem to have redirected anything. In fact, when you examine the file into which you were directing the output, that file is empty (zero bytes long):

$ ls -l save.it
-rw-r--r-- 1 albing users 0 2005-11-13 15:30 save.it
$ cat save.it
$

Solution

Redirect the error output, as follows:

gcc bad.c 2> save.it

The contents of save.it are now the error messages that you saw before.

Discussion

So what’s going on here? Every process in Unix and Linux typically starts out with three open file descriptors: one for input called standard input (STDIN), one for output called standard output (STDOUT), and one for error messages called standard error (STDERR). It is really up to the programmer who writes any particular program to stick to these conventions and write error messages to standard error and to the normally expected output to standard out, so there is no guarantee that every error message that you ever get will go to standard error. But most of the long-established utilities are well behaved this way. That is why these compiler messages are not being diverted with a simple > redirect; it only redirects standard output, not standard error.

As mentioned in the previous recipe, each file descriptor is indicated by a number, starting at zero. Standard input is 0, output is 1, and error is 2. That means that you could redirect standard output with the slightly more verbose: 1> (rather than a simple >) followed by a filename, but there’s no need. The shorthand > is fine. To redirect standard error, use 2>.

One important difference between standard output and standard error is that standard output is buffered but standard error is unbuffered; that is, every character is written individually, and they aren’t collected together and written as a bunch. This means that you see the error messages right away and that there is less chance of them being dropped when a fault occurs, but the cost is one of efficiency. It’s not that standard output is unreliable, but in error situations (e.g., when a program dies unexpectedly), the buffered output may not have made it to the screen before the program stops executing. That’s why standard error is unbuffered: to be sure the message gets written. By contrast, with standard output, only when the buffer is full (or when the file is closed) does the output actually get written. It’s more efficient for the more frequently used output, but efficiency isn’t as important when an error is being reported.

What if you want to see the output as you are saving it? The tee command we discussed in Recipe 2.16 seems just the thing:

gcc bad.c 2>&1 | tee save.it

This will take standard error and redirect it to standard out, piping them both into tee. The tee command will write its input to both the file (save.it) and tee’s standard out, which will go to your screen since it isn’t otherwise redirected.

This is a special case of redirecting because normally the order of the redirections is important. Compare these two commands:

somecmd >my.file 2>&1
somecmd 2>&1 >my.file

In the first case, standard output is redirected to a file (my.file), and then standard error is redirected to the same place as standard out. All output will appear in my.file.

But that is not the case with the second command. In the second command, standard error is redirected to standard output (which at that point is connected to the screen), after which standard output is redirected to my.file. Thus, only standard output messages will be put in the file, and errors will still show on the screen.

However, this ordering had to be subverted for pipes—you can’t put the second redirect after the pipe symbol, because after the pipe comes the next command. So, bash makes an exception when you write:

somecmd 2>&1 | othercmd

and recognizes that standard output is being piped. It therefore assumes that you want to include standard error in the piping when you write 2>&1 even though its normal ordering wouldn’t work that way.

The other result of this, and of pipe syntax in general, is that it gives us no way to pipe just standard error and not standard output into another command—unless we first swap the file descriptors (see the next recipe).

somecmd |& othercmd

See Also

• Recipe 2.17, “Connecting Two Programs by Using Output as Arguments”

• Recipe 2.20, “Swapping STDERR and STDOUT”

Problem

You need to swap STDERR and STDOUT so you can send STDOUT to a logfile, but then send STDERR to the screen and to a file using the tee command. But pipes only work with STDOUT.

Solution

Swap STDERR and STDOUT before the pipe redirection using a third file descriptor:

./myscript 3>&1 1>stdout.logfile 2>&3- | tee -a stderr.logfile

Discussion

Whenever you redirect file descriptors, you are duplicating the open descriptor to another descriptor. This gives you a way to swap descriptors, much like how any program swaps two values—by means of a third, temporary holder. Copy A into C, copy B into A, copy C into B, and then you have swapped the values of A and B. For file descriptors, it looks like this:

./myscript 3>&1 1>&2 2>&3

Read the syntax 3>&1 as “give file descriptor 3 the same value as output file descriptor 1.” What happens here is that it duplicates file descriptor 1 (i.e., STDOUT) into file descriptor 3, our temporary holding place. Then it duplicates file descriptor 2 (i.e., STDERR) into STDOUT, and finally duplicates file descriptor 3 into STDERR. The net effect is that the STDERR and STDOUT file descriptors have swapped places.

So far, so good. Now we just change this slightly. Once we’ve copied STDOUT (into file descriptor 3), we are free to redirect STDOUT into the logfile we want to have capture the output of our script or other program. Then we can copy the file descriptor from its temporary holding place (file descriptor 3) into STDERR. Adding the pipe will now work because the pipe connects to the (original) STDOUT. That gets us to the solution shown earlier:

./myscript 3>&1 1>stdout.logfile 2>&3- | tee -a stderr.logfile

Note the trailing - on the 2>&3- term. We do that so that we close file descriptor 3 when we are done with it. That way our program doesn’t have an extra open file descriptor. We are tidying up after ourselves.

We’re also using the -a option to tee to append instead of replace.

See Also

• Linux Server Hacks, by Rob Flickenger (O’Reilly), Hack #5, “n>&m: Swap STDOUT and STDERR”

• Recipe 2.19, “Saving Output When Redirect Doesn’t Seem to Work”

• Recipe 10.1, ““Daemon-izing” Your Script”

Problem

You don’t want to delete the contents of a file by mistake. It can be too easy to mistype a filename and find that you’ve redirected output into a file that you meant to save.

Solution

Tell the shell to be more careful, as follows:

set -o noclobber

If you decide you don’t want to be so careful after all, then turn the option off:

set +o noclobber

Discussion

The noclobber option tells bash not to overwrite any existing files when you redirect output. If the file to which you redirect output doesn’t (yet) exist, everything works as normal, with bash creating the file as it opens it for output. If the file already exists, however, you will get an error message.

Here it is in action. We begin by turning the option off, just so that your shell is in a known state, regardless of how your particular system may be configured:

$ set +o noclobber                                
$ echo something > my.file                        
$ echo some more > my.file
$ set -o noclobber                                
$ echo something > my.file
bash: my.file: cannot overwrite existing file
$ echo some more >> my.file                       
$

The first time we redirect output to my.file the shell will create it for us.

The second time we redirect, bash overwrites the file (it truncates the file to 0 bytes and starts writing from there).

Then we set the noclobber option and we get an error message when we try to write to that file.

As we show in the last part of this example, we can append to the file (using >>) just fine.

$ echo useless data > some.file
$ echo important data > other.file
$ set -o noclobber
$ cp some.file other.file
$

If you’re a good and careful typist this may not seem like an important option, but we will look at other recipes where filenames are generated with regular expressions or passed as variables. Those filenames could be used as the filename for output redirection. In such cases, having noclobber set may be an important safety feature for preventing unwanted side effects (whether goofs or malicious actions).

See Also

• A good Linux reference on the chmod command and file permissions, such as:
  —http://www.linuxforums.org/security/file_permissions.html
  —http://www.comptechdoc.org/os/linux/usersguide/linux_ugfilesp.html
  —http://www.faqs.org/docs/linux_intro/sect_03_04.html
  —http://www.perlfect.com/articles/chmod.shtml

• Recipe 2.22, “Clobbering a File on Purpose”

• Recipe 14.13, “Setting Permissions”

Problem

You like to have noclobber set, but every once in a while you do want to clobber a file when you redirect output. Can you override bash’s good intentions, just once?

Solution

Use >| to redirect your output. Even if noclobber is set, bash ignores its setting and overwrites the file.

Consider this example:

$ echo something > my.file
$ set -o noclobber
$ echo some more >| my.file                       
$ cat my.file
some more
$ echo once again > my.file                       
bash: my.file: cannot overwrite existing file
$

Notice that no error message occurs on the second echo.

But on the third echo, when we are no longer using the vertical bar but just the plain > character by itself, the shell warns us and does not clobber the existing file.

Discussion

Using noclobber does not take the place of file permissions. If you don’t have write permission in the directory, you won’t be able to create the file, whether or not you use the >| construct. Similarly, you must have write permission on the file itself to overwrite that existing file, whether or not you use the >|.

So why the vertical bar? According to Chet, “POSIX specifies the >| syntax, which it picked up from ksh88. I’m not sure why Korn chose it. csh does use >!.” To help you remember it, you can think of it as for emphasis. Its use in English (with the imperative mood) fits that sense of “do it anyway!” when telling bash to overwrite the file if need be. The vi and ex editors use the ! with that same meaning in their write (:w! filename) command. Without a !, the editor will complain if you try to overwrite an existing file. With it, you are telling the editor to “do it!”

See Also

• Recipe 2.21, “Keeping Files Safe from Accidental Overwriting”

• Recipe 14.13, “Setting Permissions”

Problem

You want your shell commands to read data from a file.

Solution

Use input redirection, indicated by the < character, to read data from a file:

 wc < my.file

Discussion

Just as the > sends output to a file, so the < takes input from a file. The choice and shape of the characters were meant to give a visual clue as to what was going on with redirection. Can you see it? (Think “arrowhead.”)

Many shell commands will take one or more filenames as arguments, but when no filename is given will read from standard input. Those commands can be invoked as either command filename or command < filename with the same result. That’s the case here with wc, but also with cat and others.

It may look like a simple feature, and be familiar if you’ve used the DOS command line before, but it is a significant feature of shell scripting (which the DOS command line borrowed) and was radical in both its power and its simplicity when first introduced.

See Also

• Recipe 2.6, “Saving Output to Other Files”

Problem

You need input to your script, but don’t want a separate file.

Solution

Use a here-document with the << characters, redirecting the text from the command line rather than from a file. When put into a shell script, the script file then contains the data along with the script.

Here’s an example of a shell script in a file we call ext:

$ cat ext
#
# here is a "here" document
#
grep $1 <<EOF
mike x.123
joe  x.234
sue  x.555
pete x.818
sara x.822
bill x.919
EOF
$

It can be used as a shell script for simple phone number lookups:

$ ext bill
bill x.919
$

or:

$ ext 555
sue x.555
$

Discussion

The grep command looks for occurrences of the first argument in the files that are named, or if no files are named it looks to standard input.

A typical use of grep is something like this:

grep somestring file.txt

or:

grep myvar *.c

In our ext script we’ve parameterized the grep by making the string that we’re searching for be the parameter of our shell script ($1). Whereas we often think of grep as searching for a fixed string through various different files, here we are going to vary what we search for, but search through the same data every time.

We could have put our phone numbers in a file, say phonenumbers.txt, and then used that filename on the line that invokes the grep command:

grep $1 phonenumbers.txt

However, that requires two separate files (our script and our datafile) and raises the question of where to put them and how to keep them together.

So, rather than supplying one or more filenames to search through, we set up a here-document and tell the shell to redirect standard input to come from that (temporary) document.

The << syntax says that we want to create such a temporary input source, and the EOF is just an arbitrary string (you can choose what you like) to act as the terminator of the temporary input. It is not part of the input, but acts as the marker to show where it ends. The regular shell script (if any) resumes after the marker.

We also might add -i to the grep command to make our search case-insensitive. Thus, using grep -i $1 <<EOF would allow us to search for “Bill” as well as “bill”.

See Also

• man grep

• Recipe 3.3, “Preventing Weird Behavior in a Here-Document”

• Recipe 3.4, “Indenting Here-Documents”

Problem

Your here-document is behaving weirdly. You wanted to maintain a simple list of donors using the method described previously for phone numbers, so you created a file called donors that looked like this:

$ cat donors
#
# simple lookup of our generous donors
#
grep $1 <<EOF
# name amt
pete $100
joe  $200
sam  $ 25
bill $ 9
EOF
$

But when you tried running it you got weird output:

$ ./donors bill
pete bill00
bill $  9
$ ./donors pete
pete pete00
$

Solution

Turn off the shell scripting features inside the here-document by escaping any or all of the characters in the ending marker:

grep $1 <<'EOF'
pete $100
joe  $200
sam  $ 25
bill $ 9
EOF

Discussion

It’s a very subtle difference, but the <<EOF can be replaced with <<\EOF, or <<'EOF', or even <<E\OF—they all work. It’s not the most elegant syntax, but it’s enough to tell bash that you want to treat the “here” data differently.

Normally (i.e., unless you use this escaping syntax), says the bash manpage, “…all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion.”

So what’s happening in our original donors script is that the amounts are being interpreted as shell variables. For example, $100 is being seen as the shell variable $1 followed by two zeros. That’s what gives us pete00 when we search for “pete” and bill00 when we search for “bill”.

When we escape some or all of the characters of the EOF, bash knows not to do the expansions, and the behavior is the expected behavior:

$ ./donors pete
pete $100

Of course, you may want the shell expansion on your data—it can be useful in the correct circumstances—but that isn’t what we want here. We’ve found it to be a useful practice to always escape the marker, as in <<'EOF' or <<\EOF, to avoid unexpected results, unless you know that you really want the expansion to be done on your data.

See Also

• Recipe 3.2, “Keeping Your Data with Your Script”

• Recipe 3.4, “Indenting Here-Documents”

Problem

The here-document is great, but it’s messing up your shell script’s formatting. You want to be able to indent for readability.

Solution

Use <<-, and then you can use tab characters (only!) at the beginning of lines to indent this portion of your shell script:

$ cat myscript.sh
...
     grep $1 <<-'EOF'
        lots of data
        can go here
        it's indented with tabs
        to match the script's indenting
        but the leading tabs are
        discarded when read
        EOF
    ls
...
$

Discussion

The hyphen (-) just after the << is enough to tell bash to ignore the leading tab characters. This is for tab characters only and not arbitrary whitespace. Note that this is especially important with the EOF or any other marker designation. If you have spaces there, it will not recognize the EOF as your ending marker, and the “here” data will continue through to the end of the file (swallowing the rest of your script). Therefore, you may want to always left-justify the EOF (or other marker) just to be safe, and let the formatting go on this one line.

See Also

• Recipe 3.2, “Keeping Your Data with Your Script”

• Recipe 3.3, “Preventing Weird Behavior in a Here-Document”

Problem

You need to get input from the user.

Solution

Use the read statement:

read

or:

read -p "answer me this " ANSWER

or:

read -t 3 -p "answer quickly: " ANSWER

or:

read PRE MID POST

Discussion

In its simplest form, a read statement with no arguments will read user input and place it into the shell variable REPLY.

If you want bash to print a prompt string before reading the input, use the -p option. The next word following the -p will be the prompt, but quoting allows you to supply multiple words for a prompt. Remember to end the prompt with punctuation and/or a space, as the cursor will wait for input right at the end of the prompt string.

The -t option sets a timeout. The read statement will return after the specified number of seconds regardless of whether the user has responded. Our example uses both the -t and -p options together, but you can use the -t option on its own. As of bash version 4 you can even specify fractional numbers of seconds, like .25 or 3.5 for the timeout value. The exit status ($?) will be greater than 128 if the read timed out.

If you supply multiple variable names in the read statement, then read parses the input into words, assigning them in order. If the user enters fewer words, the extra variables will be set to null. If the user enters more words than there are variables in the read statement, then all of the extra words will be part of the last variable in the list.

See Also

• help read for more options to the read builtin

• Recipe 3.8, “Prompting for a Password”

• Recipe 6.11, “Looping with a read”

• Recipe 13.6, “Parsing Text with a read Statement”

• Recipe 14.12, “Validating Input”

Problem

You need to get a simple yes or no input from the user, and you want to be as user-friendly as possible. In particular, you do not want to be case-sensitive, and you want to provide a useful default if the user presses the Enter key.

Solution

If the actions to take are simple, use the self-contained function in Example 3-1.

# cookbook filename: func_choose

# Let the user make a choice about something and execute code based on
# the answer
# Called like: choose <default (y or n)> <prompt> <yes action> <no action>
# e.g. choose "y" \
#       "Do you want to play a game?" \
#       /usr/games/GlobalThermonuclearWar \
#       'printf "%b" "See you later Professor Falkin.\n"' >&2
# Returns: nothing
function choose {

    local default="$1"
    local prompt="$2"
    local choice_yes="$3"
    local choice_no="$4"
    local answer

    read -p "$prompt" answer
    [ -z "$answer" ] && answer="$default"

    case "$answer" in
        [yY1] ) eval "$choice_yes"
            # error check
            ;;
        [nN0] ) eval "$choice_no"
            # error check
            ;;
        *     ) printf "%b" "Unexpected answer '$answer'!" >&2 ;;
    esac
} # end of function choose

If the actions are complicated, use the function in Example 3-2 and handle the results in your main code.

# cookbook filename: func_choice.1

# Let the user make a choice about something and return a standardized
# answer. How the default is handled and what happens next is up to
# the if/then after the choice in main.
# Called like: choice <prompt>
# e.g. choice "Do you want to play a game?"
# Returns: global variable CHOICE
function choice {

    CHOICE=''
    local prompt="$*"
    local answer

    read -p "$prompt" answer
    case "$answer" in
        [yY1] ) CHOICE='y';;
        [nN0] ) CHOICE='n';;
        *     ) CHOICE="$answer";;
    esac
} # end of function choice

The code in Example 3-3 calls the choice function to prompt for and verify a package date. Assuming $THISPACKAGE is set, the function displays the date and asks for verification. If the user types y, Y, or presses Enter, then that date is accepted. If the user enters a new date, the function loops and verifies it (for a different treatment of this problem, see Recipe 11.7).

# cookbook filename: func_choice.2
CHOICE=''
until [ "$CHOICE" = "y" ]; do
    printf "%b" "This package's date is $THISPACKAGE\n" >&2
    choice "Is that correct? [Y/,<New date>]: "
    if [ -z "$CHOICE" ]; then
        CHOICE='y'
    elif [ "$CHOICE" != "y" ]; then
        printf "%b" "Overriding $THISPACKAGE with $CHOICE\n"
        THISPACKAGE=$CHOICE
    fi
done

# Build the package here

Next we’ll show different ways to handle some yes or no questions. Carefully read the prompts and look at the defaults. In both cases the user can simply hit the Enter key, and the script will then take the default the programmer intended:

# If the user types anything except a case-insensitive 'n', they will
# see the error log
choice "Do you want to look at the error logfile? [Y/n]: "
if [ "$CHOICE" != "n" ]; then
    less error.log
fi

# If the user types anything except a case-insensitive 'y', they will
# not see the message log
choice "Do you want to look at the message logfile? [y/N]: "
if [ "$CHOICE" = "y" ]; then
    less message.log
fi

Finally, the function in Example 3-4 asks for input that might not exist.

# cookbook filename: func_choice.3

choice "Enter your favorite color, if you have one: "
if [ -n "$CHOICE" ]; then
    printf "%b" "You chose: $CHOICE\n"
else
    printf "%b" "You do not have a favorite color.\n"
fi

Discussion

Asking the user to make a decision is often necessary in scripting. For getting arbitrary input, see Recipe 3.5. For choosing an option from a list, see Recipe 3.7.

If the possible choices and the code to handle them are fairly straightforward, the first self-contained function is easier to use, but it’s not always flexible enough. The second function is flexible at the expense of having to do more in the main code.

Note that we’ve sent the user prompts to STDERR so that the main script output on STDOUT may be redirected without the prompts cluttering it up.

See Also

• Recipe 3.5, “Getting User Input”

• Recipe 3.7, “Selecting from a List of Options”

• Recipe 11.7, “Figuring Out Date and Time Arithmetic”

Problem

You need to provide the user with a list of options to choose from and you don’t want to make them type any more than necessary.

Solution

Use bash’s builtin select construct to generate a menu, then have the user choose by typing the number of the selection (see Example 3-5).

# cookbook filename: select_dir

directorylist="Finished $(for i in /*;do [ -d "$i" ] && echo $i; done)"

PS3='Directory to process? ' # Set a useful select prompt
until [ "$directory" == "Finished" ]; do

    printf "%b" "\a\n\nSelect a directory to process:\n" >&2
    select directory in $directorylist; do

        # User types a number which is stored in $REPLY, but select
        # returns the value of the entry
        if [ "$directory" == "Finished" ]; then
            echo "Finished processing directories."
            break
        elif [ -n "$directory" ]; then
            echo "You chose number $REPLY, processing $directory..."
            # Do something here
            break
        else
            echo "Invalid selection!"
        fi # end of handle user's selection

    done # end of select a directory
done # end of until dir == finished

Discussion

The select statement makes it trivial to present a numbered list to the user on STDERR, from which they may make a choice. Don’t forget to provide an “exit” or “finished” choice, though Ctrl-D will end the select and empty input will print the menu again.

The number the user typed is returned in $REPLY, and the value of that entry is returned in the variable you specified in the select construct.

See Also

• help select

• help read

• Recipe 3.6, “Getting Yes or No Input”

Problem

You need to prompt the user for a password, but you don’t want it echoed on the screen.

Solution

Use the read command to read the user’s input, but with a special option to turn off echoing:

read -s -p "password: " PASSWD
printf "%b" "\n"

Discussion

The -s option tells the read command not to echo the characters typed (s is for silent) and the -p option says that the next argument is the prompt to be displayed prior to reading input.

The line of input that is read from the user is put into the variable named $PASSWD.

We follow the read with a printf to print out a newline. The printf is necessary because read -s turns off the echoing of characters. With echoing disabled, when the user presses the Enter key no newline is echoed and any subsequent output would appear on the same line as the prompt. Printing the newline gets us to the next line, as you would expect. It may even be handy for you to write the code all on one line to avoid intervening logic (putting it on one line also prevents mistakes should you cut and paste this line elsewhere):

read -s -p "password: " PASSWD ; printf "%b" "\n"

Be aware that if you read a password into an environment variable it is in memory in plain text, and thus may be accessed via a core dump or /proc/core (if your OS provides /proc/). It is also in the process environment, which may be accessible by other processes. You may be better off using certificates with SSH, if possible. In any case, it is wise to assume that root and possibly other users on the machine may gain access to the password, so you should handle the situation accordingly.

See Also

• help read

• Recipe 10.6, “Trapping Interrupts”

• Recipe 14.14, “Leaking Passwords into the Process List”

• Recipe 14.20, “Using Passwords in Scripts”

• Recipe 14.21, “Using SSH Without a Password”

• Recipe 19.9, “Making Your Terminal Sane Again”

Problem

You need to run a command on a Linux or Unix system.

Solution

Use bash and type the name of the command at the prompt:

$ someprog

Discussion

This seems rather simple, and in a way it is, but a lot goes on behind the scenes that you never see. What’s important to understand about bash is that its basic operation is to load and execute programs. All the rest is just window dressing to get ready to run programs. Sure, there are shell variables and control statements for looping and if/then/else branching, and there are ways to control input and output, but they are all icing on the cake of program execution.

So where does it get the program to run?

bash uses a shell variable called $PATH to locate your executable. The $PATH variable is a list of directories. The directories are separated by colons (:). bash searches in each of those directories for a file with the name that you specified. The order of the directories is important—bash looks at the order in which the directories are listed in the variable, and takes the first executable found:

$ echo $PATH
/bin:/usr/bin:/usr/local/bin:.
$

In the $PATH variable shown here, four directories are included. The last directory in the list is just a single dot (called the dot directory, or just dot), which represents the current directory on a Linux or Unix filesystem—wherever you are, that’s the directory to which dot refers. For example, when you copy a file from someplace to dot (i.e., cp /other/place/file . ), you are copying the file into the current directory. Listing the dot directory in your path tells bash to look for commands not just in those other directories, but also in the current directory (.).

Many people feel that putting dot in the $PATH is too great a security risk—someone could trick you and get you to run their own malicious version of a command (say, ls) in place of one that you were expecting. If dot were listed first, then someone else’s version of ls would supersede the normal ls command, and you might unwittingly run that command. Don’t believe us? Try this:

$ bash
$ cd
$ touch ls
$ chmod 755 ls
$ PATH=".:$PATH"
$ ls
$

Suddenly, the ls appears not to work in your home directory. You get no output. When you cd to some other location (e.g., cd /tmp), then ls will work, but not in your home directory. Why? Because in that directory there is an empty file called ls that is run (and does nothing—it’s empty) instead of the normal ls command located at /bin/ls. Since we started this example by running a new copy of bash, you can exit from this mess by exiting this subshell—but you might want to remove the bogus ls command first:

$ cd
$ rm ls
$ exit
$

Can you see the potential danger of wandering into a strange directory with your $PATH set to search the dot directory before anywhere else?

If you put dot as the last directory in your $PATH variable, at least you won’t be tricked that easily. Of course, if you leave it off altogether it is arguably even safer, and you can still run commands in your local directory by typing a leading dot and slash character, as in:

./myscript

The choice is yours.

Don’t forget to set execute permissions on the file before you invoke your script:

chmod +x myscript

You only need to set the permissions once. Thereafter, you can invoke the script as a command.

A common practice among some bash users is to create a personal bin directory, analogous to the system directories /bin and /usr/bin where executables are kept. In your personal bin (if you create it in your home directory, its path is ~/bin) you can put copies of your favorite shell scripts and other customized or private commands. Then add that directory to your $PATH, even to the front (PATH=~/bin:$PATH). That way, you can still have your own customized favorites without the security risk of running commands from strangers.

See Also

• Chapter 16 for more on customizing your environment

• Recipe 1.5, “Finding and Running Commands”

• Recipe 14.9, “Finding World-Writable Directories in Your $PATH”

• Recipe 14.10, “Adding the Current Directory to the $PATH”

• Recipe 16.11, “Keeping a Private Stash of Utilities by Adding ~/bin”

• Recipe 19.1, “Forgetting to Set Execute Permissions”

Problem

You need to run several commands, but some take a while and you don’t want to wait for each one to finish before issuing the next command.

Solution

There are three solutions to this problem, although the first is rather trivial: just keep typing. A Linux or Unix system is advanced enough to be able to let you type while it works on your previous commands, so you can simply keep typing one command after another.

Another rather simple solution is to type those commands into a file and then tell bash to execute the commands in the file—i.e., a simple shell script. For example, assume that we want to run three commands, long, medium, and short, each of whose execution time is reflected in its name. We need to run them in that order, but don’t want to wait around for the long script to finish before typing the other commands. We could use a shell script (a.k.a. batch file). Here’s a primitive way to do that:

$ cat > simple.script
long
medium
short
^D                      # Ctrl-D, not visible
$ bash ./simple.script

The third, and arguably best, solution is to run each command in sequence. If you want to run each program regardless of whether the preceding ones fail, separate them with semicolons:

long ; medium ; short

If you only want to run the next program if the preceding program worked, and all the programs correctly set exit codes, separate them with double ampersands:

long && medium && short

Discussion

The cat example was just a very primitive way to enter text into a file: we redirected the output from the command into the file named simple.script (for more on redirecting output, see Chapter 2). Better you should use a real editor, but such things are harder to show in examples like this. From now on, when we want to show a script, we’ll either just show the text as disembodied text not on a command line, or start the example with a command like cat filename to dump the contents of the file to the screen (rather than redirecting output from our typing into the file), and thus display it in the example.

The main point of this simple solution is to demonstrate that more than one command can be put on the bash command line. In the first case the second command isn’t run until the first command exits, the third doesn’t execute until the second exits, and so on, for as many commands as you have on the line. In the second case the second command isn’t run unless the first command succeeds, the third doesn’t execute unless the second succeeds, and so on, for as many commands as you have on the line.

See Also

• Chapter 2 on redirecting output

Problem

You need to run three commands, but they are independent of each other and don’t need to wait for the previous ones to complete.

Solution

You can run a command in the background by putting an ampersand (&) at the end of the command line. Thus, you could fire off all three commands in rapid succession as follows:

$ long &
[1] 4592
$ medium &
[2] 4593
$ short
$

Or better yet, you can do it all on one command line:

$ long & medium & short
[1] 4592
[2] 4593
$

Discussion

When we run a command “in the background” (there really is no such place in Linux), all that really means is that we disconnect keyboard input from the command and the shell doesn’t wait for the command to complete before it gives another prompt and accepts more command input. Output from the command (unless we take explicit action to change this behavior) will still come to the screen, so in this example all three commands will be interspersing output to the screen.

The odd bits of numerical output are the job number in square brackets, followed by the process ID of the command that we just started in the background. In our example, job 1 (process 4592) is the long command, and job 2 (process 4593) is medium.

We didn’t put short into the background since we didn’t put an ampersand at the end of the line, so bash will wait for it to complete before giving us the shell prompt (the $).

The job number or process ID can be used to provide limited control over a job. For example, we could kill the long job with kill %1 (since its job number was 1), or we could specify the process number (i.e., kill 4592) with the same deadly results.

You can also use the job number to reconnect to a background job. For instance, we could connect the long job back to the foreground with fg %1. If you only have one job running in the background, you don’t even need the job number; just use fg by itself.

Problem

You need to know whether the command you ran succeeded.

Solution

The shell variable $? is set with a nonzero value if the command fails—provided that the programmer who wrote that command or shell script followed the established convention:

$ somecommand
# it works...
$ echo $?
0
$ badcommand
# it fails...
$ echo $?
1
$

Discussion

The exit status of a command is kept in the shell variable referenced with $?. Its value can range from 0 to 255. When you write a shell script, it’s a good idea to have your script exit with zero if all is well and a nonzero value if you encounter an error condition. We recommend using only 0 to 127 because the shell uses 128+N to denote killed by signal N. Also, if you use a number greater than 255 or less than 0, the numbers will wrap around. You return an exit status with the exit statement (e.g., exit 1 or exit 0). But be aware that you only get one shot at reading a command’s exit status:

$ badcommand
# it fails...
$ echo $?
1
$ echo $?
0
$

Why does the second echo give us 0 as a result? It’s actually reporting on the status of the immediately preceding echo command. The first time we typed echo $? it returned a 1, which was the return value of badcommand. But the echo command itself succeeds, and therefore the new, most recent status is success (i.e., a 0 value). Because you only get one chance to check the exit status, many shell scripts will immediately assign the status to another shell variable, as in:

$ badcommand
# it fails...
$ STAT=$?
$ echo $STAT
1
$ echo $STAT
1
$

We can keep the value around in the variable $STAT and check its value later on.

Although we’re showing this in command-line examples, the real use of variables like $? comes in writing scripts. You can usually see whether a command worked or not if you are watching it run on your screen. But in a script, the commands may be running unattended.

One of the great features of bash is that the scripting language is identical to commands as you type them at a prompt in a terminal window. This makes it much easier to check out syntax and logic as you write your scripts.

The exit status is more often used in scripts, and often in if statements, to take different actions depending on the success or failure of a command. Here’s a simple example for now, but we will revisit this topic in future recipes:

somecommand
...
if (( $? )) ; then echo failed ; else echo OK; fi

(( )) evaluates an arithmetic expression; see Recipes 6.1 and 6.2.

We also do not recommend using negative numbers. The shell will accept them without an error, but it won’t do what you expect:

$ bash -c 'exit -2' ; echo $?
254

$ bash -c 'exit -200' ; echo $?
56

See Also

• Recipe 4.5, “Running a Command Only if Another Command Succeeded”

• Recipe 4.8, “Displaying Error Messages When Failures Occur”

• Recipe 6.1, “Doing Arithmetic in Your Shell Script”

• Recipe 6.2, “Branching on Conditions”

Problem

You need to run some commands, but you only want to run certain commands if certain other ones succeed. For example, you’d like to change directories (using the cd command) into a temporary directory and remove all the files. However, you don’t want to remove any files if the cd fails (e.g., if permissions don’t allow you into the directory, or if you spell the directory name wrong).

Solution

You can use the exit status ($?) of the cd command in combination with an if statement to do the rm only if the cd was successful:

cd mytmp
if (( $? == 0 )); then rm * ; fi

if cd mytmp; then rm * ; fi

Discussion

Obviously, you wouldn’t need to do this if you were typing the commands by hand. You would see any error messages from the cd command, and thus you wouldn’t type the rm command. But scripting is another matter, and this test is very well worth doing in a script like our example to make sure that you don’t accidentally erase all the files in the directory where you are running it.

Let’s say you ran that script from the wrong directory, one that didn’t have a subdirectory named mytmp. The cd would fail, so the current directory would remain unchanged. Without the if check (for the cd having failed) the script would just continue on to the next statement. Running the rm * would remove all the files in your current directory. Ouch. The if is worth it.

So how does $? get its value? It is the exit code of the command (see Recipe 4.4). C language programmers will recognize this as the value of the argument supplied to the exit() function; e.g., exit(4); would return a 4. For the shell, an exit code of zero is considered success and a nonzero value means failure.

If you’re writing bash scripts, you’ll want to be sure to explicitly set return values, so that $? is set properly from your script. If you don’t, the value set will be the value of the last command run, which you may not want as your result.

See Also

• Recipe 4.4, “Telling Whether a Command Succeeded or Not”

• Recipe 4.6, “Using Fewer if Statements”

Problem

As a conscientious programmer, you took to heart what we described in the previous recipe. You applied the concept to your latest shell script, but now you find that the shell script is unreadable, with all those if statements checking the return code of every command. Isn’t there an alternative?

Solution

Use the double-ampersand operator in bash to provide conditional execution:

cd mytmp && rm *

Discussion

Separating two commands by the double ampersands tells bash to run the first command and then to run the second command only if the first command succeeds (i.e., its exit status is 0). This is very much like using an if statement to check the exit status of the first command in order to protect the running of the second command:

cd mytmp
if (( $? == 0 )); then rm * ; fi

The double-ampersand syntax is meant to be reminiscent of the logical AND operator in the C language. If you know your logic (and your C) then you’ll recall that if you are evaluating the logical expression A AND B, the entire expression can only be true if both (sub)expression A and (sub)expression B evaluate to true. If either one is false, the whole expression is false. The C language makes use of this fact, and when you code an expression like if (A && B) { ... }, it will evaluate expression A first. If it is false, it won’t even bother to evaluate B since the overall outcome (false) has already been determined (by A being false).

So what does this have to do with bash? Well, if the exit status of the first command (the one to the left of the &&) is nonzero (i.e., failed), then it won’t bother to evaluate the second expression—it won’t run the other command at all.

If you want to be thorough about your error checking, but don’t want if statements all over the place, you can have bash exit any time it encounters a failure (i.e., a nonzero exit status) from every command in your script (except in while loops and if statements where it is already capturing and using the exit status) by setting the -e flag:

set -e
cd mytmp
rm *

Setting the -e flag will cause the shell to exit when a command fails. If the cd in this example fails, the script will exit and never even try to execute the rm * command. We don’t recommend doing this on an interactive shell, however, because when the shell exits it will make your shell window go away.

See Also

• Recipe 4.8, “Displaying Error Messages When Failures Occur” for an explanation of the || syntax, which is similar in some ways to but also quite different from the && construct

Problem

You ran a job in the background, then exited the shell and went for coffee. When you came back to check, the job was no longer running and it hadn’t completed. In fact, your job hadn’t progressed very far at all. It seems to have quit as soon as you exited the shell.

Solution

If you want to run a job in the background and expect to exit the shell before the job completes, then you need to nohup the job:

$ nohup long &
nohup: appending output to `nohup.out'
$

Discussion

When you put a job in the background (via the &, as described in Recipe 4.3), it is still a child process of the bash shell. When you exit an instance of the shell, bash sends a hangup (hup) signal to all of its child processes. That’s why your job didn’t run for very long. As soon as you exited bash, it killed your background job. (Hey, you were leaving; how was it supposed to know?)

The nohup command simply sets up the child process to ignore hangup signals. You can still kill the job with the kill command, because kill sends a SIGTERM signal, not a SIGHUP signal. But with nohup, bash won’t inadvertently kill your job when you exit.

The message that nohup gives about appending your output is just nohup trying to be helpful. Since you are likely to exit the shell after issuing a nohup command, your output destination will likely go away—i.e., the bash session in your terminal will no longer be active, so the job won’t be able to write to STDOUT. More importantly, writing to a nonexistent destination would cause a failure. So nohup redirects the output for you, appending it (not overwriting, but adding at the end) to a file named nohup.out in the current directory. You can explicitly redirect the output elsewhere on the command line, and nohup is smart enough to detect that this has happened and not use nohup.out for your output.

See Also

• Chapter 2 for various recipes on redirecting output, since you probably want to do that for a background job

• Recipe 4.3, “Running Several Commands All at Once”

• Recipe 10.1, ““Daemon-izing” Your Script” for more on running your script unattended

• Recipe 17.4, “Recovering Disconnected Sessions Using screen”

Problem

You need your shell script to be verbose about failures. You want to see error messages when commands don’t work, but if statements tend to distract from the visual flow of statements.

Solution

A common idiom among some shell programmers is to use the || with commands to spit out debug or error messages. Here’s an example:

cmd || printf "%b" "cmd failed. You're on your own\n"

Discussion

Similar to how the && in Recipe 4.6 tells bash not to bother to evaluate the second expression if the first one is false, the || tells the shell not to bother to evaluate the second expression if the first one is true (i.e., succeeds). As with &&, the || syntax harkens back to logic and the C language, where the outcome is determined (as true) if the first expression in A OR B evaluates to true—so there’s no need to evaluate the second expression. In bash, if the first expression returns 0 (i.e., succeeds) then it just continues on. Only if the first expression returns a nonzero value (i.e., if the exit value of the command indicates failure) must it evaluate the second part, and thus run the other command.

Warning—don’t be fooled by this:

cmd || printf "%b" "FAILED.\n" ; exit 1

The exit will be executed in either case! The OR is only between the first two commands. If we want to have the exit happen only on error, we need to group it with the printf so that both are considered as a unit. The desired syntax would be:

cmd || { printf "%b" "FAILED.\n" ; exit 1 ; }

Note that the semicolon after the last command and just before the } is required, and that the closing brace must be separated by whitespace from the surrounding text. See Recipe 2.14 for a discussion.

See Also

• Recipe 2.14, “Saving or Grouping Output from Several Commands”

• Recipe 4.6, “Using Fewer if Statements” for an explanation of && syntax

Problem

You want to run different commands in your script depending on circumstances. How can you vary which commands run?

Solution

There are many solutions to this problem—it’s what scripting is all about. In coming chapters we’ll discuss various programming constructs that can be used to solve this problem, such as if/then/else, case statements, and more. But here’s a slightly different approach that reveals something about bash. We can use the contents of a variable (more on those in Chapter 5) not just for parameters, but also for the command itself:

FN=/tmp/x.x
PROG=echo
$PROG $FN
PROG=cat
$PROG $FN

Discussion

We can assign the program name to a variable (here we use $PROG), and then when we refer to that variable in the place where a command name would be expected, bash uses the value of that variable ($PROG) as the command to run. It parses the command line, substitutes the values of its variables, and takes the result of all the substitutions and treats that as the command line, as if it had been typed that way verbatim.

See Also

• Chapter 11

• Recipe 14.3, “Setting a Secure $PATH”

• Recipe 16.21, “Creating Self-Contained, Portable rc Files”

• Recipe 16.22, “Getting Started with a Custom Configuration”

• Appendix C for a description of all the various substitutions that are performed on a command line; you’ll want to read a few more chapters before tackling that subject

Problem

You want to run a series of scripts, but the list keeps changing; you’re always adding new scripts, but you don’t want to continuously modify a master list.

Solution

Put the scripts you want to run in a directory, and let bash run everything that it finds. Instead of keeping a master list, simply use the contents of that directory as your master list. Here’s a script that will run everything it finds in a particular directory:

for SCRIPT in /path/to/scripts/dir/*
do
    if [ -f "$SCRIPT" -a -x "$SCRIPT" ]
    then
        $SCRIPT
    fi
done

Discussion

We discuss the for loop and the if statement in greater detail in Chapter 6, but this gives you a taste. The variable $SCRIPT will take on successive values for each file that matches the wildcard pattern *, which matches everything in the named directory (except invisible dot files, whose names begin with a period). If it is a file (the -f test) and has execute permissions set (the -x test), the shell will then try to run that script.

In this simple example, we have provided no way to specify any arguments to the scripts as they are executed. This simple script may work well for your personal needs, but wouldn’t be considered robust; some might consider it downright dangerous. But we hope it gives you an idea of what lies ahead: some programming language–style scripting capabilities.

See Also

• Chapter 6 for more about for loops and if statements

Problem

Before we say one more word about shell scripts or variables, we have to say something about documenting your scripts. After all, you need to be able to understand your script even when several months have passed since you wrote it.

Solution

Document your script with comments. The # character denotes the beginning of a comment. All the characters after it on that line are ignored by the shell:

#
# This is a comment.
#
# Use comments frequently.
# Comments are your friends.

Discussion

Some people have described shell syntax, regular expressions, and other parts of shell scripting as write-only syntax, implying that it is nearly impossible to understand the intricacies of many shell scripts.

One of your best defenses against letting your shell scripts fall into this trap is the liberal use of comments (another is the use of meaningful variable names). It helps to put a comment before strange syntax or terse expressions:

# replace the semicolon with a space
NEWPATH=${PATH/;/ }
#
# switch the text on either side of a semicolon
sed -e 's/^\(.*\);\(.*\)$/\2;\1/' < $FILE

Comments can even be typed in at the command prompt with an interactive shell. This feature can be turned off, but it is on by default. There may be a few occasions when it is useful to make interactive comments.

Problem

You want a simple way to provide formatted end-user documentation (e.g., manpages or HTML pages) for your script. You want to keep both code and documentation markup in the same file to simplify updates, distribution, and revision control.

Solution

Embed documentation in the script using the “do nothing” builtin (a colon) and a here-document, as illustrated in Example 5-1.

#!/usr/bin/env bash
# cookbook filename: embedded_documentation

echo 'Shell script code goes here'

# Use a : NOOP and here document to embed documentation,
: <<'END_OF_DOCS'

Embedded documentation such as Perl's Plain Old Documentation (POD),
or even plain text here.

Any accurate documentation is better than none at all.

Sample documentation in Perl's Plain Old Documentation (POD) format adapted from
CODE/ch07/Ch07.001_Best_Ex7.1 and 7.2 in the Perl Best Practices example tarball
"PBP_code.tar.gz".

=head1 NAME

MY~PROGRAM--One-line description here

=head1 SYNOPSIS

  MY~PROGRAM [OPTIONS] <file>

=head1 OPTIONS

  -h = This usage.
  -v = Be verbose.
  -V = Show version, copyright, and license information.


=head1 DESCRIPTION

A full description of the application and its features.
May include numerous subsections (i.e., =head2, =head3, etc.)

[...]


=head1 LICENSE AND COPYRIGHT

=cut

END_OF_DOCS

Then, to extract and use that POD documentation, try these commands:

# To read on-screen, automatically paginated
$ perldoc myscript

# Just the "usage" sections
$ pod2usage myscript

# Create an HTML version
$ pod2html myscript > myscript.html

# Create a manpage
$ pod2man myscript > myscript.1

Discussion

Any plain-text documentation or markup can be used this way, either interspersed throughout the code, or better yet, collected at the end of the script. Since computer systems that have bash will probably also have Perl, its Plain Old Documentation (POD) format may be a good choice. Perl usually comes with pod2* programs to convert POD to HTML, LaTeX, manpage, text, and usage files.

Damian Conway’s Perl Best Practices (O’Reilly) has some excellent library module and application documentation templates that could be easily translated into any documentation format, including plain text. See CODE/ch07/Ch07.001_Best_Ex7.1 and 7.2 in that book’s examples tarball.

If you keep all of your embedded documentation at the very bottom of the script, you could also add an exit 0 right before the documentation begins. That will simply exit the script rather than forcing the shell to parse each line looking for the end of the here-document, so it will be a little faster. You need to be careful not to override a previous exit code from a command that failed, though, so consider using set -e. And do not use this trick if you intersperse code and embedded documentation in the body of the script.

See Also

• set -e in Recipe 4.6, “Using Fewer if Statements”

• http://examples.oreilly.com/perlbp/PBP_code.tar.gz

Problem

You’d like to make your script as readable as possible for ease of understanding and future maintenance.

Solution

Follow these best practices:

• Document your script as noted in Recipe 5.1 and Recipe 5.2.

• Indent and use vertical whitespace wisely.

• Use meaningful variable names.

• Use functions (Recipe 10.4), and give them meaningful names.

• Break lines at meaningful places at less than 76 characters or so.

• Put the most meaningful bits to the left.

Discussion

Document your intent, not the trivial details of the code. If you follow the rest of the points, the code should be pretty clear. Write reminders, provide sample data layouts or headers, and make a note of all the details that are in your head now, as you write the code. Document the code itself too if it is subtle or obscure.

We recommend indenting using four spaces per level, with no tabs and especially no mixed tabs and spaces. There are many reasons for this, though it often is a matter of personal preference or company standards. Four spaces is always four spaces, no matter how your editor (excepting proportional fonts) or printer is set. It’s big enough to be easily visible as you glance across the script but small enough that you can have several levels of indenting without running the lines off the right side of your screen or printed page. We also suggest indenting continued lines with two additional spaces, or as needed, to make the code the most clear.

Use vertical whitespace, with separators if you like them, to create blocks of similar code. Of course, you’ll do that with functions as well.

Use meaningful names for variables and functions, and spell them out. The only time $i or $x is ever acceptable is in a for loop. You may think that short, cryptic names are saving you time and typing now, but we guarantee that you will lose that time 10- or 100-fold somewhere down the line when you have to fix or modify your script.

Break long lines at around 76 characters. Yes, we know that most screens (or rather, terminal programs) can handle a lot more than that, but 80-character paper and screens are still the default, and it never hurts to have some whitespace to the right of the code. Constantly having to scroll to the right or having lines wrap awkwardly on the screen or printout is annoying and distracting. Don’t cause it.

Unfortunately, there are sometimes exceptions to the long line rule. When creating lines to pass elsewhere, perhaps via Secure Shell (SSH), and in certain other cases, breaking up the line can cause many more code headaches than it solves. But in most cases, it makes sense.

Try to put the most meaningful bits to the left when you break a line—we read shell code left-to-right, so the unusual fact of a continued line will stand out more. It’s also easier to scan down the left edge of the code for continued lines, should you need to find them. Which is more clear?

# Good
[ -n "$results" ] \
  && echo "Got a good result in $results" \
  || echo 'Got an empty result, something is wrong'

# Also good
[ -n "$results" ] && echo "Got a good result in $results" \
                  || echo 'Got an empty result, something is wrong'

# OK, but not ideal
[ -n "$results" ] && echo "Got a good result in $results" \
  || echo 'Got an empty result, something is wrong'

# Bad
[ -n "$results" ] && echo "Got a good result in $results" || \
echo 'Got an empty result, something is wrong'

# Bad (trailing \s are optional here, but recommended for clarity)
[ -n "$results" ] && \
  echo "Got a good result in $results" || \
  echo 'Got an empty result, something is wrong'

See Also

• Recipe 5.1, “Documenting Your Script”

• Recipe 5.2, “Embedding Documentation in Shell Scripts”

• Recipe 10.4, “Defining Functions”

Problem

You need to print a variable along with other text. You are using the dollar sign in referring to the variable, but how do you distinguish the end of the variable name from other text that follows? For example, say you wanted to use a shell variable as part of a filename, as in:

for FN in 1 2 3 4 5
do
    somescript /tmp/rep$FNport.txt
done

How will the shell read that? It will think that the variable name starts with the $ and ends with the punctuation. In other words, it will think that $FNport is the variable name, not the intended $FN.

Solution

Use the full syntax for a variable reference, which includes not just the dollar sign, but also braces around the variable name:

somescript /tmp/rep${FN}port.txt

Discussion

Because shell variables can contain only alphanumeric characters and the underscore, there are many instances where you won’t need to use the braces. Any whitespace or punctuation (except the underscore) provides enough of a clue to where the variable name ends. But when in doubt, use the braces. In fact, some people would argue that always using the braces is a good habit so you never have to worry about when they are needed or not, and provides a consistent look throughout your scripts. Others find that to be too much typing of characters that are optional but awkward to reach, and think they can make the code look very busy or noisy. Ultimately, it’s a matter of personal preference.

See Also

• Recipe 1.8, “Using Shell Quoting”

Problem

You defined a variable in one script, but when you called another script it didn’t know about the variable.

Solution

Export variables that you want to pass on to other scripts:

export MYVAR
export NAME=value

Discussion

Sometimes it’s a good thing that one script doesn’t know about the other script’s variables. If you called a shell script from within a for loop in the first script, you wouldn’t want the second script messing up the iterations of your for loop (which it probably can’t do anyway since it’s almost certainly running in a subshell, but work with us here).

But sometimes you do want the information passed along. In those cases, you can export the variable so that its value is passed along to any other program that the script invokes.

If you want to see a list of all the exported variables, just type the command env (or use the builtin export -p) for a list of each variable and its value. All of these are available for your script when it runs. Many will have already been set up by the bash startup scripts (see Chapter 16 for more on configuring and customizing bash).

You can make the export part of any variable assignment, though that won’t work in old versions of the shell. You can also have the export statement just name the variable that will be exported. Though the export statement can be put anywhere prior to where you need the value to be exported, script writers often group these statements together, like variable declarations, at the top of a script.

Once exported, you can assign repeatedly to the variable without exporting it each time. So, sometimes you’ll see statements like:

export FNAME
export SIZE
export MAX
...
MAX=2048
SIZE=64
FNAME=/tmp/scratch

and at other times you’ll see:

export FNAME=/tmp/scratch
export SIZE=64
export MAX=2048
...
FNAME=/tmp/scratch2
...
FNAME=/tmp/stillexported

One word of caution: the exported variables are, in effect, call by value. Changing the value of the exported variable in the called script does not change that variable’s value back in the calling script.

This begs the question, “How would you pass back a changed value from the called script?” Answer: you can’t.

You can only design your scripts so that they don’t need to do this. What mechanisms have people used to cope with this limitation?

One approach might be to have the called script echo its changed value as output from the script, letting you read the output with the resulting changed value. For example, suppose one script exports a variable $VAL and then calls another script that modifies $VAL. To get the new value of $VAL in the original script, you have to write the changed value to standard output, capture it, and assign it to the variable, as in:

VAL=$(anotherscript)

(See Recipe 10.5 for an explanation of the $() syntax.) You could even change multiple values and echo them each in turn to standard output. The calling program could then use a shell read to capture each line of output one at a time into the appropriate variables. This requires that the called script write no other output to standard output (at least not before or among the variables), however, and sets up a very strong interdependency between the scripts (not good from a maintenance standpoint).

Problem

How can you see which variables have been exported and what values they have? Do you have to echo each one by hand? How can you tell if they are exported?

Solution

Use the set command to see the values of all variables and function definitions in the current shell.

Use the env (or export -p) command to see only those variables that have been exported and would be available to a subshell.

In bash version 4 or newer, you can also use the declare -p command.

Discussion

The set command, with no other arguments, produces (on standard output) a list of all the shell variables currently defined along with their values, in a name=value format. The env command is similar. If you run either, you will find a rather long list of variables, many of which you might not recognize. Those variables have been created for you, as part of the shell’s startup process.

The list produced by env is a subset of the list produced by set, since not all variables are exported.

If there are particular variables or values that are of interest, and you don’t want the entire list, just pipe it into a grep command. For example:

set | grep MY

will show only those variables whose name or value has the two-character sequence MY somewhere in it.

The output from the newer declare -p command shows the variable names and values as if they were being declared and initialized. Here is a snippet of output:

$ declare -p
...
declare -i MYCOUNT="5"
declare -x MYENV="10.5.1.2"
declare -r MYFIXED="unchangeable"
declare -a MYRA=([0]="5" [1]="10" [2]="15")
...
$

The output is in the form of declare statements that could be used as source code in a shell script to recreate these variables and their values. The various arguments (-i, -x, -r, -a) indicate that the variable is an integer, has been exported, is read-only, or is an array, respectively.

See Also

• help set

• help export

• help declare

• man env

• Chapter 16 for more on configuring and customizing bash

• Appendix A for reference lists for all of the built-in shell variables

Problem

You want users to be able to invoke your script with a parameter. You could require that users set a shell variable, but that seems clunky. You also need to pass data to another script. You could agree on environment variables, but that ties the two scripts together too closely.

Solution

Use command-line parameters. Any words on the command line that follow the name of a shell script are available to the script as numbered variables. Suppose we have the following script, simplest.sh:

# simple shell script
echo $1

The script will echo the first parameter supplied on the command line when it is invoked. Here it is in action:

$ cat simplest.sh
# simple shell script
echo ${1}
$ ./simplest.sh you see what I mean
you
$ ./simplest.sh one more time
one
$

Discussion

The other parameters are available as ${2}, ${3}, ${4}, ${5}, and so on. You don’t need the braces for the single-digit numbers, except to separate the variable name from the surrounding text. Typical scripts have only a handful of parameters, but when you get to ${10} you need to use the braces because the shell will interpret $10 as ${1} followed immediately by the literal string 0, as we see here:

$ cat tricky.sh
echo $1 $10 ${10}
$ ./tricky.sh I II III IV V VI VII VIII IX X XI
I I0 X
$

The tenth argument has the value X, but if you write $10 in your script, the shell will give you $1, the first parameter, followed immediately by a zero, the literal character that you put next to the $1 in your echo statement.

See Also

• Recipe 5.4, “Separating Variable Names from Surrounding Text”

Problem

You want to take some set of actions for a given list of arguments. You could write your shell script to perform those actions for one argument and use $1 to reference the parameter. But what if you’d like to do this for a whole bunch of files? You would like to be able to invoke your script like this:

./actall *.txt

knowing that the shell will pattern match and build a list of filenames that match the *.txt pattern (any filename ending with .txt).

Solution

Use the shell special variable $* to refer to all of your arguments, and use that in a for loop as in Example 5-2.

#!/usr/bin/env bash
# cookbook filename: chmod_all.1
#
# change permissions on a bunch of files
#
for FN in $*
do
    echo changing $FN
    chmod 0750 $FN
done

Discussion

The variable $FN is our choice; we could have used any shell variable name we wanted there. The $* refers to all the arguments supplied on the command line. For example, if the user types:

./actall abc.txt another.txt allmynotes.txt

the script will be invoked with $1 equal to abc.txt, $2 equal to another.txt, and $3 equal to allmynotes.txt, but $* will be equal to the entire list. In other words, after the shell has substituted the list for $* in the for statement, it will be as if the script had read:

for FN in abc.txt another.txt allmynotes.txt
do
  echo changing $FN
  chmod 0750 $FN
done

The for loop will take the first value from the list, assign it to the variable $FN, and proceed through the list of statements between the do and the done. It will then repeat that loop for each of the other values.

But you’re not finished yet! This script works fine when filenames have no spaces in them, but sometimes you encounter filenames with spaces. Read the next two recipes to see how this script can be improved.

See Also

• help for

• Recipe 5.9, “Handling Parameters with Spaces”

• Recipe 5.10, “Handling Lists of Parameters with Spaces”

• Recipe 6.12, “Looping with a Count”

Problem

You wrote a script that took a filename as a parameter and it seemed to work, but then one time your script failed. The filename, it turns out, had an embedded space.

Solution

You’ll need to be careful to quote any shell parameters that might contain filenames. When referring to a variable, put the variable reference inside double quotes.

Discussion

Thanks a lot, Apple! Trying to be user-friendly, the designers popularized the concept of space characters as valid characters in filenames, so users could name their files with names like My Report and Our Dept Data instead of the ugly and unreadable MyReport and Our_Dept_Data. (How could anyone possibly understand what those old-fashioned names meant?) Well, that makes life tough for the shell, where the space is the fundamental separator between words, so filenames were always kept to a single word. Not so anymore.

So how do we handle this?

Where a shell script once had simply ls -l $1, it is better to write ls -l "$1", with quotes around the parameter. Otherwise, if the parameter has an embedded space, it will be parsed into separate words, and only part of the name will be in $1. Let’s show you how this doesn’t work:

$ cat simpls.sh
# simple shell script
ls -l ${1}
$
$ ./simple.sh Oh the Waste
ls: Oh: No such file or directory
$

If we don’t put quotes around the filename when we invoke the script, bash sees three arguments and substitutes the first argument (Oh) for $1. The ls command runs with Oh as its only argument and can’t find that file.

So now let’s put quotes around the filename when we invoke the script:

$ ./simpls.sh "Oh the Waste"
ls: Oh: No such file or directory
ls: the: No such file or directory
ls: Waste: No such file or directory
$

Still not good. bash has taken the three-word filename and substituted it for $1 on the ls command line in our script. So far so good. Since we don’t have quotes around the variable reference in our script, however, ls sees each word as a separate argument, (i.e., as separate filenames). Again, it can’t find any of them.

Let’s try a script that quotes the variable reference:

$ cat quoted.sh
# note the quotes
ls -l "${1}"
$
$ ./quoted.sh "Oh the Waste"
-rw-r--r--  1 smith users 28470 2007-01-11 19:22 Oh the Waste
$

When we quoted the reference "${1}" it was treated as a single word (a single file-name), and the ls then had only one argument—the filename—and it could complete its task.

See Also

• Chapter 19 for common goofs

• Recipe 1.8, “Using Shell Quoting” for tips on shell quoting

• Appendix C for more information on command-line processing

Problem

OK, you have quotes around your variable as the previous recipe recommended. But you’re still getting errors. It’s just like the script from Recipe 5.8, but it fails when a file has a space in its name:

for FN in $*
do
    chmod 0750 "$FN"
done

Solution

It has to do with the $* in the script, used in the for loop. For this case we need to use a different but related shell variable, $@. When it is quoted, the resulting list has quotes around each argument separately. The shell script should be written as shown in Example 5-3.

#!/usr/bin/env bash
# cookbook filename: chmod_all.2
#
# change permissions on a bunch of files
# with better quoting in case of filenames with spaces
#
for FN in "$@"
do
    chmod 0750 "$FN"
done

Discussion

The parameter $* expands to the list of arguments supplied to the shell script. If you invoke your script like this:

myscript these are args

then $* refers to the three arguments these are args. And when it’s used in a for loop, such as:

for FN in $*

the first time through the loop $FN is assigned the first word (these), the second time the second word (are), etc.

If the arguments are filenames and they are put on the command line by pattern matching, as when you invoke the script this way:

myscript *.mp3

then the shell will match all the files in the current directory whose names end with the four characters .mp3, and they will be passed to the script. So consider an example where there are three MP3 files whose names are:

vocals.mp3
cool music.mp3
tophit.mp3

The second song title has a space in the filename between cool and music. When you invoke the script with:

myscript *.mp3

you’ll get, in effect:

myscript vocals.mp3 cool music.mp3 tophit.mp3

If your script contains the line:

for FN in $*

that will expand to:

for FN in vocals.mp3 cool music.mp3 tophit.mp3

which has four words in its list, not three. The second song title has a space as the fifth character (cool music.mp3), and the space causes the shell to see that as two separate words (cool and music.mp3), so $FN will be cool on the second iteration through the for loop. On the third iteration $FN will have the value music.mp3, but that is not the name of your file either, so you’ll get file-not-found error messages.

It might seem logical to try quoting the $*, but this:

for FN in "$*"

will expand to:

for FN in "vocals.mp3 cool music.mp3 tophit.mp3"

and you will end up with a single value for $FN equal to the entire list. You’ll get an error message like this:

chmod: cannot access 'vocals.mp3 cool music.mp3 tophit.mp3': No such file or
directory

Instead, you need to use the shell variable $@ and quote it. Left unquoted, $* and $& give you the same thing. But when quoted, bash treats them differently. A reference to $* inside of quotes gives the entire list inside one set of quotes, as we just saw. But a reference to $@ inside of quotes returns not one string but a list of quoted strings, one for each argument.

In our example using the MP3 filenames, this:

for FN in "$@"

will expand to:

for FN in "vocals.mp3" "cool music.mp3" "tophit.mp3"

You can see that the second filename is now quoted, so that its space will be kept as part of its name and not considered a separator between two words.

The second time through this loop, $FN will be assigned the value cool music.mp3, which has an embedded space. So, be careful how you refer to $FN—you’ll probably want to put it in quotes too, so that the space in the filename is kept as part of that string and not used as a separator. That is, you’ll want to use "$FN", as in:

chmod 0750 "$FN"

Shouldn’t you always use "$@" in your for loop? Well, it’s a lot harder to type, so for quick-and-dirty scripts, when you know your filenames don’t have spaces, it’s probably OK to keep using the old-fashioned $* syntax. For more robust scripting though, we recommend "$@" as the safer way to go. We’ll probably use them interchangeably throughout this book, because even though we know better, old habits die hard—and some of us never use spaces in our filenames! (Famous last words.)

See Also

• Recipe 5.8, “Looping Over Arguments Passed to a Script”

• Recipe 5.9, “Handling Parameters with Spaces”

• Recipe 5.12, “Consuming Arguments”

• Recipe 6.12, “Looping with a Count”

Problem

You need to know how many parameters the script was invoked with.

Solution

Use the shell builtin variable ${}. Example 5-4 shows some scripting to enforce an exact count of three arguments.

#!/usr/bin/env bash
# cookbook filename: check_arg_count
#
# Check for the correct # of arguments:
# Use this syntax or use: if [ $# -lt 3 ]
if (( $# < 3 ))
then
    printf "%b" "Error. Not enough arguments.\n" >&2
    printf "%b" "usage: myscript file1 op file2\n" >&2
    exit 1
elif (( $# > 3 ))
then
    printf "%b" "Error. Too many arguments.\n" >&2
    printf "%b" "usage: myscript file1 op file2\n" >&2
    exit 2
else
    printf "%b" "Argument count correct. Proceeding...\n"
fi

And here is what it looks like when we run it, once with too many arguments and once with the correct number of arguments:

$ ./myscript myfile is copied into yourfile
Error. Too many arguments.
usage: myscript file1 op file2

$ ./myscript myfile copy yourfile
Argument count correct. Proceeding...

Discussion

After the opening comments (always a helpful thing to have in a script), we have the if test to see whether the number of arguments supplied (found in $#) is greater than three. If so, we print an error message, remind the user of the correct usage, and exit.

The output from the error messages is redirected to standard error. This is in keeping with the intent of standard error as the channel for all error messages.

The script also has a different return value depending on the error that was detected. While not that significant here, it is useful for any script that might be invoked by other scripts, so that there is a programmatic way not only to detect failure (a nonzero exit value), but to distinguish between error types.

One word of caution: don’t confuse ${#} with ${#VAR} or even ${VAR#alt} just because they all use the hash character (#) inside of braces. The first gives the number of arguments, whereas the second gives the length of the value in the variable VAR and the third does a certain kind of substitution.

See Also

• Recipe 4.4, “Telling Whether a Command Succeeded or Not”

• Recipe 5.1, “Documenting Your Script”

• Recipe 5.12, “Consuming Arguments”

• Recipe 5.18, “Changing Pieces of a String”

• Recipe 6.12, “Looping with a Count”

Problem

For any serious shell script, you are likely to have two kinds of arguments—options that modify the behavior of the script and the real arguments you want to work with. You need a way to get rid of the option arguments after you’ve processed them.

For example, you have this script:

for FN in "$@"
do
    echo changing $FN
    chmod 0750 "$FN"
done

It’s simple enough—it echoes the filename that it is working on, then it changes that file’s permissions. But you want it to work quietly sometimes, not echoing the filename. How can you add an option to turn off this verbose behavior while preserving the for loop?

Solution

Use shift to remove an argument after you’ve handled it, as illustrated in Example 5-5.

#!/usr/bin/env bash
# cookbook filename: use_up_option
#
# use and consume an option
#
# parse the optional argument
VERBOSE=0
if [[ $1 = -v ]]
then
    VERBOSE=1
    shift
fi
#
# the real work is here
#
for FN in "$@"
do
    if (( VERBOSE == 1 ))
    then
        echo changing $FN
    fi
    chmod 0750 "$FN"
done

Discussion

We add a flag variable, $VERBOSE, to tell us whether or not to echo the filename as we work. But once the shell script has seen the -v and set the flag, we don’t want the -v in the argument list any more. The shift statement tells bash to shift its arguments down one position, getting rid of the first argument ($1) as $2 becomes $1, $3 becomes $2, and so on.

That way, when the for loop runs, the list of parameters (in $@) no longer contains the -v but starts with the next parameter.

This approach of parsing arguments is all right for handling a single option, but if you want more than one option, you need a bit more logic. By convention, options to a shell script should not be dependent on position; e.g., myscript -a -p should be the same as myscript -p -a. Moreover, a robust script should be able to handle repeated options and either ignore them or report an error. For more robust parsing, see the recipe on bash’s getopts builtin (Recipe 13.1).

Problem

You have a shell script that takes arguments supplied on the command line. You’d like to provide default values so that the most common values can be used without the user needing to type them every time.

Solution

Use the ${:-} syntax when referring to the parameter, and use it to supply a default value:

FILEDIR=${1:-/tmp}

Discussion

There are a series of special operators available when referencing a shell variable. The :- operator says that if the specified parameter (here, $1) is not set or is null, whatever follows (/tmp in our example) should be used as the value. Otherwise, it will use the value that is already set. It can be used on any shell variable, not just the positional parameters ($1, $2, $3, etc.), but they are probably the most common use.

Of course, you could do this the long way by constructing an if statement and checking to see if the variable is null or unset (we leave that as an exercise to the reader), but this sort of thing is so common in shell scripts that this syntax has been welcomed as a convenient shorthand.

See Also

• The bash manpage on parameter substitution

• Learning the bash Shell, 3rd Edition, by Cameron Newham (O’Reilly), pages 91–92

• Classic Shell Scripting by Nelson H. F. Beebe and Arnold Robbins (O’Reilly), pages 113–114

• Recipe 5.14, “Setting Default Values”

• Table 5-1

Problem

Your script relies on certain environment variables, either widely used ones (e.g., $USER) or ones specific to your own business. If you want to build a robust shell script, you should make sure that these variables each have a reasonable value. So how do you guarantee a reasonable default value?

Solution

Use the assignment operator in the shell variable reference the first time you refer to it to assign a value to the variable if it doesn’t already have one, as in:

cd ${HOME:=/tmp}

Discussion

The reference to $HOME in the example will return the current value of $HOME unless it is empty or not set at all. In those cases (empty or not set), it will return the value /tmp, which will also be assigned to $HOME so that further references to $HOME will have this new value.

We can see this in action here:

$ echo ${HOME:=/tmp}
/home/uid002
$ unset HOME # generally not wise to do
$ echo ${HOME:=/tmp}
/tmp
$ echo $HOME
/tmp
$ cd ; pwd
/tmp
$

Once we unset the variable, it no longer had any value. When we then used the := operator as part of our reference to it, the new value (/tmp) was substituted. The subsequent references to $HOME returned its new value.

One important exception to keep in mind about the assignment operator: this mechanism will not work with positional parameter arguments (e.g., $1 or $*). For those cases, use :- in expressions like ${1:-default}, which will return the value without trying to do the assignment.

As an aside, it might help you to remember some of these crazy symbols if you think of the visual difference between ${VAR:=value} and ${VAR:-value}. The := will do an assignment as well as returning the value to the right of the operator. The :- will do half of that—it returns the value but doesn’t do the assignment—so its symbol is only half of an equals sign (i.e., one horizontal bar, not two). If this doesn’t help, forget that we mentioned it.

See Also

• Recipe 5.13, “Getting Default Values”

• Table 5-1

Problem

You need to set a default value, but you want to allow an empty string as a valid value. You only want to substitute the default in the case where the value is unset.

The ${:=} operator has two cases where the new value will be used: first, when the value of the shell variable has previously not been set (or has been explicitly unset); and second, where the value has been set but is empty, as in HOME="" or HOME=$OTHER (where $OTHER has no value).

Solution

The shell can distinguish between these two cases, and omitting the colon (:) indicates that you want to make the substitution only if the value is unset. If you write only ${HOME=/tmp} without the colon, the assignment will take place only in the case where the variable is not set (never set or explicitly unset).

Discussion

Let’s play with the $HOME variable again, but this time without the colon in the operator:

$ echo ${HOME=/tmp}  # no substitution needed
/home/uid002
$ HOME=""       # generally not wise
$ echo ${HOME=/tmp}  # will NOT substitute

$ unset HOME    # generally not wise
$ echo ${HOME=/tmp}     # will substitute
/tmp
$ echo $HOME
/tmp
$

In the case where we simply made the $HOME variable an empty string, the = operator didn’t do the substitution since $HOME did have a value, albeit null. But when we unset the variable, the substitution occurred. If you want to allow for empty strings, use just the = with no colon. Most times, though, the := is used because you can do little with an empty value, deliberate or not.

See Also

• Recipe 5.13, “Getting Default Values”

• Recipe 5.14, “Setting Default Values”

• Table 5-1

Problem

You need something more than just a constant string as the default value for a variable.

Solution

You can use quite a bit more on the righthand side of these shell variable references. For example:

cd ${BASE:="$(pwd)"}

Discussion

As the example shows, the value that will be substituted doesn’t have to be just a string constant. Rather, it can be the result of a more complex shell expression, including running commands in a subshell (as in the example). In our example, if $BASE is not set, the shell will run the pwd builtin command (to get the current directory) and use the string that it returns as the value.

So what can you do on the righthand side of this (and the other similar) operators? The bash manpage says that what we put to the right of the operator “is subject to tilde expansion, parameter expansion, command substitution, and arithmetic expansion.”

Here is what that means:

• Parameter expansion means that we could use other shell variables in this expression, as in ${BASE:=${HOME}}.

• Tilde expansion means that we can use an expression like ~bob, and it will expand that to refer to the home directory of the user bob. Use ${BASE:=~uid17} to set the default value to the home directory for user uid17, but don’t put quotes around this string, as that will defeat the tilde expansion.

• Command substitution is what we used in the example; it will run the commands and take their output as the value for the variable. Commands are enclosed in the single parentheses syntax, $(cmds).

• Arithmetic expansion means that we can do integer arithmetic, using the $(( )) syntax in this expression. Here’s an example:

  echo ${BASE:=/home/uid$((ID+1))}

See Also

• Table 5-1

• Recipe 2.15, “Connecting Two Programs by Using Output as Input”

• Recipe 5.13, “Getting Default Values”

• Recipe 6.1, “Doing Arithmetic in Your Shell Script”

Problem

Those shorthands for giving a default value are cool, but sometimes you need to force the users to give you a value; otherwise, you don’t want to proceed. Perhaps if they left off a parameter, they don’t really understand how to invoke your script. You want to leave nothing to guesswork. Is there anything shorter than lots of if statements to check each of your several parameters?

Solution

Use the ${:?} syntax when referring to the parameters, as in Example 5-6. bash will print an error message and then exit if a parameter is unset or null.

#!/usr/bin/env bash
# cookbook filename: check_unset_parms
#
USAGE="usage: myscript scratchdir sourcefile conversion"
FILEDIR=${1:?"Error. You must supply a scratch directory."}
FILESRC=${2:?"Error. You must supply a source file."}
CVTTYPE=${3:?"Error. ${USAGE}"}

Here’s what happens when we run that script with insufficient arguments:

$ ./myscript /tmp /dev/null
./myscript: line 7: 3: Error. usage: myscript scratchdir sourcefile conversion
$

Discussion

The check is made to see if each parameter is set (or null); if not, bash will print an error message and exit.

The third variable uses another shell variable in its message. You can even run another command inside it:

CVTTYPE=${3:?"Error. $USAGE. $(rm $SCRATCHFILE)"}

If parameter three is not set, then the error message will contain the phrase “Error.” along with the value of the variable named $USAGE and then any output from the command that removes the file named by the variable $SCRATCHFILE. OK, so we’re getting carried away. You can make your shell scripts awfully compact, and we do mean awfully. It is better to waste some whitespace and a few bytes to make the logic ever so much more readable, as in:

if [ -z "$3" ]
then
    echo "Error. $USAGE"
    rm $SCRATCHFILE
fi

One other consideration: the error message produced by the ${:?} feature comes out with the shell script filename and line number. For example, the script fragment in Example 5-6 produces:

$ ./check_unset_parms
./check_unset_parms: line 5: 1: Error. You must supply a scratch directory.

$ ./check_unset_parms somedir
/tmp/check_unset_parms: line 6: 2: Error. You must supply a source file.

$ ./check_unset_parms somedir somefile
./check_unset_parms: line 7: 3: Error. usage: myscript scratchdir sourcefile \
conversion

Because you have no control over this part of the message, and since it looks like an error in the shell script itself, combined with the issue of readability, this technique is not so popular in commercial-grade shell scripts. (It is handy for debugging, though.)

If you’d rather have this behavior for all variables without having to change each one of them, use the set -u command to “treat unset variables as an error when substituting”:

$ echo "$foo"                       
$ set -u                            
$ echo "$foo"
bash: foo: unbound variable         
$ echo $?    # exit code
1
$ set +u                            
$ echo "$foo"                       
$ echo $?    # exit code
0
$

At first we have the normal behavior.

Then we turn on nounset (-u).

Now we see an error message and a failure exit code.

We turn nounset (-u) off again...

And return to the usual behavior.

See Also

• Recipe 5.13, “Getting Default Values”

• Recipe 5.14, “Setting Default Values”

• Recipe 5.16, “Using More than Just a Constant String for Default”

Problem

You want to rename a number of files. The filenames are almost right, but they have the wrong suffix.

Solution

Use a bash parameter expansion feature that will remove text that matches a pattern, as illustrated in Example 5-7.

#!/usr/bin/env bash
# cookbook filename: suffixer
#
# rename files that end in .bad to be .bash

for FN in *.bad
do
    mv "${FN}" "${FN%bad}bash"
done

Discussion

The for loop will iterate over a list of filenames in the current directory that all end in .bad. The variable $FN will take the value of each name, one at a time. Inside the loop, the mv command will rename the file (move it from the old name to the new name). We need to put quotes around each filename in case the filename contains embedded spaces.

The crux of this operation is the reference to $FN that includes an automatic deletion of the trailing bad characters. The ${ } delimits the reference so that the bash adjacent to it is just appended right onto the end of the string.

Here it is broken down into a few more steps:

NOBAD="${FN%bad}"
NEWNAME="${NOBAD}bash"
mv "${FN}" "${NEWNAME}"

This way you can see the individual steps of stripping off the unwanted suffix, creating the new name, and then renaming the files. Putting it all on one line isn’t so bad though, once you get used to the special operators.

Since we are not just removing a substring from the variable but are replacing the bad with bash, we might have used the substitution operator for variable references, the slash (/). Similar to editor commands (e.g., those found in vi and sed) that use the slash to delimit substitutions, we could have written:

# Not anchored, don't do this
mv "${FN}" "${FN/bad/bash}"

(Unlike with the editor commands, you don’t use a final slash—the righthand brace serves that function.)

However, one reason that we didn’t do it this way is because the substitution isn’t anchored, and can be made anywhere in the variable. If, for example, we had a file named subaddon.bad the substitution would leave us with subashdon.bad, which is not what we want. If we used a double slash in place of the first slash, it would substitute every occurrence within the variable. That would result in subashdon.bash, which isn’t what we want either. This is better:

# Add the "." to "anchor" the pattern; this is better, but not foolproof
mv "${FN}" "${FN/.bad/.bash}"

The ${FN%bad}bash we used in our solution is already anchored—it will only remove the text from the end of the string, which in this case is exactly what we want.

There are several operators that do various sorts of manipulation on the string values of variables when referenced. Table 5-1 summarizes them.

Try them all. They are very handy.

See Also

• man rename

• Recipe 12.5, “Comparing Two Documents”

• Recipe 13.10, “Taking It One Character at a Time” for more with substrings

• Recipe 13.15, “Trimming Whitespace”

• Recipe 5.19, “Getting the Absolute Value of a Number”

Problem

You have a numeric value in a variable, but it may be negative, zero, or positive. You would like to get its magnitude—that is, its absolute value—but bash doesn’t seem to have an absolute value function.

Solution

Use string manipulation:

${MYVAR#-}

Discussion

This is simple string manipulation. The # searches from the front of the string, looking for, in this case, the minus sign (-). If found, it will remove it. If no minus is found, it simply results in the original value. Either way, that leaves the value without a leading minus, which gives us its magnitude; i.e., its absolute value.

You could use if/then/else logic as a mathematically oriented approach:

# why bother?
if (( MYVAR < 0 ))
then
    let MYVAR=MYVAR*-1
fi

but as the comment says, why bother? The string manipulation technique is short and sweet. You may want to comment it for readability, though:

MYVAR=${MYVAR#-}        # ABS(MYVAR)

See Also

• Recipe 5.18, “Changing Pieces of a String”

• Table 5-1

Problem

The basename command does what you want, but can you get the same result without calling an external executable? Is bash string manipulation able to do that?

Solution

Yes, bash can strip the directory path from a shell variable string and leave just the last part of the path (the filename). Where you may want to write:

FILE=$(basename $FULLPATHTOFILE)

instead you need only write:

FILE=${FULLPATHTOFILE##*/}

Discussion

The big difference between the first and second examples is the braces. The first example, using parentheses, will launch a subshell to run the executable basename with the argument that is the value of $FULLPATHTOFILE (the old way of doing this was ``). The second example uses curly braces, which is just part of the syntax for evaluating a shell variable—no subshell, no executable file. It looks for, and removes from the front of the string (because of the #), the longest match (because of the double ##) of the pattern described by the asterisk and the slash (*/). The asterisk matches any number of characters and the slash is just a literal slash. In the string /usr/local/bin/mycmd, that pattern will match (and thus remove) the /usr/local/bin/ part of the string, leaving mycmd as the value to be assigned into the variable $FILE.

FILE=$(basename $MYIMAGEFILE .jpg)

FILE=${MYIMAGEFILE%/}  # remove a trailing slash
FILE=${FILE##*/}       # remove all chars up to last /
FILE=${FILE%.jpg}      # remove .jpg suffix if present

See Also

• man basename

• Recipe 5.18, “Changing Pieces of a String”

• Recipe 5.21, “Using bash for dirname”

• Table 5-1

Problem

The dirname command does what you want, but like basename, it causes a separate executable to be launched and run in a subshell. Can you use string manipulations instead?

Solution

Yes. Use a string manipulation operator to remove the filename—the last part of a path in a string—leaving as much of the directory path to that filename as was in the string:

DIR=${MYPATHTOFILE%/*}

Discussion

If the variable holds /usr/local/bin/mycmd, we want the result of this manipulation to give us just /usr/local/bin and drop the last part (the filename). Since each piece of the path is separated by a slash, we just remove from the righthand side (because of the %) the shortest string (because there is only one %, not two) that matches the pattern “a slash followed by any number of characters” (/*).

See Also

• man dirname for other options and subtle differences

• Recipe 5.18, “Changing Pieces of a String”

• Recipe 5.20, “Using bash for basename”

• Table 5-1

Problem

You want to make a list of values separated by commas, but you don’t want a leading or trailing comma.

Solution

If you write LIST="${LIST},${NEWVAL}" inside a loop to build up the list, then the first time (when LIST is null) you’ll end up with a leading comma. You could special-case the initialization of LIST so that it gets the first element before entering the loop, but if that’s not practical, or to avoid duplicate code (for getting a new value), you can instead use the ${:+} syntax in bash:

LIST="${LIST}${LIST:+,}${NEWVAL}"

If ${LIST} is null or unset, then both expressions of $LIST are replaced with nothing. That means that the first time through the loop LIST will be assigned NEWVAL’s value and nothing more. When LIST is not null, the second expression (${LIST:+,}) is replaced with a comma, separating the previous value from the new value.

Here is an example code segment for reading and constructing a CSV list:

#
# read names one at a time
# and build a comma-separated list
#
while read NEWVAL
do
    LIST="${LIST}${LIST:+,}${NEWVAL}"
done
echo $LIST

See Also

• Recipe 5.18, “Changing Pieces of a String”

• Table 5-1

Problem

You’ve seen plenty of scripts so far with variables, but can bash deal with an array of variables?

Solution

Yes. bash has an array syntax for single-dimension arrays.

Discussion

Arrays are easy to initialize if you know the values as you write the script. The format is simple:

MYRA=(first second third home)

Each element of the array is a separate word in the list enclosed in parentheses. Then you can refer to each this way:

echo runners on ${MYRA[0]} and ${MYRA[2]}

This output is the result:

runners on first and third

If you write only $MYRA, you will get only the first element, just as if you had written ${MYRA[0]}.

See Also

• Learning the bash Shell, 3rd Edition, by Cameron Newham (O’Reilly), pages 157–161, for more information about arrays

• Recipe 7.15, “Counting String Values with bash” for another type of array in bash, associative arrays

• Recipe 13.4, “Parsing Output into an Array”

Problem

Your digital camera left you with a pile of files all named in uppercase, like IMG0001.JPG. You want the names in lowercase, but don’t want to have to retype each name.

Solution

As of bash 4.0 there are a few operators to do case conversion when referencing a variable name. If $FN is the variable in which you put a filename (i.e., string) that you want converted to lowercase, then ${FN,,} will return that string in all lowercase. Similarly, ${FN^^} will return the string in all uppercase. There is even the ${FN~~} operator to swap case, changing all lower- to upper- and all upper- to lowercase characters (but why would you want to do that?).

Here is a for loop that will rename all the .JPG files to lowercase names:

for FN in *.JPG
do
    mv "$FN" "${FN,,}"
done

or as a one-liner:

for FN in *.JPG; do mv "$FN" "${FN,,}" ; done

There is another approach, also available in version 4 of bash or newer: you can declare your variable to be a type that is always lowercase. Any text assigned to it will be converted to lowercase. Using that approach our for loop to rename files just does a simple assignment rather than requiring a string substitution operator:

declare -l lcfn    # contents will be converted to lowercase
for FN in *.JPG
do
    lcfn="$FN"
    mv "$FN" "$lcfn"
done

There are similar declarations for variables that change the case of all letters or only the first letter. Here’s a simple demonstration program to show how they work:

declare -u UP     # all UPPERCASE
declare -l dn     # all lowercase
declare -c Ca     # only the first Uppercase

while read TXT
do
    UP="${TXT}"
    dn="${TXT}"
    Ca="${TXT}"

    echo $TXT  $UP  $dn  $Ca
done

In the case of the variable declared with -c, only the first letter is capitalized even if there are multiple words in the string. Try running it and see how it works.

See Also

• man rename

• Recipe 5.25, “Converting to Camel Case”

Problem

You want each word to begin with a capital letter, not just the first letter of the string.

Solution

Use a combination of an array and case conversion substitution:

while read TXT
do
    RA=($TXT)        # must be ($  not $(
    echo ${RA[@]^}
done

Discussion

The parentheses around $TXT cause it to be treated as array initialization. Whitespace separating the words in the text delineates the array elements. The [@] notation references all the elements of the array at once (individually), and the ^ operator converts the first character (of each element) to uppercase.

See Also

• Recipe 5.24, “Converting Between Upper- and Lowercase”

Problem

You need to do some simple arithmetic in your shell script.

Solution

Use $(( )) or let for integer arithmetic expressions. For example:

COUNT=$((COUNT + 5 + MAX * 2))
let COUNT+='5+MAX*2'

Discussion

As long as you keep to integer arithmetic, you can use all the standard (i.e., C-like) operators inside of $(( )) for arithmetic expressions. There is one additional operator, too: you can use ** for raising to a power, as in MAX=$((2**8)), which yields 256.

Spaces are not needed, nor are they prohibited around operators and arguments (though ** must be together) within a $(( )) expression. But you must not have spaces around the equals sign, as with any bash variable assignment. Also, be sure to quote let expressions since the let statement is a bash builtin and its arguments will undergo word expansion.

COUNT  =  $((COUNT+5)) # not what you think!

Another oddity to these expressions is that the $ that we normally put in front of a shell variable to say we want its value (as in $COUNT or $MAX) is not needed inside the double parentheses. For example, we can write:

$((COUNT + 5 + MAX * 2))

without including the dollar sign on the shell variables—in effect, the outer $ applies to the entire expression. We do need the dollar sign, though, if we are using a positional parameter (e.g., $2), to distinguish it from a numeric constant (e.g., 2). Here’s an example:

COUNT=$((COUNT + $2 + OFFSET))

There is a similar mechanism for integer arithmetic with shell variables using the bash builtin let statement. It uses the same arithmetic operators as the $(( )) construct:

let COUNT=COUNT+5

When using let there are some fancy assignment operators we can use, such as these (which will accomplish the same thing as the previous line):

let COUNT+=5

(These should look familiar to programmers of C/C++ and Java.) This example adds five to the previous value of COUNT without our having to repeat the variable name.

Table 6-1 shows a list of those special assignment operators.

These assignment operators are also available with $(( )) provided they occur inside the double parentheses. The outermost assignment is still just plain old shell variable assignment.

The assignments can also be cascaded, through the use of the comma operator:

echo $(( X+=5 , Y*=3 ))

which will do both assignments and then echo the result of the second expression (since the comma operator returns the value of its second operand). If you don’t want to echo the result, the more common usage would be with the let statement:

let   X+=5 Y*=3

The comma operator is not needed here, as each word of a let statement is its own arithmetic expression.

One other important difference between the let statement and the $(( )) syntax is how they handle whitespace (i.e., the space character). The let statement either requires quotes or that there be no spaces around not only the assignment operator (the equals sign), but any of the other operators as well; it must all be packed together into a single word. These both work:

let i=2+2
let "i = 2 + 2"

The $(( )) syntax, however, can be much more generous, allowing all sorts of whitespace within the parentheses. For that reason, it is less prone to errors and makes the code much more readable, and is, therefore our preferred way of doing bash integer arithmetic. However, an exception can be made for the occasional += assignment or ++ operator, or when we get nostalgic for the early days of BASIC programming (which had a LET statement).

See Also

• help let

• The bash manpage

Problem

You want to check if you have the right number of arguments and take actions accordingly. You need a branching construct.

Solution

The if statement in bash is similar in appearance to that in other programming languages:

if [ $# -lt 3 ]
then
    printf "%b" "Error. Not enough arguments.\n"
    printf "%b" "usage: myscript file1 op file2\n"
    exit 1
fi

or alternatively:

if (( $# < 3 ))
then
    printf "%b" "Error. Not enough arguments.\n"
    printf "%b" "usage: myscript file1 op file2\n"
    exit 1
fi

Here’s a full-blown if with an elif (bash-talk for else-if) and an else clause:

if (( $# < 3 ))
then
    printf "%b" "Error. Not enough arguments.\n"
    printf "%b" "usage: myscript file1 op file2\n"
    exit 1
elif (( $# > 3 ))
then
    printf "%b" "Error. Too many arguments.\n"
    printf "%b" "usage: myscript file1 op file2\n"
    exit 2
else
    printf "%b" "Argument count correct. Proceeding...\n"
fi

You can even do things like this:

[ $result = 1 ] \
  && { echo "Result is 1; excellent." ; exit 0;   } \
  || { echo "Uh-oh, ummm, RUN AWAY! " ; exit 120; }