Commands and Arguments

The basic operation of bash is to execute a command, that is, to run another program. When several words appear on the command line bash assumes that the first word is the name of the program to run and the remaining words are the arguments to the command. For example:

mkdir -p /tmp/scratch/garble

will have bash run the command called mkdir and it will pass it two arguments -p and /tmp/scratch/garble. By convention programs generally put their options first, and have them begin with a leading "-“, as is the case here with the -p option. This particular command will create a directory called /tmp/scratch/garble. The -p option will mean that no errors will be reported and any intervening directories will be created (or attempted) as needed (e.g., if only /tmp exists, it will create /tmp/scratch before attempting to create /tmp/scratch/garble).

Standard Input/Output/Error

A running program is called a process and every process in the Unix/Linux/Posix (and thus Windows) environment has three distinct input/output file descriptors. These three are called “standard input” (or stdin, for short), “standard output” (stdout), and “standard error” (stderr).

As you might guess by its name, stdin is the default source for input to a program, by default the characters coming from the keyboard. When your script reads from stdin it is reading characters typed on the keyboard or (as we shall see shortly) it can be changed to read from a file. Stdout is the default place for sending output from a program. By default the output appears in the window which is running your shell or shell script. Standard error can also be sent output from a program, but it is (or should be) where error messages are written. It’s up to the person writing the program to direct any output to either stdout or stderr. So be conscientious when writing your scripts to send any error messages not to stdout but to stderr (as shown below).

Redirection and Piping

One of the great innovations of the shell was that it gave you a mechanism whereby you could take a running program and change where it got its input and/or change where it sent its output without modifying the program itself. If you have a program called handywork and it reads its input from stdin and writes its results to stdout, then you can change its behavior as simply as this:

handywork < data.in  > results.out

which will run handywork but will have the input come not from the keyboard but instead from the data file called data.in (assuming such a file exists and has input in the format we want). Similarly the output is being sent not to the screen but into a file called results.out (which will be created if it doesn’t exist and overwritten if it does). This technique is called “redirection” because we are re-directing input to come from a different place and re-directing output to go somewhere other than the screen.

What about stderr? The syntax is similar. We have to distinguish between stout and stderr when redirecting data coming out of the program and we make this distinction through the use of the file descriptor numbers. Stdin is file descriptor 0, stdout is file descriptor 1 and stderr is file descriptor 2 so we can redirect error messages this way:

handywork 2> err.msgs

which will redirect only stderr and send any such error message output to a file we called err.msgs (for obvious reasons).

Of course we can do all three on the same line:

handywork < data.in  > results.out  2> err.msgs

Sometimes we want the error messages combined with the normal output (as it does by default when both are written to the screen). We can do this with the following syntax:

handywork < data.in  > results.out 2>&1

which says to send stderr (2) to the same location as file descriptor 1 (”&1“). Note that without the ampersand, the error messages would just be sent to a file named “1”. This combining of stdout and stderr is so common that there is a useful shorthand notation:

handywork < data.in  &> results.out

If you want to discard standard output you can redirect it to a special file called /dev/null as follows:

handywork < data.in > /dev/null

To view output on the command line and simultaneously redirect that same output to a file, use the tee command. The following will display the output of handywork to the screen and also save it to results.out:

handywork < data.in | tee results.out

A file will be created or truncated (i.e., contents discarded) when output is redirected. If you want to preserve the file’s existing content you can, instead, append to the file using a double greater-than sign, like this:

handywork < data.in  >> results.out

This will execute handywork and then any output from stdout will be appended to the file results.out rather than overwriting its existing content.

Similarly this command line:

handywork < data.in  &>> results.out

will execute handywork and then append both stdout and stderr to the file results.out rather than overwriting its existing content.

Running Commands in the Background

Throughout this book we will be going beyond one-line commands and will be building complex scripts. Some of these scripts can take a significant amount of time to execute, so much so that you may not want to spend time waiting for them to complete. Instead, you can run any command or script in the background using the & operator. The script will continue to run, but you can issue other commands and/or run other scripts. For example, to run ping in the background and redirect standard output to a file:

ping 192.168.10.56 > ping.log &

You will likely want to redirect standard output and/or standard error to a file when sending tasks to the background, or the task will continue to print to the screen and interrupt other activities you are performing.

You can use the jobs command to list any tasks currently running in the background.

$ jobs
[1]+  Running                 ping 192.168.10.56 > ping.log &

Use the fg command and the corresponding job number to bring the task back into the foreground.

$ fg 1
ping 192.168.10.56 > ping.log

If your task is currently executing in the foreground you can use CTRL-Z to suspend the process and then bg to continue the process in the background. From there you can use jobs and fg as described above.

From Command Line to Script

A shell script is just a file that contains the same commands that you could type on the command line. Put one or more commands into a file and you have a shell script. If you called your file myscript you can run that script by typing: bash myscript or you can give it “execute permission” (e.g., chmod 755 myscript) and then you can invoke it directly: ./myscript to run the script. We often include the following line as the first line of the script, which tells the operating system which scripting language we are using:

#!/bin/bash -

Of course this assumes that bash is located in the /bin directory. If your script needs to be more portable, you could use this approach instead:

#!/usr/bin/env bash

It uses the env command to look up the location of bash and is considered the standard way to address the portability problem. It makes the assumption, however, that the env command is to be found in /usr/bin.

Summary

In this chapter you saw how to run single commands and redirect input and output. In the next chapter we will discuss the real power of scripting, which comes from being able to run commands repeatedly, make decisions in the script, and loop over a variety of inputs.

Exercises

1. Write a command that executes ifconfig and redirects standard output to a file named ipaddress.txt.

2. Write a command that executes ifconfig and redirects standard output and appends it to a file named ipaddress.txt.

3. Write a command that copies all of the files in the directory /etc/a to the directory /etc/b and redirects standard error to the file copyerror.log.

4. Write a command that performs a directory listing (ls) on the root file directory and pipes the output into the more command.

5. Write a command that executes mytask.sh and sends it to the background.

6. Given the job list below, write the command that brings the Amazon ping task to the foreground.

   [1]   Running                 ping www.google.com > /dev/null &
   [2]-  Running                 ping www.amazon.com > /dev/null &
   [3]+  Running                 ping www.oreilly.com > /dev/null &