Problem

You need to run a bash script on several machines, but bash is not always in the same place (see Recipe 1.14).

Solution

Use the /usr/bin/env command in the shebang line, as in #!/usr/bin/env bash. If your system doesn’t have env in /usr/bin, ask your system administrator to install it, move it, or create a symbolic link because this is the required location.

You could also create symbolic links for bash itself, but using env is the canonical and correct solution.

Discussion

env’s purpose is to “run a program in a modified environment,” but since it will search the path for the command it is given to run, it works very well for this use.

You may be tempted to use !/bin/sh instead. Don’t. If you are using bash-specific features in your script, they will not work on machines that do not use bash in Bourne shell mode for /bin/sh (e.g., BSD, Solaris, Ubuntu 6.10+). And even if you aren’t using bash-specific features now, you may forget about that in the future. If you are committed to using only POSIX features, by all means use !/bin/sh (and don’t develop on Linux; see Recipe 15.3), but otherwise be specific.

You may sometimes see a space between #! and /bin/whatever. Historically there were some systems that required the space, though in practice we haven’t seen one in a long time. It’s very unlikely any system running bash will require the space, and leaving it out seems to be the most common usage now. But for the utmost historical compatibility, use the space.

We have chosen to use #!/usr/bin/env bash in the longer scripts and functions we’ve made available to download (see the end of the Preface for details), because that will run unchanged on most systems. However, since env uses the $PATH to find bash, this is arguably a security issue (see Recipe 14.2), albeit a minor one in our opinion.

/usr/bin/env: bash -: No such file or directory

See Also

• For information on the shebang line (/usr/bin/env):

  • http://srfi.schemers.org/srfi-22/mail-archive/msg00069.html

  • http://www.in-ulm.de/~mascheck/various/shebang/

  • http://homepages.cwi.nl/~aeb/std/hashexclam-1.html

  • Section 3.16 of the Unix FAQs

• Recipe 1.14, “Getting bash for xBSD”

• Recipe 15.2, “Setting a POSIX $PATH”

• Recipe 15.3, “Developing Portable Shell Scripts”

• Recipe 15.6, “Using echo Portably”

Problem

You are writing a shell script that will need to run on multiple versions of multiple Unix or POSIX operating systems.

Solution

First, try using the command builtin with its -p option to find the POSIX version of program. For example, in /usr/xpg4 or /usr/xpg6 on Solaris:

command -p program args

Then, if possible, find the oldest or least capable Unix machine you have access to and develop the script on that platform. If you aren’t sure what the least capable platform is, use a BSD variant or Solaris (and the older a version you can find, the better).

Discussion

command -p uses a default path that is guaranteed to find all of the POSIX-standard utilities. If you’re sure your script will only ever run on Linux (famous last words), then don’t worry about it; otherwise, avoid developing cross-platform scripts on Linux or Windows (e.g., via Cygwin).

The problems with writing cross-platform shell scripts on Linux are:

1. /bin/sh is not the Bourne shell; it’s really /bin/bash in POSIX mode, except when it’s /bin/dash (for example, Ubuntu 6.10+). Both are very good, but not perfect, and none of the three work exactly the same, which can be very confusing. In particular, the behavior of echo can change.

2. Linux uses the GNU tools instead of the original Unix tools.

Don’t get us wrong, we love Linux and use it every day. But it isn’t really Unix: it does some things differently, and it has the GNU tools. The GNU tools are great, and that’s the problem. They have a lot of switches and features that aren’t present on other platforms, and your script will break in odd ways no matter how careful you are about that. Conversely, Linux is so compatible with everything that scripts written for any other Unix-like systems will almost always run on it. They may not be perfect (e.g., echo’s default behavior is to display \n instead of printing a newline), but they’re often good enough.

There is an ironic Catch-22 here—the more shell features you use, the less you have to depend on external programs that may or may not be there or work as expected. While bash is far more capable than sh, it’s also one of the tools that may or may not be there. Some form of sh will be on virtually any Unix or Unix-like system, but it isn’t always quite what you think it is.

Another Catch-22 is that the GNU long options are much more readable in shell code, but are often not present on other systems. So instead of being able to say sort --field-separator=, unsorted_file > sorted_file, you have to use sort -t, unsorted_file>sorted_file for portability.

But take heart: developing on a non-Linux system is easier than it’s ever been. If you already have and use such systems, then this is obviously a nonissue. But if you don’t have such systems in-house, it’s now trivial to get them for free. Solaris and the BSDs all run in virtual environments (see Recipe 15.4).

If you have a Mac running macOS (previously OS X), then you already have bash and BSD so you’re all set. You might want to make sure you have a recent version, though; see Recipe 1.15.

You can also easily test scripts using a virtualization environment (see Recipe 15.4). The flaw in this solution is the systems such as AIX and HP-UX that don’t run on an x86 architecture, and thus don’t run under x86 virtualization. Again, if you have these systems, use them. If not, see Recipe 1.18.

See Also

• help command

• http://en.wikipedia.org/wiki/Debian_Almquist_shell

• http://en.wikipedia.org/wiki/Bash

• http://partnerweb.vmware.com/GOSIG/Solaris_11.html

• http://www.polarhome.com/

• comp.sys.hp.hpux FAQ

• History of Unix

• Unix History repo

• Recipe 1.18, “Getting bash Without Getting bash”

• Recipe 15.4, “Testing Scripts Using Virtual Machines”

• Recipe 15.6, “Using echo Portably”

• “echo Options and Escape Sequences” in Appendix A

Problem

You need to develop cross-platform scripts but do not have the appropriate systems or hardware.

Solution

If the target platforms run on the x86 architecture, use one of the many free and commercial virtualization solutions and build your own test virtual machine (VM) or search for prebuilt virtual machines on the OS vendor or distributor’s site, or the internet. Or use a free (for a trial period) or low-cost VM from a cloud vendor.

The flaw in this solution is the systems such as AIX and HP-UX that don’t run on an x86 architecture, and thus don’t run under x86 virtualization. Again, if you have these systems, use them. If not, see Recipe 1.18.

Discussion

Testing shell scripts is usually not very resource-intensive, so even moderate hard-ware capable of running VirtualBox or a similar virtualization package should be fine. We mention VirtualBox specifically because it’s without cost, runs on Linux, macOS, and Windows, is used in countless examples around the web and tools such as Vagrant, and is flexible and easy to use; but there are certainly other alternatives available.

Minimal virtual machines with 128 MB of RAM, or sometimes even less, should be more than enough for a shell environment for testing. Set up an NFS share to store your test scripts and data, and then simply SSH to the test system. Debian is a good place to start if you are building your own; just remember to uncheck everything you can during the install.

There are a great many pre-built VMs available on the internet, but quality and security will vary. If you are testing at work, be sure to check your corporate policies; many companies prohibit bringing “random internet downloads” into the corporate network. On the other hand, your company may build or provide its own VM images for internal use. You will probably want only a very minimal VM for testing shell scripts, but the definition of “minimal” will also vary greatly among different sources. You’ll need to do a little research to find a fit for your needs. Some good places to start are:

• TurnKey Linux

• VMware

• OSBoxes

• KVM

• Parallels

Depending on your needs and corporate policy, you may also be able to get a free or low-cost VM in the cloud. See Recipe 1.18 for details about getting an almost free shell account from http://polarhome.com, which has a tiny, symbolic one-time fee, or another vendor.

Amazon has a “free tier” offering that may be useful, and it and many other vendors like Linode and Digital Ocean have very inexpensive pay-as-you-go options.

Don’t forget about just booting a LiveCD/LiveDVD either, as we mentioned in Recipe 1.18.

Finally, if all that is not enough, the initiator of the QEMU emulator, Fabrice Bellard, has written a PC emulator in JavaScript that lets you boot VM images with just a web browser!

No matter what option you choose, there will be a lot more information, documentation, and how-to guides available on the internet than we can fit in this recipe. Our main goal here is just to get you thinking about some possibilities.

See Also

• https://www.virtualbox.org/

• https://www.debian.org/distrib/netinst

• https://www.turnkeylinux.org/core (bash 4.3 or newer)

• http://www.vmware.com/ (commercial)

  • https://solutionexchange.vmware.com/store/category_groups/virtual-appliances

• https://www.osboxes.org/

• http://www.linux-kvm.org

  • http://xmodulo.com/use-kvm-command-line-debian-ubuntu.html

  • http://www.thegeekstuff.com/2014/10/linux-kvm-create-guest-vm/

• http://polarhome.com

• https://shells.red-pill.eu/

• https://aws.amazon.com/free/

• https://www.linode.com/pricing

• https://www.digitalocean.com/pricing/

• http://wiki.qemu.org

• http://copy.sh/v86/

• Recipe 1.14, “Getting bash for xBSD”

• Recipe 1.18, “Getting bash Without Getting bash”

Problem

You need to do a for loop but want it to work on older versions of bash.

Solution

This method is portable back to bash 2.04+:

$ for ((i=0; i<10; i++)); do echo $i; done
0
1
2
3
4
5
6
7
8
9

Discussion

There are nicer ways of writing this loop in newer versions of bash, but they are not backward compatible. As of bash 3.0+ you can use the syntax for {x..y}, as in:

$ for i in {1..10}; do echo $i; done
1
2
3
4
5
6
7
8
9
10

If your system has the seq command, you could also do this:

$ for i in $(seq 1 10); do echo $i; done
1
2
3
4
5
6
7
8
9
10

See Also

• help for

• man seq

• Recipe 6.12, “Looping with a Count”

• Recipe 6.13, “Looping with Floating-Point Values”

• Recipe 17.24, “Writing Sequences”

Problem

You are writing a script that will run on multiple versions of Unix and Linux and you need echo to behave consistently even if it is not running on bash.

Solution

Use printf "%b" whatever, or test for the system and set xpg_echo using shopt -s xpg_echo as needed.

If you omit the "%b" format string (for example, printf whatever), then printf will try to interpret any % characters in whatever, which is probably not what you want. The "%b" format is an addition to the standard printf format that will prevent that misinterpretation and also expand backslash escape sequences in whatever.

Setting xpg_echo is less consistent since it only works on bash. It can be effective if you are sure that you’ll only every run under bash, and not under sh or another similar shell that doesn’t use xpg_echo.

Using printf requires changes to how you write echo statements, but it’s defined by POSIX and should be consistent across any POSIX shell anywhere. Specifically, you have to write printf "%b" instead of just echo.

$ printf "%b" "Works"
Works
$ printf "$b" "Broken"
$

Discussion

In some shells, the builtin echo behaves differently than the external echo used on other systems. This is not always obvious when running on Linux since /bin/sh is actually bash (usually; it could also be dash on Ubuntu 6.10+), and there are similar circumstances on some BSDs. The difference is in how echo does or does not expand backslash-escape sequences. Shell builtin versions tend not to expand, while external versions (e.g., /bin/echo and /usr/bin/echo) tend to expand; but again, that can change from system to system.

Typical Linux (/bin/bash):

$ type -a echo
echo is a shell builtin
echo is /bin/echo

$ builtin echo "one\ttwo\nthree"
one\ttwo\nthree\n

$ /bin/echo "one\ttwo\nthree"
one\ttwo\nthree\n

$ echo -e "one\ttwo\nthree"
one → two
three

$ /bin/echo -e "one\ttwo\nthree"
one → two
three

$ shopt -s xpg_echo

$ builtin echo "one\ttwo\nthree"
one → two
three

$ shopt -u xpg_echo

$ builtin echo "one\ttwo\nthree"
one\ttwo\nthree\n

Typical BSD (/bin/csh, then /bin/sh):

$ which echo
echo: shell builtin command.

$ echo "one\ttwo\nthree"
one\ttwo\nthree\n

$ /bin/echo "one\ttwo\nthree"
one\ttwo\nthree\n

$ echo -e "one\ttwo\nthree"
-e one\ttwo\nthree\n

$ /bin/echo -e "one\ttwo\nthree"
-e one\ttwo\nthree\n

$ printf "%b" "one\ttwo\nthree"
one → two

$ /bin/sh

$ echo "one\ttwo\nthree"
one\ttwo\nthree\n

$ echo -e "one\ttwo\nthree"
one → two
three

$ printf "%b" "one\ttwo\nthree"
one → two
three

Solaris 10 (/bin/sh):

$ which echo
/usr/bin/echo

$ type echo
echo is a shell builtin

$ echo "one\ttwo\nthree"
one → two
three

$ echo -e "one\ttwo\nthree"
-e one → two
three

$ printf "%b" "one\ttwo\nthree"
one → two
three

See Also

• help printf

• man 1 printf

• http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html

• http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html

• Recipe 2.3, “Writing Output with More Formatting Control”

• Recipe 2.4, “Writing Output Without the Newline”

• Recipe 15.1, “Finding bash Portably for #!”

• Recipe 15.3, “Developing Portable Shell Scripts”

• Recipe 19.11, “Seeing Odd Behavior from printf”

• “printf” in Appendix A

Problem

You want to split output only if the input exceeds your limit, but the split command always creates at least one new file.

Solution

Example 15-1 illustrates a way to break up input into fixed sizes only if the input exceeds the size limit.

# cookbook filename: func_split

#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# Output fixed-size pieces of input ONLY if the limit is exceeded
# Called like: Split <file> <prefix> <limit option> <limit argument>
# e.g. Split $output ${output}_ --lines 100
# See split(1) and wc(1) for option details
function Split {
    local file=$1
    local prefix=$2
    local limit_type=$3
    local limit_size=$4
    local wc_option

    # Sanity checks
    if [ -z "$file" ]; then
        printf "%b" "Split: requires a file name!\n"
        return 1
    fi
    if [ -z "$prefix" ]; then
        printf "%b" "Split: requires an output file prefix!\n"
        return 1
    fi
    if [ -z "$limit_type" ]; then
        printf "%b" \
          "Split: requires a limit option (e.g. --lines), see 'man split'!\n"
        return 1
    fi
    if [ -z "$limit_size" ]; then
        printf "%b" "Split: requires a limit size (e.g. 100), see 'man split'!\n"
        return 1
    fi

    # Convert split options to wc options. Sigh.
    # Not all options supported by all wc/splits on all systems
    case $limit_type in
        -b|--bytes)      wc_option='-c';;
        -C|--line-bytes) wc_option='-L';;
        -l|--lines)      wc_option='-l';;
    esac

    # If whatever limit is exceeded
    if [ "$(wc $wc_option $file | awk '{print $1}')" -gt $limit_size ]; then
        # Actually do something
        split --verbose $limit_type $limit_size $file $prefix
    fi
} # end of function Split

Discussion

Depending on your system, some options (e.g., -C) may not be available in split or wc.

See Also

• Recipe 8.13, “Counting Lines, Words, or Characters in a File”

Problem

You need to see output in hex mode to verify that a certain whitespace or unprintable character is as expected.

Solution

Pipe the output though hexdump using the -C option for canonical output:

$ hexdump -C filename
00000000  4c 69 6e 65 20 31 0a 4c 69 6e 65 20 32 0a 0a 4c |Line 1.Line 2..L|
00000010  69 6e 65 20 34 0a 4c 69 6e 65 20 35 0a 0a       |ine 4.Line 5..|
0000001e
$

For example, nl uses spaces (ASCII 20), then the line number, then a tab (ASCII 09) in its output:

$ nl -ba filename | hexdump -C
00000000  20 20 20 20 20 31 09 4c  69 6e 65 20 31 0a 20 20 |     1.Line 1.  |
00000010  20 20 20 32 09 4c 69 6e  65 20 32 0a 20 20 20 20 |   2.Line 2.    |
00000020  20 33 09 0a 20 20 20 20  20 34 09 4c 69 6e 65 20 | 3..     4.Line |
00000030  34 0a 20 20 20 20 20 35  09 4c 69 6e 65 20 35 0a |4.     5.Line 5.|
00000040  20 20 20 20 20 36 09 0a                          |     6..|
00000048
$

Discussion

hexdump is a BSD utility that also comes with many Linux distributions. Other systems, notably Solaris, do not have it by default. You can use the octal dump command od, but its output is only one format at a time, and its addresses (lefthand column) are in octal, not hex:

$ nl -ba filename | od -x
0000000 2020 2020 3120 4c09 6e69 2065 0a31 2020
0000020 2020 3220 4c09 6e69 2065 0a32 2020 2020
0000040 3320 0a09 2020 2020 3420 4c09 6e69 2065
0000060 0a34 2020 2020 3520 4c09 6e69 2065 0a35
0000100 2020 2020 3620 0a09
0000110

$ nl -ba filename | od -tx1
0000000 20 20 20 20 20 31 09 4c 69 6e 65 20 31 0a 20 20
0000020 20 20 20 32 09 4c 69 6e 65 20 32 0a 20 20 20 20
0000040 20 33 09 0a 20 20 20 20 20 34 09 4c 69 6e 65 20
0000060 34 0a 20 20 20 20 20 35 09 4c 69 6e 65 20 35 0a
0000100 20 20 20 20 20 36 09 0a
0000110

$ nl -ba filename | od -c
0000000                       1  \t   L   i   n   e       1  \n
0000020               2  \t   L   i   n   e       2  \n
0000040       3  \t  \n                       4  \t   L   i   n   e
0000060   4  \n                       5  \t   L   i   n   e       5  \n
0000100                       6  \t  \n
0000110

There is also a simple Perl script that might work:

$ ./hexdump.pl filename

       /0 /1 /2 /3 /4 /5 /6 /7 /8 /9/ A /B /C /D /E /F  0123456789ABCDEF
0000 : 4C 69 6E 65 20 31 0A 4C 69 6E 65 20 32 0A 0A 4C  Line 1.Line 2..L
0010 : 69 6E 65 20 34 0A 4C 69 6E 65 20 35 0A 0A        ine 4.Line 5..

See Also

• man hexdump

• man od

• http://www.khngai.com/perl/bin/hexdump.txt

• http://gnuwin32.sourceforge.net/packages/hextools.htm

• “Table of ASCII Values” in Appendix A

Problem

You need to send or receive very simple network traffic but you do not have a tool such as Netcat installed.

Solution

If you have bash version 2.04+ compiled with --enable-net-redirections (default), you can use bash itself. The following example is also used in Recipe 15.10:

$ exec 3<> /dev/tcp/checkip.dyndns.org/80
$ echo -e "GET / HTTP/1.0\n" >&3
$ cat <&3
HTTP/1.1 200 OK
Content-Type: text/html
Server: DynDNS-CheckIP/1.0
Connection: close
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 105

<html><head><title>Current IP Check</title></head>
<body>Current IP Address: 72.NN.NN.225</body></html>

$ exec 3<> /dev/tcp/checkip.dyndns.org/80
$ echo -e "GET / HTTP/1.0\n" >&3
$ egrep --only-matching 'Current IP Address: [0-9.]+' <&3
Current IP Address: 72.NN.NN.225
$

Discussion

As noted in Recipe 15.12, it is possible to use exec to permanently redirect file handles within the current shell session, so the first command sets up input and output on file handle 3. The second line sends a trivial command to a path on the web server defined in the first command. Note that the user agent will appear as “-” on the web server side, which is what is causing the “flagged User Agent” warning. The third command simply displays the results.

Both TCP and UDP are supported. Here is a trivial way to send syslog messages to a remote server (although in production we recommend using the logger utility, which is much more user-friendly and robust):

echo "<133>${0##*/}[$$]: Test syslog message from bash" \
  > /dev/udp/loghost.example.com/514

Since UDP is connectionless, this is actually much easier to use than the previous TCP example. <133> is the syslog priority value for local0.notice, calculated according to RFC 3164. See section 4.1.1 of the RFC and the logger manpage for details. $0 is the name, so ${0##*/} is the “basename” and $$ is the process ID of the current program. The name will be -bash for a login shell.

See Also

• man logger

• RFC 3164

• Recipe 15.10, “Finding My IP Address”

• Recipe 15.12, “Redirecting Output for the Life of a Script”

• Recipe 15.14, “Logging to syslog from Your Script”

• The bash documentation

• https://bugs.launchpad.net/ubuntu/+source/bash/+bug/215034

Problem

You need to know the IP address of the machine you are running on.

Solution

There is no good way to do this that will work on all systems in all situations, so we will present several possible solutions.

First, you can parse output from ifconfig to look for IP addresses. The commands in Example 15-2 will either return the first IP address that is not a loopback or nothing if there are no interfaces configured or up.

# cookbook filename: finding_ipas

# IPv4 Using awk, cut, and head
$ /sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | head -1

# IPv4 Using Perl, just for fun
$ /sbin/ifconfig -a | perl -ne 'if ( m/^\s*inet (?:addr:)?([\d.]+).*?cast/ )
> { print qq($1\n); exit 0; }'

# IPv6 Using awk, cut, and head
$ /sbin/ifconfig -a | egrep 'inet6 addr: |address: ' | cut -d':' -f2- \
    | cut -d'/' -f1 | head -1 | tr -d ' '

# IPv6 Using Perl, just for fun
$ /sbin/ifconfig -a | perl -ne 'if
> ( m/^\s*(?:inet6)? \s*addr(?:ess)?: ([0-9A-Fa-f:]+)/ ) { print qq($1\n);
> exit 0; }'

Second, you can get your hostname and resolve it back to an IP address. This is often unreliable because today’s systems (especially workstations) might have incomplete or incorrect hostnames and/or might be on a dynamic network that lacks proper reverse lookup. Use at your own risk and test well:

host $(hostname)

Third, you may be more interested in your host’s external, routable address than its internal RFC 1918 address. In that case you can use an external host such as http://whatismyip.akamai.com, http://checkip.amazonaws.com/, http://ipinfo.io/, or others to learn the address of your firewall or NAT device. The catch here is that non-Linux systems often have no command-line tool like wget installed by default. lynx or curl will also work, but they aren’t usually installed by default either (although macOS 10.4+ has curl). Note the IP address and other information is deliberately obscured in the following examples:

$ wget -qO - http://ipinfo.io/
{
  "ip": "8.8.8.8",
  "hostname": "google-public-dns-a.google.com",
  "city": "Mountain View",
  "region": "California",
  "country": "US",
  "loc": "37.3860,-122.0840",
  "org": "AS15169 Google Inc.",
  "postal": "94035",
  "phone": "650"
}

$ wget -qO - http://ipinfo.io/ip/
72.NN.NN.225

$ lynx -dump http://ipinfo.io/ip/
   72.NN.NN.225

$ curl whatismyip.akamai.com
72.NN.NN.225

$ curl http://checkip.amazonaws.com
72.NN.NN.225

If you do not have any of the programs used here, but you do have bash version 2.04+ compiled with --enable-net-redirections (it isn’t compiled this way prior to bash 4 in Debian and derivatives), you can use bash itself (see Recipe 15.9 for details).

$ exec 3<> /dev/tcp/checkip.dyndns.org/80
$ echo -e "GET / HTTP/1.0\n" >&3
$ cat <&3
HTTP/1.1 200 OK
Content-Type: text/html
Server: DynDNS-CheckIP/1.0
Connection: close
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 105

<html><head><title>Current IP Check</title></head>
<body>Current IP Address: 96.245.41.129</body></html>

$ exec 3<> /dev/tcp/checkip.dyndns.org/80
$ echo -e "GET / HTTP/1.0\n" >&3
$ egrep --only-matching 'Current IP Address: [0-9.]+' <&3
Current IP Address: 72.NN.NN.225
$

Discussion

The awk and Perl code in the first solution is interesting because of the operating system variations we will note here. But it turns out that the lines we’re interested in all contain either Bcast or broadcast (or inet6addr: or address:), so once we get those lines it’s just a matter of parsing out the field we want. Of course Linux makes that harder by using a different format, but we’ve dealt with that too.

Not all systems require the path (if you aren’t root) or -a argument to ifconfig, but all accept it, so it’s best to use /sbin/ifconfig -a and be done with it.

Here are some ifconfig output examples from different machines:

# Linux
$ /sbin/ifconfig
eth0      Link encap:Ethernet HWaddr 00:C0:9F:0B:8F:F6
          inet addr:192.168.99.11 Bcast:192.168.99.255 Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:33073511 errors:0 dropped:0 overruns:0 frame:827
          TX packets:52865023 errors:0 dropped:0 overruns:1 carrier:7
          collisions:12922745 txqueuelen:100
          RX bytes:2224430163 (2121.3 Mb) TX bytes:51266497 (48.8 Mb)
          Interrupt:11 Base address:0xd000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:659102 errors:0 dropped:0 overruns:0 frame:0
          TX packets:659102 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:89603190 (85.4 Mb) TX bytes:89603190 (85.4 Mb)

$ /sbin/ifconfig
eth0      Link encap:Ethernet HWaddr 00:06:29:33:4D:42
          inet addr:192.168.99.144 Bcast:192.168.99.255 Mask:255.255.255.0
          inet6 addr: fe80::206:29ff:fe33:4d42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1246774 errors:14 dropped:0 overruns:0 frame:14
          TX packets:1063160 errors:0 dropped:0 overruns:0 carrier:5
          collisions:65476 txqueuelen:1000
          RX bytes:731714472 (697.8 MiB) TX bytes:942695735 (899.0 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:144664 errors:0 dropped:0 overruns:0 frame:0
          TX packets:144664 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:152181602 (145.1 MiB) TX bytes:152181602 (145.1 MiB)

sit0      Link encap:IPv6-in-IPv4
          inet6 addr: ::127.0.0.1/96 Scope:Unknown
          UP RUNNING NOARP MTU:1480 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:101910 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

# NetBSD
$ /sbin/ifconfig -a
pcn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:0c:29:31:eb:19
        media: Ethernet autoselect (autoselect)
        inet 192.168.99.56 netmask 0xffffff00 broadcast 192.168.99.255
        inet6 fe80::20c:29ff:fe31:eb19%pcn0 prefixlen 64 scopeid 0x1
lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33196
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
ppp0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500
ppp1: flags=8010<POINTOPOINT,MULTICAST> mtu 1500
sl0: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 296
sl1: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 296
strip0: flags=0 mtu 1100
strip1: flags=0 mtu 1100

# OpenBSD, FreeBSD
$ /sbin/ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
le1: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:0c:29:25:df:00
        inet6 fe80::20c:29ff:fe25:df00%le1 prefixlen 64 scopeid 0x1
        inet 192.168.99.193 netmask 0xffffff00 broadcast 192.168.99.255
pflog0: flags=0<> mtu 33224
pfsync0: flags=0<> mtu 2020

# Solaris
$ /sbin/ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
pcn0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2
        inet 192.168.99.159 netmask ffffff00 broadcast 192.168.99.255

# Mac
$ /sbin/ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet6 fe80::20d:93ff:fe65:f720%en0 prefixlen 64 scopeid 0x4
        inet 192.168.99.155 netmask 0xffffff00 broadcast 192.168.99.255
        ether 00:0d:93:65:f7:20
        media: autoselect (100baseTX <half-duplex>) status: active
        supported media: none autoselect 10baseT/UTP <half-duplex> 10baseT/UTP <full-
duplex>10baseT/UTP <full-duplex,hw-loopback> 100baseTX <half-duplex> 100baseTX
<full-duplex> 100baseTX <full-duplex,hw-loopback>
fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 2030
    lladdr 00:0d:93:ff:fe:65:f7:20
        media: autoselect <full-duplex> status: inactive
        supported media: autoselect <full-duplex>

See Also

• man awk

• man curl

• man cut

• man head

• man lynx

• man perl

• man wget

• http://checkip.amazonaws.com

• http://checkip.dyndns.org

• http://whatismyip.akamai.com

• http://ipinfo.io

• http://xmodulo.com/how-to-find-the-public-ip-address-from-command-line.html

• http://www.faqs.org/rfcs/rfc1918.html

• Recipe 15.9, “Using bash Net-Redirection”

• Recipe 15.12, “Redirecting Output for the Life of a Script”

Problem

Your script needs to get input from another machine, perhaps to check if a file exists or a process is running.

Solution

Use SSH with public keys and command substitution. To do this, set up SSH so that you do not need a password, as described in Recipe 14.21. Next, tailor the command that SSH runs to output exactly what your script needs as input. Then simply use command substitution (see Example 15-3).

#!/usr/bin/env bash
# cookbook filename: command_substitution

REMOTE_HOST='host.example.com'  # Required
REMOTE_FILE='/etc/passwd'       # Required
SSH_USER='user@'                # Optional, set to '' to not use
#SSH_ID='-i ~/.ssh/foo.id'       # Optional, set to '' to not use
SSH_ID=''

result=$(
    ssh $SSH_ID $SSH_USER$REMOTE_HOST \
      "[ -r $REMOTE_FILE ] && echo 1 || echo 0"
) || { echo "SSH command failed!" >&2; exit 1; }

if [ $result = 1 ]; then
    echo "$REMOTE_FILE present on $REMOTE_HOST"
else
    echo "$REMOTE_FILE not present on $REMOTE_HOST"
fi

Discussion

We do a few interesting things here. First, notice how both $SSH_USER and $SSH_ID work. They have an effect when they have a value, but when they are empty they interpolate to the empty set and are ignored. This allows us to abstract the values in the code, which lends itself to putting those values in a configuration file, putting the code into a function, or both:

# Interpolated line of the variables have values:
ssh -i ~/.ssh/foo.id user@host.example.com [...]

# No values:
ssh host.example.com [...]

Next, we set up the command that SSH runs so that there is always output (0 or 1), then check that $result is not empty. That’s one way to make sure that the SSH command runs (see also Recipe 4.4). If $result is empty, we group commands using a {} code block to issue an error message and exit. But since we’re always getting output from the SSH command, we have to test the value; we can’t just use if [$result]; then.

If we didn’t use the code block, we’d only issue the warning if the SSH command returned an empty $result, but we’d always exit. Read the code again until you understand why, because this is an easy way to get bitten. Likewise, if we’d tried to use a () subshell instead of the {} code block, our intent would fail because the exit 1 would exit the subshell, not the script. The script would then continue even after the SSH command had failed—but the code would look almost correct, so this might be tricky to debug.

We could have written the last test case as follows:

[ $result = 1 ] && echo "$REMOTE_FILE present on $REMOTE_HOST" \
                || echo "$REMOTE_FILE not present on $REMOTE_HOST"

Which form to use depends on your style and the number of statements to execute in each situation. In this case it doesn’t matter.

Finally, we’ve also been careful about formatting so that no lines are too long, but the code is still readable and our intent is clear.

See Also

• Recipe 2.14, “Saving or Grouping Output from Several Commands”

• Recipe 4.4, “Telling Whether a Command Succeeded or Not”

• Recipe 14.21, “Using SSH Without a Password”

• Recipe 17.20, “Grepping ps Output Without Also Getting the grep Process Itself”

• Recipe 17.21, “Finding Out Whether a Process Is Running”

Problem

You’d like to redirect output for an entire script, and you’d rather not have to edit every echo or printf statement.

Solution

Use a little-known feature of the exec command to redirect STDOUT or STDERR:

# Optional, save the "old" STDERR
exec 3>&2

# Redirect any output to STDERR to an error logfile instead
exec 2> /path/to/error_log

# Script with "globally" redirected STDERR goes here

# Turn off redirect by reverting STDERR and closing FH3
exec 2>&3-

Discussion

Usually exec replaces the running shell with the command supplied in its arguments, destroying the original shell. However, if no command is given, it can manipulate redirection in the current shell. You are not limited to redirecting STDOUT or STDERR, but they are the most common targets for redirection in this case.

See Also

• help exec

• Recipe 15.9, “Using bash Net-Redirection”

Problem

You get an “Argument list too long” error while trying to do an operation involving shell wildcard expansion.

Solution

Use the xargs command, possibly in conjunction with find, to break up your argument list.

For simple cases, just use a for loop or find instead of ls:

$ ls /path/with/many/many/files/*e*
-/bin/bash: /bin/ls: Argument list too long

# Short demo, surrounding ~ are for illustration only
$ for i in ./some_files/*e*; do echo "~$i~"; done
~./some_files/A file with (parens)~
~./some_files/A file with [brackets]~
~./some_files/File with embedded
newline~
~./some_files/file with = sign~
~./some_files/file with spaces~
~./some_files/file with |~
~./some_files/file with:~
~./some_files/file with;~
~./some_files/regular_file~


$ find ./some_files -name '*e*' -exec echo ~{}~ \;
~./some_files~
~./some_files/A file with [brackets]~
~./some_files/A file with (parens)~
~./some_files/regular_file~
~./some_files/file with spaces~
~./some_files/file with = sign~
~./some_files/File with embedded
newline~
~./some_files/file with;~
~./some_files/file with:~
~./some_files/file with |~


$ for i in /path/with/many/many/files/*e*; do echo "$i"; done
[This works, but the output is too long to list]


$ find /path/with/many/many/files/ -name '*e*'
[This works, but the output is too long to list]

This example works correctly with the echo command, but when you feed that "$i" into other programs, especially other shell constructs, $IFS and other parsing may come into play. The GNU find and xargs take that into account with find - print0 and xargs -0. (No, we don’t know why it’s -print0 and -0 instead of being consistent.) These arguments cause find to use the null character (which can’t appear in a filename) instead of whitespace as an output record separator, and xargs to use null as its input record separator. That will correctly parse files containing odd characters:

find /path/with/many/many/files/ -name '*e*' -print0 | xargs -0 proggy

Discussion

Note that the default behavior of bash (and sh) is to return unmatched patterns unchanged. That means you could end up with your for loop setting $i to ./some_files/*e* if no files match the wildcard pattern. You can set the shopt -s nullglob option to cause filename patterns that match no files to expand to a null string, rather than expanding to themselves.

You might assume that the for loop solution in the simple case would run into the same problem as the ls command, but it doesn’t. Chet Ramey tells us:

ARG_MAX bounds the total space requirement of the exec* family of system calls, so the kernel knows the largest buffer it will have to allocate. This is all three arguments to execve: program name, argument vector, and environment.

The [ls command] fails because the total bytes taken up by the arguments to execve exceeds ARG_MAX. The [for loop] succeeds because everything is done internally: though the entire list is generated and stored, execve is never called.

Be careful that find doesn’t find too many files, since it will recursively descend into all subdirectories by default while ls will not. Some versions of find have a -maxdepth option to control how deep it goes. Using the for loop may be easier.

Use the getconf ARG_MAX command to see what the limit is on your system. It varies wildly (see also getconf LINE_MAX;). Table 15-1 lists some examples.

See Also

• Question 19 in the GNU Core Utilities FAQ

• Recipe 9.2, “Handling Filenames Containing Odd Characters”

Problem

You’d like your script to be able to log to syslog.

Solution

Use logger, Netcat, or bash’s built-in network redirection features.

logger is installed by default on most systems and is an easy way to send messages to the local syslog service:

logger -p local0.notice -t ${0##*/}[$$] test message

However, it does not send syslog to remote hosts by itself. If you need to do that, you can use bash:

echo "<133>${0##*/}[$$]: Test syslog message from bash" \
  > /dev/udp/loghost.example.com/514

or Netcat:

echo "<133>${0##*/}[$$]: Test syslog message from Netcat" | nc -w1 -u loghost 514

Netcat is known as the “TCP/IP Swiss Army knife” and is usually not installed by default. It may also be prohibited as a hacking tool by some security policies, though bash’s net-redirection features do pretty much the same thing. See the discussion in Recipe 15.9 for details on the <133>${0##*/}[$$] part.

Discussion

logger and Netcat have many more features than we include here. See the respective manpages for details.

See Also

• man logger

• man nc

• Recipe 15.9, “Using bash Net-Redirection”

Problem

You want to use the logger tool so your script can send syslog messages, but the defaults do not provide enough useful information.

Solution

Use logger as follows:

logger -t "${0##*/}[$$]" 'Your message here'

Discussion

In our opinion, failing to use the -t option to logger should at least trigger a warning, if not a fatal error. The t is for “tag,” and as the manpage says it will “[mark] every line to be logged with the specified tag.” In other words, without -t you will have a hard time telling where your message came from!

The tag of ${0##*/}[$$] may look like gibberish, but it’s actually what you usually see when you look at syslog lines. It is just the basename of your script and the process ID ($$) in square brackets. Compare the command with and without the -t option:

$ logger -t "${0##*/}[$$]" 'Your message here'
$ tail -1 /var/log/syslog
Oct 26 12:16:01 hostname yourscript[977]: Your message here
$ logger 'Your message here'
$ tail -1 /var/log/syslog
Oct 26 12:16:01 hostname Your message here
$

logger has other interesting options and it’s well worth reading the manpage, but be aware that some options may vary by age, version, and distribution, so you need to consider that if your script will run in the wild. For example, CentOS 5 and 6 versions of logger do not have the very useful -n option that the Debian/Ubuntu version has:

    -n, --server server
        Write to the specified remote syslog server using UDP instead of
        to the builtin syslog routines.

See Also

• Recipe 5.20, “Using bash for basename”

• Recipe 11.10, “Logging with Dates”

• Recipe 15.14, “Logging to syslog from Your Script”

• man logger

Problem

You’d like your script to be able to send email, optionally with attachments.

Solution

These solutions depend on your system having a compatible mailer (such as mail, mailx, or mailto), a message transfer agent (MTA) being installed and running, and proper configuration of your email environment. Unfortunately, you can’t always count on all of that, so these solutions must be well tested in your intended environment.

The first way to send mail from your script is to write some code to generate and send a message, as follows:

# Simple
cat email_body | \
mail -s "Message subject" recipient1@example.com recipient2@example.com

or:

# Attachment only
uuencode /path/to/attachment_file attachment_name | \
mail -s "Message Subject" recipient1@example.com recipient2@example.com

or:

# Attachment and body
(cat email_body ; uuencode /path/to/attachment_file attachment_name) | \
mail -s "Message Subject" recipient1@example.com recipient2@example.com

In practice, it’s not always that easy. For one thing, while uuencode will probably be there, mail and friends may or may not, and their capabilities may vary. In some cases mail and mailx are even the same program, hard- or soft-linked together. In production, you will want to use some abstraction to allow for portability. For example, mail works on Linux and the BSDs, but mailx is required for Solaris since its mail lacks support for -s. mailx works on some Linux distributions (e.g., Debian), but not others. We choose the mailer based on hostname in Example 15-4, but depending on your environment using uname -o might make more sense.

# cookbook filename: email_sample

# Define some mail settings. Use a case statement with uname or hostname
# to tweak settings as required for your environment.
case $HOSTNAME in
    *.company.com     ) MAILER='mail'   ;;  # Linux and BSD
    host1.*           ) MAILER='mailx'  ;;  # Solaris, BSD, and some Linuxes
    host2.*           ) MAILER='mailto' ;;  # Handy, if installed
esac
RECIPIENTS='recipient1@example.com recipient2@example.com'
SUBJECT="Data from $0"

[...]
# Create the body as a file or variable using echo, printf, or a here-document
# Create or modify $SUBJECT and/or $RECIPIENTS as needed
[...]

( echo $email_body ; uuencode $attachment $(basename $attachment) ) \
  | $MAILER -s "$SUBJECT" "$RECIPIENTS"

We should also note that sending attachments in this way depends somewhat on the client you use to read the resulting message. Modern clients like Thunderbird (and Outlook) will detect a uuencoded message and present it as an attachment. Other clients may not. You can always save the message and uudecode it (uudecode is smart enough to skip the message part and just handle the attachment part), but that’s a pain.

The second way to send mail from your scripts is to outsource the task to cron. While the exact feature set of cron varies from system to system, one thing in common is that any output from a cron job is mailed to the job’s owner or the user defined using the MAILTO variable. You can take advantage of that fact to get emailing for free, assuming that your email infrastructure works.

The proper way to design a script intended to run from cron (and, many would argue, any script or Unix tool at all) is to make it silent unless it encounters a warning or error. If necessary, use a -v argument to optionally allow a more verbose mode, but don’t run it that way from cron, at least after you’ve finished testing. The reason for this is as noted: cron emails you all the output. If you get an email message from cron every time your script runs, you’ll soon start ignoring them. But if your script is silent except when there’s a problem, you’ll only get a notification when there is a problem, which is ideal.

Discussion

Note that mailto is intended to be a multimedia- and MIME-aware update to mail, and thus you could avoid using uuencode for sending attachments, but it’s not as widely available as mail or mailx. If all else fails, elm or mutt may be used in place of mail, mailx, or mailto, thought they are even less likely to be installed by default than mail*. Also, some versions of these programs support a -r option to supply a return address in case you want to supply one. mutt also has a -a option that makes sending attachments a breeze:

cat "$message_body" | mutt -s "$subject" -a "$attachment_file" "$recipients"

mpack is another tool worth looking into, but it is very unlikely to be installed by default. Check your system’s software repository or download the source. From the manpage:

The mpack program encodes the named file in one or more MIME messages. The resulting messages are mailed to one or more recipients, written to a named file or set of files, or posted to a set of newsgroups.

Another way to handle the various names and locations of mail clients is shown in Chapter 8 of Classic Shell Scripting by Nelson H. F. Beebe and Arnold Robbins (O’Reilly), reprinted here as Example 15-5.

# cookbook filename: email_sample_css
# From Chapter 8 of Classic Shell Scripting

for MAIL in /bin/mailx /usr/bin/mailx /usr/sbin/mailx /usr/ucb/mailx /bin/mail \
/usr/bin/mail; do
    [ -x $MAIL ] && break
done
[ -x $MAIL ] || { echo 'Cannot find a mailer!' >&2; exit 1; }

uuencode is an old method for translating binary data into ASCII text for transmission over links that could not support binary, which is to say most of the internet before it became the internet and the web. We have it on good authority that at least some such links still remain, but even if you never encounter one it’s still useful to be able to convert an attachment into an otherwise ASCII medium in such a way that modern mail clients will recognize it. See also uudecode and mimeencode. Note that uuencoded files are about one-third larger than their binary equivalent, so you probably want to compress the file before uuencoding it.

The problem with email, aside from the differing frontend mail user agent (MUA) programs like mail and mailx, is that there are a lot of moving parts that must all work together. This is exacerbated by the spam problem: mail administrators have had to so severely lock down mail servers that it can easily affect your scripts. All we can say here is to fully test your solution, and talk to your system and mail administrators if necessary.

One other problem you might see is that some workstation-oriented Linux distributions, such as Ubuntu, don’t install or run an MTA by default since they assume you will be using a full-featured GUI client such as Evolution or Thunderbird. If that’s the case, command-line MUAs and email from cron won’t work either. Consult your distribution’s support groups for help with this as needed.

See Also

• man mail

• man mailx

• man mailto

• man mutt

• man uuencode

• man cron

• man 5 crontab

Problem

You have a long job or process you need to automate, but it may require manual intervention and you need to be able to restart at various points in the progress. You might use a GOTO to jump around, but bash doesn’t have that.

Solution

Use a case statement to break your script up into sections or phases.

First, we’ll define a standardized way to get answers from the user using Example 15-6, from Recipe 3.6.

# cookbook filename: func_choice.1

# Let the user make a choice about something and return a standardized
# answer. How the default is handled and what happens next is up to
# the if/then after the choice in main.
# Called like: choice <prompt>
# e.g. choice "Do you want to play a game?"
# Returns: global variable CHOICE
function choice {

    CHOICE=''
    local prompt="$*"
    local answer

    read -p "$prompt" answer
    case "$answer" in
        [yY1] ) CHOICE='y';;
        [nN0] ) CHOICE='n';;
        *     ) CHOICE="$answer";;
    esac
} # end of function choice

Then, we’ll set up our phases as shown in Example 15-7.

# cookbook filename: using_phases

# Main loop
until [ "$phase" = "Finished." ]; do

    case $phase in

        phase0 )
            ThisPhase=0
            NextPhase="$(( $ThisPhase + 1 ))"
            echo '############################################'
            echo "Phase$ThisPhase = Initialization of FooBarBaz build"
            # Things that should only be initialized at the beginning of a
            # new build cycle go here
# ...
            echo "Phase${ThisPhase}=Ending"
            phase="phase$NextPhase"
        ;;


# ...


        phase20 )
        ThisPhase=20
            NextPhase="$(( $ThisPhase + 1 ))"
            echo '############################################'
            echo "Phase$ThisPhase = Main processing for FooBarBaz build"


# ...


            choice "[P$ThisPhase] Do we need to stop and fix anything? [y/N]: "
            if [ "$choice" = "y" ]; then
                echo "Re-run '$MYNAME phase${ThisPhase}' after handling this."
                exit $ThisPhase
            fi

            echo "Phase${ThisPhase}=Ending"
            phase="phase$NextPhase"
        ;;


# ...


        * )
            echo "What the heck?!? We should never get HERE! Gonna croak!"
            echo "Try $0 -h"
            exit 99
            phase="Finished."
        ;;
    esac
    printf "%b" "\a"        # Ring the bell
done

Discussion

Since exit codes only go up to 255, the exit $ThisPhase line limits you to that many phases. And our exit 99 line limits you even more, although that one is easily adjusted. If you require more than 254 phases (plus 255 as the error code), you have our sympathy. You can either come up with a different exit code scheme, or chain several scripts together.

You should probably set up a usage and/or summary routine that lists the various phases:

Phase0 = Initialization of FooBarBaz build
...
Phase20 = Main processing for FooBarBaz build
...
Phase28 ...

You can probably grep most of the text out of the code with something like grep 'Phase$ThisPhase' my_script.

You may also want to log to a local flat file, syslog, or some other mechanism. In that case, define a function like logmsg and use it as appropriate in the code. It could be as simple as:

function logmsg {
    # Write a timestamped log message to the screen and logfile
    # Note tee -a to append
    # printf "%b" "$(date '+%Y-%m-%d %H:%M:%S'): $*" | tee -a $LOGFILE
    printf "%(%Y-%m-%d %H:%M:%S)T: %b\n" -1 "$*" | tee -a $LOGFILE
} # end of function logmsg

This function uses the newer printf format that supports time and date values. If you are using an older shell (before version 4), switch the printf with the commented printf line in this function.

You may note that this larger script violates our usual standard of being silent unless it encounters a problem. Since it is designed to be interactive, we’re OK with that.

See Also

• Recipe 3.5, “Getting User Input”

• Recipe 3.6, “Getting Yes or No Input”

• Recipe 11.10, “Logging with Dates”

• Recipe 15.14, “Logging to syslog from Your Script”

Problem

A pipeline of commands goes only one way, each process writing to the next in line. Can two processes converse with each other, each reading as its input the output of the other command?

Solution

Yes! As of version 4 of bash, the coproc command can do just that.

Example 15-8 is a simple example that uses the bc program, an arbitrary-precision calculator language, as a coprocess, allowing bash to send calculations to bc and read back the results. It’s one way of giving bash the ability to do floating-point calculations, though we’re only using it here as an example of the coproc command.

# cookbook filename: fpmath
# using coproc for floating-point math

# initialize the coprocess
#   call this first
#   before attempting any calls to fpmath
function fpinit ()
{
    coproc /usr/bin/bc

    bcin=${COPROC[1]}
    bcout=${COPROC[0]}
    echo "scale=5" >& ${bcin}
}

# compute with floating-point numbers
#   by sending the args to bc
#   then reading its response
function fpmath()
{
    echo "$@" >& ${bcin}
    if read -t 0.25 -u ${bcout} responz
    then
       echo "$responz"
    fi
}

############################
# main

fpinit

while read aline
do
    answer=$(fpmath "$aline")
    if [[ -n $answer ]]
    then
       echo $answer
    fi
done

Discussion

For our example we define two functions, fpinit and fpmath. The purpose of fpinit is to set up the coprocess. The purpose of fpmath is to get a floating-point calculation done by sending the request to the coprocess and reading back the result. To demonstrate these functions we wrote a while loop that prompts the user for input, then sends that input to the coprocess and reads back the result.

coproc will execute a command (or pipeline of commands) alongside the current shell process. In our case we gave it /usr/bin/bc (though a full path is not required; the shell will search $PATH as with any command). Furthermore, it creates two pipes, one connected to the standard output of the command and one connected to its standard input. These connections are stored in a shell array called COPROC by default. Index 0 of that array holds the output file descriptor of that process; index 1 holds the input file descriptor of that process.

That may seem backward to a systems programmer, but remember that the output of the coprocess can act as the input to the calling process (the shell script), and vice versa. To keep their usage clearer we assigned them to variables that describe how we will use them. We chose $bcin to hold the file descriptor to be used to send input to the bc command and $bcout to hold the file descriptor to be used to read its output.

We use these file descriptors in our fpmath function. To send a calculation to the bc process we echo the text of a calculation (for example, "3.4 * 7.52") and redirect that output to the input file descriptor. In our example, that means that we redirect to bcin. To get the result back from bc we use the read command, which has an option (-u) that lets us specify the file descriptor from which to read. In our case we use $bcout.

We’ve also used the -t option on the read command. That option sets a timeout value after which the read will return, possibly empty-handed. We use that here since not every valid command to bc will result in output. (For example, "x=5" will store the value 5 in the variable x but will generate no output.) Versions of bash that are new enough to have the coproc command are also new enough to support a fractional value for the timeout value. Older versions only allowed integers.

See Also

• man bash

• help coproc

Problem

You need to run a Command over SSH on Multiple Hosts.

Solution

Wrap your SSH command in a for loop:

$ for host in host1 host2 host3; do echo -n "On $host, I am: " ;
> ssh $host 'whoami' ; done
On host1, I am: root
On host2, I am: jp
On host3, I am: jp
$

Discussion

This looks very easy, and it is when everything works, but there are a few points to keep in mind.

First, all of the underlying networking and the firewall, DNS, and similar aspects have to already be working.

Second, while not strictly necessary, it’s much more convenient to do this when using SSH keys, so you’ll want to read Recipe 14.21.

Third, you can quickly run into quoting issues in the SSH command. For example, consider:

$ for host in host{1..3};
> do echo "$host:" ; 
> ssh $host 'grep "$HOSTNAME" /etc/hosts' ; 
> done

That’s straightforward—we enclose the ssh command in single quotes so that our local bash shell will not interpolate it, and we enclose our grep argument in double quotes for clarity, since that is not strictly needed. But what if we have some variables that our local bash needs to interpolate and others that the remote bash must handle? Or what if we actually need to grep for single quotes?

We can handle those problems by enclosing the ssh command in double quotes, then escaping any variables and/or double quotes needed on the remote side, but it gets ugly fast:

$ for host in host{1..3};
> do ssh $host "echo \"Local '$host' is remote '\$HOSTNAME'\""; 
> done
Local 'host1' is remote 'host1'
Local 'host2' is remote 'host2'
Local 'host3' is remote 'host3'
$

We would like to point out that you can do amazing things in the OpenSSH configuration file, and it’s well worth spending some time learning about, but unfortunately that is well beyond the scope of this book.

We should also point out that while this can be a handy technique, you are better off learning and using a real configuration management system (CMS) for these kinds of tasks. We really like Ansible, but there are many options, including at least one written in bash itself: https://github.com/wffls/waffles.

See Also

• Recipe 14.21, “Using SSH Without a Password”

• Recipe 14.22, “Restricting SSH Commands”

• man ssh

• man ssh_config

• https://github.com/wffls/waffles

• https://www.ansible.com/