Unattended SSH: Batch or cron Jobs

SSH isn’t only a great interactive tool, but also a resource for automation. Batch scripts, cron jobs, and other automated tasks can benefit from the security provided by SSH, but only if implemented properly. The major challenge is authentication: how can a client prove its identity when no human is available to type a password or passphrase? (We’ll just write “password” from now on to mean both.) You must carefully select an authentication method, and then equally carefully make it work. Once this infrastructure is established, you must invoke ssh properly to avoid prompting the user. In this case study, we discuss the pros and cons of different authentication methods for operating an SSH client unattended.

Note that any kind of unattended authentication presents a security problem and requires compromise, and SSH is no exception. Without a human present when needed to provide credentials (type a password, provide a thumbprint, etc.), those credentials must be stored persistently somewhere on the host system. Therefore, an attacker who compromises the system badly enough can use those credentials to impersonate the program and gain whatever access it has. Selecting a technique is a matter of understanding the pros and cons of the available methods, and picking your preferred poison.

11.1.1 Password Authentication

Rule number 1: forget password authentication if you care about the security of your batch jobs. As we mentioned, authentication for any unattended process will require some kind of persistent secret lying around, so it might seem that a password in a protected file will do as well as anything else, and password authentication is simple. In a strict sense that’s correct, but it’s a bad idea both practically and securitywise. Embedding a password in a command line is unwise: it may be exposed to other users by simple commands such as ps, end up in shell history files (e.g. ~/.bash_history) or system logs, etc. In fact, most SSH clients deliberately require terminal input (a “tty”) for a password, precisely to discourage this. You can use a tool like Expect to get around this limitation, but that will be awkward. Another practical limitation is that more methods tend to be available on the server side to restrict logins with public-key authentication, e.g., the “command” parameters in ~/.ssh/authorized_keys (OpenSSH) and ~/.ssh2/authorization (Tectia). This is just an implementation detail, but it’s very relevant since you definitely want to restrict unattended logins to do just what they’re intended to do.

More generally, compared to other available methods, SSH password authentication is just inherently weak: passwords tend to be short and often guessable, and the client must reveal the password to the server as part of the authentication process; so if the server has been compromised, it will get your password. Public-key authentication, however, does not reveal the private key in the process.

In the real world, though, you might be stuck using password authentication anyway. Perhaps you have to automate a transaction with a server not under your control; it only supports passwords, and you can’t get that changed. If you must, we suggest co-opting the “askpass” facility if it’s available. [6.3.3] The ssh-askpass program normally displays a window prompting for the password, but it can use instead a program that provides the password from wherever you’re storing it. It does so via a pipe, which is much better than letting it appear on a command line.

11.1.2 Public-Key Authentication

In public-key authentication, a private key is the client’s credentials. Therefore, the batch job needs access to the key, which must be stored where the job can access it. You have three choices of location for the key, which we discuss separately:

Store the encrypted key and its passphrase in the filesystem.
Store a plaintext (unencrypted) private key in the filesystem, so it doesn’t require a passphrase.
Store the key in an agent, which keeps secrets out of the filesystem but requires a human to decrypt the key at system boot time.

11.1.2.1 Storing the passphrase in the filesystem

In this technique, you store an encrypted key and its passphrase in the filesystem so that a script can access them. We don’t recommend this method, since you can store an unencrypted key in the filesystem with the same level of security (and considerably less complication). In either case, you rely solely on the filesystem’s protections to keep the key secure. This observation is the rationale for the next technique.

11.1.2.2 Using a plaintext key

A plaintext or unencrypted key requires no passphrase. To create one, run ssh-keygen and simply press the Return key when prompted for a passphrase (or similarly, remove the passphrase from an existing key using ssh-keygen -p). You can then supply the key filename on the ssh command line using the -i option, or in the client configuration file with the IdentityFile keyword. [7.4.2]

Usually plaintext keys are undesirable, equivalent to leaving your password in a file in your account. They are never a good idea for interactive logins, since the SSH agent provides the same benefits in a much more secure fashion. But a plaintext key is a viable option for automation, since the unattended aspect forces us to rely on some kind of persistent state in the machine. The filesystem is one possibility.

Plaintext keys are frightening, though. To steal the key, an attacker needs to override filesystem protections only once, and this doesn’t necessarily require any fancy hacking: stealing a single backup tape will do. You can arrange to keep them off backups, but that’s an additional complication. If you need your batch jobs to continue working after an unattended system restart, plaintext keys are pretty much your best option. If the situation allows for some leeway in this regard, however, consider using ssh-agent instead.

11.1.2.3 Using an agent

ssh-agent provides another, somewhat less vulnerable method of key storage for batch jobs. A human invokes an agent and loads the needed keys from passphrase-protected key files, just once. Thereafter, unattended jobs use this long-running agent for authentication.

In this case, the keys are still in plaintext but within the memory space of the running agent rather than in a file on disk. As a matter of practical cracking, it is more difficult to extract a data structure from the address space of a running process than to gain illicit access to a file. Also, this solution avoids the problem of an intruder walking off with a backup tape containing the plaintext key.

Security can still be compromised by other methods, though. The agent provides access to its services via a Unix-domain socket, which appears as a node in the filesystem. Anyone who can read and write that socket might be able to instruct the agent to sign authentication requests and thus gain use of the keys. Some agent implementations attempt further checks, such as ensuring the communicating process runs under the same uid, but not all flavors of Unix support this. [6.3.4.1] In any event, this compromise isn’t quite so devastating since the attacker can’t obtain the actual keys through the agent socket. She merely gains use of the keys for as long as the agent is running and as long as she can maintain her compromise of the host.

The agent method does have a down side: the system can’t continue unattended after a reboot. When the host comes up again automatically, the batch jobs won’t have their keys until someone shows up to restart the agent and provide the passphrases to load the keys. This is just a cost of the improved security, and you have a pager, right?

Another bit of complication with the agent method is that you must arrange for the batch jobs to find the agent. SSH clients locate an agent via an environment variable pointing to the agent socket, such as SSH_AUTH_SOCK for the OpenSSH agent. [6.3.2.1] When you start the agent for batch jobs, you need to record its output where the jobs can find it. For instance, if the job is a shell script, you can store the environment values in a file:

    $ ssh-agent | head -2 > ~/agent-info
    $ cat ~/agent-info
    setenv SSH_AUTH_SOCK /tmp/ssh-res/ssh-12327-agent;
    setenv SSH_AGENT_PID 12328;

You can add keys to the agent (assuming C-shell syntax here):

    $ source ~/agent-info
    $ ssh-add batch-key
    Need passphrase for batch-key (batch job SSH key).
    Enter passphrase: **************

then instrument any scripts to set the same values for the environment variables:

    #!/bin/csh
    # Source the agent-info file to get access to our ssh-agent.
    set agent = ~/agent-info
    if (-r $agent) then
      source $agent
    else
      echo "Can't find or read agent file; exiting."
      exit 1
    endif
    # Now use SSH for something...
    ssh -q -o 'BatchMode yes' user@remote-server my-job-command

You also need to ensure that the batch jobs (and nobody else!) can read and write the socket. If there’s only one uid using the agent, the simplest thing to do is start the agent under that uid (e.g., as root, do su <batch_account> ssh-agent ...). If multiple uids are using the agent, you must adjust the permissions on the socket and its containing directory so that these uids can all access it, perhaps using group permissions.

Tip

Some operating systems behave oddly with respect to permissions on Unix-domain sockets. Some versions of Solaris, for example, completely ignore the modes on a socket, allowing any process at all full access to it. To protect a socket in such situations, set the containing directory to forbid access. For example, if the containing directory is mode 700, only the directory owner may access the socket. (This assumes there’s no other shortcut to the socket located elsewhere, such as a hard link.)

Using an agent for automation is more complicated and restrictive than using a plaintext key; however, it is more resistant to attack and doesn’t leave the key on disk and tape where it can be stolen. Considering that the agent is still vulnerable to being misused via the filesystem, and that it is intended to run indefinitely, the advantages of this method are debatable. Still, we recommend the agent method as the most secure and flexible strategy for automated SSH usage in a security-conscious environment.

11.1.3 Hostbased Authentication

If security concerns are relatively light, consider hostbased authentication for batch jobs. In this case, the “credentials” are the operating system’s notion of a process’s uid: the identity under which a process is running, which determines what rights it has over protected objects. An attacker need only manage to get control of a process running under your uid, to impersonate you to a remote SSH server. If he breaks root on the client, this is particularly simple, since root may create processes under any uid. The real crux, though, is the client host key: if the attacker gets that, he can sign bogus authentication requests presenting himself as any user at all, and sshd will believe them.

Hostbased authentication is in many ways the least secure SSH authentication method. [3.4.3.6] It leaves systems vulnerable to transitive compromise: if an attacker gains access to an account on host H, she immediately has access to the same account on all machines that trust H, with no further effort. Also, hostbased configuration is limited, fragile, and easy to get wrong. Public-key authentication affords both greater security and flexibility, particularly since you can restrict the commands that may be invoked and the client hosts that may connect, using its forced commands and other options in the authorization file.

Of course, if your security policy permits and you’re already using hostbased for general user authentication, then you’re all set for batch jobs too. However if you’re using something stronger for user authentication, and you’re considering the hostbased method for batch jobs, then we recommend that you:

Restrict its use to the batch accounts only (via /etc/shosts.equiv rules); continue to use stronger methods for interactive authentication.
Use only the SSH-specific configuration files /etc/shosts.equiv and ~/.shosts, and not the legacy files /etc/hosts.equiv and ~/.rhosts. This avoids any accidental changes to the behavior of wildly insecure mechanisms like rcmd and rsh.
Set options such as OpenSSH IgnoreRhosts and IgnoreUserKnownHosts, and Tectia AllowSHosts/DenySHosts, if possible. Since per-account hostbased configuration can override the systemwide files, it’s best to disable them.

11.1.4 Kerberos

There’s no reason to deploy Kerberos [11.4] solely in order to support batch jobs; it has no special overall advantage in this regard. However, if you’re already using Kerberos, you might want to keep things simple by using it for batch as well as interactive jobs. Unattended Kerberos usage has similar security properties to using a plaintext SSH key as described earlier: the Kerberos principal’s key is stored on disk, can be similarly strong since it does not have to be derived from a user-memorable passphrase, and is not revealed in the authentication process.

To do this, use the kadmin command:

    $ kadmin -q "ktadd -k keytab principal"

to store the principal’s key in the file keytab, and protect that file appropriately (e.g., so that only the Unix batch account can read it). The batch job can then call kinit:

    $ kinit -k -t keytab

to obtain Kerberos credentials for that principal.

We suggest the following arrangement:

Arrange that the keytab file does not travel insecurely over the network, e.g., on an unsecured NFS filesystem. Perhaps also arrange that it is not dumped to backup tapes.
Create separate principals for batch jobs; do not use existing user principals.
Create a random key for the batch principal using the kadmin option, addprinc -randkey.
If feasible, periodically change these keys. An advantage of the Kerberos system is that this does not require changing corresponding authorization entries, as changing a simple SSH key would require updating the matching authorized_keys files. Any running jobs will have to be restarted, though, since their credentials will become invalid.
As always, restrict what the batch principal can do on the server side, here using the Kerberos ~/.k5login or ~/.k5users files.

Kerberos-5 contains support for long-running jobs with “renewable” tickets, but note that this is still intended for jobs started interactively; it just supports those that may run for a long time. It is not intended as a solution for truly unattended jobs.

11.1.5 General Precautions for Batch Jobs

Regardless of the method you choose, some extra precautions will help secure your environment.

11.1.5.1 Least-privilege accounts

The account under which the automated job runs should have only those privileges needed to run the job, and no more. Don’t run every batch job as root just because it’s convenient. Arrange your filesystem and other protections so that the job can run as a less-privileged user. Remember that unattended remote jobs increase the risk of account compromise, so take the extra trouble to avoid the root account whenever possible.

11.1.5.2 Separate, locked-down automation accounts

Create accounts that are used solely for automation. Try not to run system batch jobs in a user account, since you might not be able to reduce its privileges to the small set necessary to support the job. In many cases, an automation account doesn’t even need to admit interactive logins. If jobs running under its uid are created directly by the batch job manager (e.g., cron), the account doesn’t need a password and should be locked.

11.1.5.3 Restricted-use keys

As much as possible, restrict the target account to perform only the work needed for the job. With public-key authentication, automated jobs should use keys that aren’t shared by interactive logins. Imagine that someday you might need to eliminate the key for security reasons, and you don’t want to affect other users or jobs by this change. For maximum control, use a separate key for each automated task. Additionally, place all possible restrictions on the key by setting options in the authorization file. [8.2] The command option restricts the key to running only the needed remote command, and the from option restricts usage to appropriate client hosts. Consider always adding the following options as well, if they don’t interfere with the job:

    no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

These make it harder to misuse the key should it be stolen.

If you’re using hostbased authentication, these restrictions aren’t available. In this case, it’s best to use a special shell for the account, which limits the commands that may be executed. Since sshd uses the target account’s shell to run any commands on the user’s behalf, this is an effective restriction. One standard tool is the Unix “restricted shell.” Confusingly, the restricted shell is usually named “rsh,” but has nothing to do with the Berkeley r-command for opening a remote shell, rsh.

11.1.5.4 Useful ssh options

When running SSH commands in a batch job, it’s a good idea to use these options:

    ssh -q -o 'BatchMode yes'

The -q option is for quiet mode, preventing SSH from printing a variety of warnings. This is sometimes necessary if you’re using SSH as a pipe from one program to another. Otherwise, the SSH warnings may be interpreted as remote program output and confuse the local program. [7.4.17]

The BatchMode keyword tells SSH not to prompt the user, who in this case doesn’t exist. This makes error reporting more straightforward, eliminating some confusing SSH messages about failing to access a tty. [7.4.6.4]

11.1.6 Recommendations

Our recommended method for best security with unattended SSH operation is public-key authentication with keys stored in an agent. If that isn’t feasible, hostbased or plaintext-key authentication may be used instead; your local security concerns and needs will determine which is preferable, using the foregoing discussion as a guideline.

To the extent possible, use separate accounts and keys for each job. By doing so, you limit the damage caused by compromising any one account, or stealing any one key. But of course, there is a complexity trade-off here; if you have 100 batch jobs, separate accounts or keys for each one may be too much to deal with. In that case, partition the jobs into categories according to the privileges they need, and use a separate account and/or key for each category of job.

You can ease the burden of multiple keys by applying a little automation to the business of loading them. The keys can all be stored under the same passphrase: a script prompts for the passphrase, then runs ssh-add multiple times to add the various keys. Or they have different passphrases, and the human inserts a diskette containing the passphrases when loading them. Perhaps the passphrase list itself is encrypted under a single password provided by the human. For that matter, the keys themselves can be kept on the key diskette and not stored on the filesystem at all: whatever fits your needs and paranoia level.