Table of Contents for

sendmail, 4th Edition

To ensure that new filenames are not the same as the names of files that might already be in the queue, sendmail uses the following pattern for each new ident:

AApid               ← prior to V8.6
hourAApid           ← beginning with V8.6
YMDhmsSEQpid        ← beginning with V8.10

Here, pid is the process identification number of the incarnation of sendmail that is trying to create the file. Because sendmail often fork(2)s to create queue entries, that pid is likely to be unique, resulting in a unique ident. The AA is used as a clock to prevent duplicate filenames. For V8.6 through V8.9 sendmail, an extra letter prefixes the AA. Shown as hour, it is an uppercase letter that corresponds to the hour (in a 24-hour clock) that the identifier was created. For example, a file created in hour three of the day will have a D prefixed (the hour begins at midnight with A).^[176]

For V8.10 sendmail, the identifier is constructed differently. Each character stands for (in this order, reading left to right): the year (minus 1900) modulo 60, the month, the day, the hour, the minute, the second, and a sequence within the second that starts at a random value. Each is used as an offset into a special array that looks like this:^[177]

0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwx

Thus, the following identifier:

lC9GgvB04136

means the year is 2007 (the l), the month is December (the C), the day is the 9th (the 9), the time is 16:42:57 (the Ggv), the sequence is 11 (the B), and the process ID of the process that created the file is 04136. The advantage to this algorithm is that no two identifiers will ever be the same during a given 60-year period. Although this latest method has stayed the same from V8.10 through V8.14, there is no guarantee that it will remain the same in future releases.

Prior to V8.10, if sendmail could not create an exclusive filename because a file with that identifier already existed, it clocked the second A of the AA to a B and tried again. It continued this process, clocking the righthand letter from A to Z and the lefthand letter from A to ˜ until it succeeded:

AA            ← start
AB            ← second try
AC            ← third try
... and so on
˜W
˜X
˜Y            ← last try
˜Z            ← failure

If it never succeeded, the ident became one like the following and sendmail failed:

hour˜Zpid

But this ident was unlikely to ever appear because the clocking provided for more than 1,600 possibilities.

All the files associated with a given mail message share the same ident as a part of their filenames. The individual files associated with a single mail message differ only in the first letter of their names.

All mail messages are composed of a header and a body. When queued, the body is stored in the df file.

Traditionally, the message body could contain only characters that had the high (most significant) bit turned off (cleared, set to 0). But under V8 sendmail, with a version 2 or higher configuration file (The V Configuration Command on page 580) the high bit is left as is until delivery (whereupon the F=7 delivery-agent flag, see F=7 on page 764, determines whether that bit will be stripped during delivery).

Because the message body can contain sensitive or personal information, the df file should be protected from reading by ordinary users. If the queue directory is world-readable, the TempFileMode option (TempFileMode on page 1097) should specify minimum permissions (such as 0600) for queued files. But if the queue directory is protected by both narrow permissions and a secure machine, the TempFileMode option can be relaxed for easier administration.

There is currently no plan to provide for encryption of df files. If you are concerned about the privacy of your message, you should use an end-to-end encryption package or an encrypting filesystem (not discussed in this book).

When old versions of sendmail process a queued message (attempt to redeliver it) they create an empty lock file. That lock file was needed to signal other running sendmail processes that the mail message was busy so that they shouldn’t try to deliver the message too. Current versions simply flock(2) or fcntl(2) lock the qf file.

O_CREAT|O_WRONLY|O_EXCL

/var/spool/mqueue/df (1 request)
 ----Q-ID---- --Size-- -----Q-Time----- ------------Sender/Recipient------------
 dB91UPA04168*       0 Wed Dec  8 17:30 <gw@wash.dc.gov>
             ↑
                                             <ben@franklin.edu
             note

Apr 12 00:33:38 ourhost sendmail[641]: dB91UPA04168: locked
Apr 12 01:22:14 ourhost sendmail[976]: dB91UPA04168: locked
Apr 12 02:49:23 ourhost sendmail[3251]: dB91XUs04170: locked
Apr 12 02:49:51 ourhost sendmail[5977]: dB91UPA04168: locked
Apr 12 03:53:05 ourhost sendmail[9839]: dB91UPA04168: locked

root      5338 160  -dB91UPA04168 To wash.dc.gov (sendmail)

Old versions of sendmail used an nf file when creating a message identifier to avoid race conditions.^[178] But contemporary versions of sendmail create the queue identifier when first creating the qf file. The nf file is obsolete.

A queued mail message is composed of two primary parts. The df file contains the message body. The qf file contains the message header.

In addition to the header, the qf file also contains all the information necessary to:

The qf file is line-oriented, with one item of information per line. Each line begins with a single uppercase character (the code letter), which specifies the contents of the line. Each code letter is then followed by the information appropriate to the letter. The code letters and their meanings are shown in Table 11-6 on page 446.

Here is an example of a version 8 (for V8.14 sendmail) qf file:

V8
T944703473
K0
N0
P1
I7/22/19133
Fwbs
$_you@localhost
${daemon_flags}c u
Syou@your.domain
Ayou@your.domain
rRFC822; george@wash.dc.gov
RPFD:george@wash.dc.gov
H?P?Return-Path: <you>
H??Received: (from you@localhost)
        by your.domain (8.14.1/8.14.1) id g38DcXCL026713
        for george@wash.dc.gov; Fri, 14 Dec 2007 17:37:53 −0800 (PST)
H?D?Date: Fri, 14 Dec 2007 17:37:53 −0800 (PST)
H?F?From: Your Name <you>
H?x?Full-Name: Your Name
H?M?Message-Id: <200704081338.g38DcXCL026713@your.domain>

This fictional qf file shows the information that will be used to send a mail message from you@your.domain (the S line) to one recipient: george@wash.dc.gov (the R line). It also shows the various headers that appear in that message (the H lines). We discuss the individual lines of the qf file at the end of this chapter.

When processing a queued message, it is often necessary for sendmail to modify the contents of the qf file. This usually occurs if delivery has failed or if delivery for only a part of the recipient list succeeded. In either event, at least the message priority needs to be incremented.

To prevent damage to the original qf file, sendmail makes changes to a temporary copy of that file. The temporary copy has the same queue identifier as the original, but its name begins with a t.

After the tf file has been successfully written and closed, sendmail calls rename(2) to replace the original with the copy. If the renaming fails, sendmail syslog(3)s at LOG_CRIT a message such as the following:

cannot rename(tfdB91brx04175, qfdB91brx04175), df=dfdB91brx04175

Failure to rename is an unusual but serious problem: a queued message has been processed, but its qf file contains old and incorrect information. This failure might, for example, indicate a hardware error, a corrupted queue directory, or that the system administrator accidentally removed the queue directory.

A given mail message can be destined for many recipients, requiring different delivery agents. During the process of delivery, error messages (such as “User unknown” and “Permission denied”) can be printed back to sendmail by each delivery agent.

While calling the necessary delivery agents, sendmail saves all the error messages it receives in a temporary file. The name of that temporary file begins with the letters xf. After all delivery agents have been called, sendmail returns any collected error messages to the sender and deletes the temporary xf file. If there are no errors, the empty xf file is silently deleted. A -d51.104 debugging switch setting can be used to prevent deletion of the xf file.

See Using qf, df, and xf Subdirectories on page 403 for a way to relocate xf files to a memory-based filesystem.

V8.10 sendmail offers the ability to distribute queued messages across multiple directories. In general, this is a good idea. If, for example, a high volume of email is stressing your current disk, you can improve efficiency by using multiple queue directories spread over multiple disks and controllers.

To illustrate, we will set up a machine that has three brand-new disks to use as multiple queue directories. The disks have already been formatted and a filesystem has been placed on each. We next create directories on which to mount them:

# mkdir /var/queues /var/queues/q.1 /var/queues/q.2 /var/queues/q.3
# chmod 700 /var/queues /var/queues/q.?

Because of the way multiple queue directories are implemented inside sendmail, the queue directory names must differ only in their suffixes, hence the trailing 1, 2, and 3. First the directories are created with mkdir(1) or a symbolic link, and then the permission on each is reduced to readable and writable only by root for security reasons. Note that these are the permissions after all queue disks are mounted.

Next, arrange for the disks to be mounted by placing the appropriate entries in /etc/fstab or /etc/vfstab. Here, we illustrate with the partial contents of /etc/fstab for Linux:

/dev/hda2         /var/queues/q.1     ext2     defaults    1 1
/dev/hdb1         /var/queues/q.2     ext2     defaults    1 1
/dev/hdc1         /var/queues/q.3     ext2     defaults    1 1

Note that we are mounting a separate disk on each queue directory. Your disk device names will doubtless differ, and you can use any directory locations and names you wish. Note that after you mount the disks, you might need to change the permissions again to 700 for each mount point.

The idea is to prepare the directories for use as multiple queue directories first, and after that, to modify the configuration file so that sendmail can use those queue directories:

define(`QUEUE_DIR',`/var/queues/q.*')

Here, the QUEUE_DIR mc configuration macro is given the value /var/queues/q.*, which will become the value for the QueueDirectory option. The trailing * character is a literal asterisk (not a wildcard character) and must appear as a suffix, in the last position of the path specification. It tells sendmail to use all the queue directories that begin with the path /var/queues/q. and end with any other characters. In our example, sendmail will match /var/queues/q.1, /var/queues/q.2, and /var/queues/q.3.

It is not strictly necessary to mount a disk on each queue directory. If the directory name is a symbolic link to another directory, sendmail will use that other directory as a queue directory. The only requirement is that the other directory has as restrictive set of permissions as the original queue has.

Mail queue is empty            ← when nothing is queued (pre-V8.10)
Mail queue (1 request)         ← when one message is queued (pre-V8.10)

/var/spool/mqueue is empty      ← when nothing is queued (V8.10 and later)
/var/spool/mqueue (1 request)   ← when one message is queued (V8.10 and later)

/var/spool/mqueues/q.1 is empty
                /var/spool/mqueues/q.2 (1 request)
----Q-ID---- --Size-- -----Q-Time----- ------------Sender/Recipient------------
dB9Fdaa06420     4567 Thu Dec  9 07:39 you@your.domain
                                       <gw@us.gov>
                Total Requests: 1

% mailq -OMaxQueueRunSize=1 | tail −1
                           Total Requests: 41291

Beginning with V8.10, sendmail allows the qf, df, and xf files to reside in separate directories. One advantage to this is that it produces directories that are one-third smaller. Another advantage is that each part can reside on a separate disk for further performance enhancements.

This feature is enabled by simply creating the appropriately named subdirectories, or symbolic links, in each queue directory. The names of those subdirectories or symbolic links are the literals qf, df, and xf. But be aware that you should not create those directories or links when mail is already queued. If you do, that queued mail will disappear from sendmail’s view and will never be delivered. If you need to make the change while mail is queued, first stop sendmail, and then execute the following commands and restart sendmail:

# mkdir df qf xf
# chmod 700 df qf xf
# mv df?* df/           ← if mail is already queued
# mv qf?* qf/           ← if mail is already queued
# mv xf?* xf/           ← if mail is already queued

Here, we first create the new subdirectories in the queue directory. Then we reduce their permissions to the narrow ones that match the queue directory. Finally, if queued mail already existed in the queue directory, we move that mail into the new subdirectories where sendmail will find it.

Because xf files are empty for all successfully delivered mail, there is a penalty for creating and deleting those files just because they might be needed. When performance is of concern, you can either mount a memory filesystem on the xf subdirectory, or replace the xf subdirectory with a symbolic link to a directory on a memory filesystem. In the following, we show an /etc/fstab file for a SunOS machine that uses the direct-mount approach:

/dev/sd0g  /var/spool/mqueue/df        4.2 rw        1 4
/dev/sd2g  /var/spool/mqueue/qf        4.2 rw        1 2
swap       /var/spool/mqueue/xf        tmp rw        0 0

Shortly. we will describe how to use a different type of disk for each part, and how performance is impacted by such choices.

An artifact of using qf, df, and xf subdirectories is seen when printing the queue. The df directory is always the one listed:

/var/spool/mqueue/df is empty

To understand the potential problems associated with deep queues, first consider how sendmail processes a single queue when its QueueSortOrder option (QueueSortOrder on page 1073) is set to the default of priority.^[179] When sendmail is instructed to process a queue it opens the queue directory for reading and reads that directory to gather a list of qf files to process. Each qf file sendmail finds is opened for reading and scanned for important pieces of information. The N line in each qf file, for example, holds the number of times the message has been tried. The P line holds each message’s current priority.

After all messages have been opened, read, and closed, and after the information from each has been saved internally, sendmail sorts that information. The purpose of the sort is to ensure that new mail is tried before old, and that high-priority mail is tried before low-priority mail.

Under normal circumstances, this process occurs quickly. But when queues get abnormally deep, things can go wrong. In the following, which illustrates a problem that can occur, we show one way that sendmail could be run on a major mail-sending machine:

/usr/sbin/sendmail -bd
/usr/sbin/sendmail -q10m

The idea here is to create two mail-handling daemons. The first handles inbound mail, and because this is a mail-sending machine, we expect that this inbound daemon will perform little work. The second daemon sends all mail it finds in its queue. It will fork(2) a copy of itself once every 10 minutes, and that copy will process all the messages in the queue. As described earlier, each queued message is opened and read so that all the messages can be sorted before delivery begins.

Because this hypothetical site is a major mail-sending site, we expect a high rate for the number of sent messages. For the sake of argument, let’s say 30,000 messages need to be sent per hour.

Now suppose a backhoe, a power failure, clumsy fingers, or any of a thousand possible disasters causes this site’s only connection to the Internet to fail for an hour, and the site can neither look up host information with DNS, nor connect to any remote sites. All the mail it tries to send that hour fails, and instead of being removed from the queue, this failed mail is left there to be tried again later (presumably after the problem is fixed).

An hour later, service is restored. First, the default:

/usr/sbin/sendmail -q10m

causes a forked copy of sendmail to start processing the queue. This time, however, the processing is not swift. When a queue fills to 30,000 or more messages, the amount of time it takes to preread the queue (to open and read every message) increases to more than 20 minutes.^[180] And those 20 minutes are only for the preread. During those 20 minutes no mail will be sent.

After that, things get worse. Ten minutes later a second sendmail daemon is forked, and it, too, starts to preread the queue. Now, instead of one sendmail daemon opening and reading all messages in a queue, we have two sendmail daemons doing the same thing in parallel.

Contrary to what you might think, twice as much I/O on a disk is not twice as fast. Disks are finite devices that perform a limited number of disk-head moves^[181] per second and can transmit only a fixed number of bytes per second. Because the two sendmail daemons are 10 minutes out of step with each other, each is reading and processing separate files. Depending on the size of your in-memory disk cache, neither will likely be able to take advantage of the efficiencies of such caching. In short, two sendmail daemons processing a deep queue in parallel is worse than a single sendmail daemon processing that same queue alone.

And if that weren’t enough, another 10 minutes later a third sendmail daemon starts to process the queue.

By now, the first sendmail daemon might have finished its preread of the queue and might have actually begun to send messages. But even if it has, three sendmail daemons are now processing that single deep queue and a curious thing happens. Because the disk that holds the queue is finite, the addition of a third sendmail daemon slows the operation of the first two. The second one, instead of taking 20 minutes to preread the queue, will now take 30 minutes.

This means that every 10 minutes another sendmail queue-processing daemon is added to the mix. As each is added, each slows all the others that are already running, and it isn’t long before the load on the machine starts to climb and the rate at which messages are delivered falls at an alarming rate. In fact, when this sort of behavior hits a very large-volume site, a sendmail queue-processing daemon can start and seem to never finish.

Depending on the speed of your disk system, even limiting the number of queue processors per queue might not save you from this sluggish performance. Under V8.12 sendmail, for example, you can limit the number of queue runners per queue with a queue group (Queue Groups (V8.12 and Later) on page 408) definition such as this in your mc configuration file:

QUEUE_GROUP(`fastq', `P=/q/fastq*, I=10m, R=10')

Here, the fastq group uses the queue disks mounted as /q/fastq*, processes those disks once per 10 minutes (the I=10m), and limits itself to 10 queue runners maximum (the R=10) across all the disks. If there are few fastq* queue disks, and if they fill to more than 30,000 messages each, they too can become sluggish, even with only 10 runners processing them. In fact, with sufficient filled queue depth, as few as two simultaneous queue runners can seriously affect performance.

In extreme situations such as this, one alternative is to use persistent queue runners (Persistent Queue Runners with -qp on page 434). With persistent queue runners, you maintain a single queue runner that alone reads the queue. After that single queue runner has read the queue, it forks multiple child queue runners to process the queue, with each child sharing the parent’s queue information:

/usr/sbin/sendmail -qp10m

Here, the -qp causes one or more persistent queue runners to launch. One is launched for each queue group, and will persist to run, sleeping 10 minutes between each reading of the queue. When it awakes, it gathers a list of queue files and launches multiple child processes to handle that list. After the last child has finished delivery and exited, the parent sleeps again.

Even with queue groups and persistent queue runners, you are encouraged to spread queues across many directories and across many disks and controllers. This increases parallelism and dramatically lessens the likelihood that any given queue will overfill.

When a queue directory is exceptionally full, you will likely notice the problem only when performance on your queue-handling machine becomes unusually sluggish. By that time, however, a drastic measure, such as rebooting the server, might be the only cure. Clearly, early detection is desirable.

Early signs that a queue is filling can be seen in the logging messages that sendmail produces. You can develop scripts that watch for lines such as these:

Dec 13 10:27:53 your.domain sendmail[642]: grew WorkList for /var/spool/mqueue to 2000
Dec 13 10:29:05 your.domain sendmail[642]: grew WorkList for /var/spool/mqueue to 3000
Dec 13 10:34:31 your.domain sendmail[642]: grew WorkList for /var/spool/mqueue to 4000
... etc., to:
Dec 13 12:40:22 your.domain sendmail[642]: grew WorkList for /var/spool/mqueue to
29000
Dec 13 12:42:50 your.domain sendmail[642]: grew WorkList for /var/spool/mqueue to
30000

Here, the WorkList refers to the number of messages preread so far. By searching for unusual sizes, you can determine when a queue is about to overfill.

Another technique is to run the mailq command to observe the total number of messages queued across all queues:

% mailq -OMaxQueueRunSize=0 | tail −1        ← V8.7 through V8.11
                           Total Requests: 34190

% mailq -bP                                  ← V8.12 and later
/var/spool/mqueues/q.1/df: entries=34190
                Total requests: 34190

For V8.7 through V8.11, the MaxQueueRunSize=0 allows mailq to run swiftly, regardless of how deep the queue or queues might be. Without that option, and with deep queues, mailq would be just as slow as the sluggish queue runs, but beginning with V8.12, the -bP command-line switch does the same thing more quickly.

No matter how you detect the problem, the solution will be the same. First, you need to kill all the competing sendmail queue-processing daemons. There are a wide number of ways to do this. The most common is to use ps(1) to gather PID numbers and then kill each queue-processing daemon individually. No matter how you kill the queue-processing daemons, be sure to kill them all. If you don’t, you might find the problem surfacing again before you have had a chance to fix it.

The best way to flush a full queue is with a command line something like this:

# /usr/sbin/sendmail -OQueueSortOrder=filename -q10m -d99.100
# /usr/sbin/sendmail -OQueueSortOrder=random   -q10m -d99.100    ← V8.12 and later
# /usr/sbin/sendmail -OQueueSortOrder=none     -q10m -d99.100    ← V8.13 and later

Here, the -d99.100 tells sendmail to run in the foreground (so that you can kill it easily when done). The -q10m causes a queue-processing daemon to be launched once each 10 minutes (just like before). You need this because one daemon can seem to hang when delivering mail to a slow host. By running parallel daemons, you avoid this pitfall.

Sorting by filename or random (How the Queue Is Processed on page 426) or none (V8.13 and later) causes sendmail to skip the opening and reading of each queued message. Instead, it only looks at the filename for its sorting or randomizing order. On the downside, this prevents sendmail from grouping messages for optimum delivery. On the upside, this reduces the time to preread a huge queue from 20 or so minutes to less than 2 seconds.^[182]

The QueueSortOrder=random (QueueSortOrder=random (V8.12 and later) on page 1074) is just like the QueueSortOrder=filename shown earlier, except that it randomizes the list before beginning delivery. This method is preferred, but is only available beginning with V8.12 sendmail.

After draining the full queue to a more manageable level, you can discontinue this special process and rerun sendmail in its normal manner.

If the full queue has to remain in service while the full state is being solved, you can use the techniques in Handling a Down Site on page 437 to move that full queue out of the way so that it can be processed in the background.

Prior to V8.12 sendmail, there were no queue groups. Instead, every -q command and every queue option (such as QueueDirectory) applied to all the queue directories you had.^[183]

Beginning with V8.12, sendmail offers a way to define multiple queue directories and a way to group them by function or specialty. For compatibility with old versions, a special queue group named mqueue is the default queue group. It takes on all the properties of every -q command, and every queue option, just like before.

When you later declare particular queue groups (as we show in the next section), those additional groups take all their properties from the default group, unless you override a particular property with a specific equate. Those equates and the command-line arguments or options they override are shown in Table 11-2 on page 410.

For example, the following declares two different queue directories:

define(`QUEUE_DIR', `/var/spool/mqueue')
QUEUE_GROUP(`regularmail', `')
QUEUE_GROUP(`slowmail', `P=/var/spool/mqueue/slowqueue')

The first line declares the queue used by the default group (always known as mqueue). Any other queue groups that are declared (such as regularmail) will use that same directory unless the directory is overridden by the P= equate, as shown in the third line. That is, the default queue group’s queue directory and everything else that is set for the default queue group is inherited by the regularmail group. For the slowmail queue group, however, everything but the queue directory is inherited. (See The Path= (P=) queue-group equate on page 413 for a description of the P= equate, and for the reason queue group directories must be subdirectories under QUEUE_DIR.)

Queue groups are declared with the Q configuration command. That command can take a wide range of appearances, but in all guises it takes the name of the queue group and then a sequence of equates:

Qgroupname, equates

The name of the queue group (here groupname) must follow the Q with no intervening spaces. If spaces are present, an error such as the following is printed and logged, and that Q line is ignored:

file.cf: line line number: queue : `=' expected

The equates are optional, but if they are present they must follow the queue group’s name and a comma or whitespace, or both:

Qgroupname, equates

The equates are formed by selecting one of the keywords shown in the leftmost column of Table 11-2, and following it with an equals sign and the value you wish to assign to that key letter. Note that only the first letter is looked at by sendmail, so you can use the shorthand shown in parentheses if you wish. Also note that the first letter is case-sensitive—that is, R and r are different.

For example, both of the following declare a queue directory (the Path= and P=) and a queue-processing interval of 10 minutes (the Interval= and I=):

Qslowmail, Path=/disk1/mail/slowqueues, Interval=10m
Qslowmail, P=/disk1/mail/slowqueues, I=10m

A comma separates one equate from another. The comma can be optionally surrounded by whitespace characters (spaces and tabs). If the value following the comma is missing, an appropriate error will be printed and logged.

If an equate other than those shown in the table is used, an error such as the following is printed and logged, and that Q line is ignored:

file.cf: line line number: Qgroupname: unknown queue equate bad equate here

Warning: Q=queuegroup name: R=number: multiple queue runners specified
       but flag 'f' is not set

I=interval

Interval=1h12

Jobs=number

Nice=10       ← increase niceness by 10, lower priority
Nice=0        ← no change
Nice=−10      ← same as zero
Nice=b        ← same as zero

define(`QUEUE_DIR',`/var/spool/mqueues/q.*')            ← the default
QUEUE_GROUP(`aolmail', `P=/var/spool/mqueues/aolmail')  ← good, a subdirectory
QUEUE_GROUP(`bobmail', `P=/var/spool/mqueues/bob.*')    ← good, a subdirectory
QUEUE_GROUP(`hotmail', `P=hotmail')                     ← bad, not a full path
QUEUE_GROUP(`slow', `P=/var/spool/slowqueue')           ← bad, not a subdirectory

QueuePath hotmail not absolute

QueuePath /var/spool/slowqueue not subpath of QueueDirectory /var/spool/mqueues:
No such file or directory

define(`QUEUE_DIR',`/var/spool/mqueues/q.*')            ← the default
QUEUE_GROUP(`slow', `P=/var/spool/mqueues/slowqueue')
                                             ↑
                                a symbolic link to /var/spool/slowqueue

recipients=99        ← set the limit to 99 recipients
recipients=0         ← set unlimited recipients
recipients=-99       ← same as r=0
recipients=none      ← same as r=0

Runners=12     ← 12 per queue run
Runners=0      ← no limit, so one per queue each queue run
Runners=none   ← the same as R=0

Q=queuegroup: R=number exceeds MaxQueueChildren=limit, set to MaxQueueChildren

You declare queue groups inside your mc configuration file with the QUEUE_GROUP mc configuration macro. As you saw in the previous sections, it is used like this:

QUEUE_GROUP(`group name', `equates')

The queue group name can contain any characters except a comma or a whitespace character (a space or a tab).^[184] It must not be surrounded (inside the quotes) with whitespace characters.

The equates form the second argument to the QUEUE_GROUP mc configuration macro. The equates are described in The Q Configuration Command on page 409.

To illustrate, consider the following QUEUE_GROUP mc configuration macro declaration:

QUEUE_GROUP(`slowmail', `P=/var/spool/mqueues/slowqueue')

Here, the name of the queue group is set to slowmail. The second argument is a single equate, the P= queue-group equate, which defines the queue directory or directories to be used by this queue group.

If you want to define which queue group to use for certain delivery agents, you can use the Q= delivery-agent equate (Q= on page 750) as set, for example, with the LOCAL_MAILER_QGRP mc macro. For example, the following tells sendmail to queue all local mail in the /queues/lq queue directory:

QUEUE_DIR(`/queue')
QUEUE_GROUP(`localgroup', `P=/queue/lq')
define(`LOCAL_MAILER_QGRP', `localgroup')    ← must be before MAILER(local)
MAILER(`local')

In the first line we set the default queue directory. In the second line we define the queue group localgroup, and set its queue directory to be /queue/lq. In the third line we declare that the Q= equate for the local delivery agent will be:

Q=localgroup

The fourth line declares support for the local delivery agent. Note that the definition of LOCAL_MAILER_QGRP must precede the MAILER(local); otherwise, that definition will be silently ignored.

Those four lines cause all mail for local users to be queued in the /queue/lq directory. Note that you can dedicate queue groups for other delivery agents. See Q= on page 750 for a full description of this process.

The easiest way to select queue groups based on recipient addresses or recipient domains is by using the FEATURE(queuegroup). It is declared in your mc configuration file like this:

FEATURE(`queuegroup')
FEATURE(`queuegroup', `default group')

The first line causes the queue group to default to mqueue if a queue group in the access database is missing or nonexistent. The second line allows you to set a different default queue group. For example, consider the following lines from an mc file:

QUEUEGROUP(`localgroup', `/queue/lq')
FEATURE(`queuegroup', `localgroup')

This causes sendmail to use the group named localgroup instead of mqueue as the default if a queue group in the access database is missing or nonexistent.

Once you have enabled the FEATURE(queuegroup), the next step is to add lines such as the following to the source file for your access database:

QGRP:slow-poke.com      slowgroup
QGRP:root@notify.com    fastgroup
QGRP:your.domain        localgroup

Each line that selects queue groups must begin with the literal expression:

QGRP:

This prefix tells sendmail that you wish to map recipient addresses or domains to queue groups.

The first line causes mail to the slow-poke.com domain to use the queue group called slowgroup. This shows that you can list just a domain in the lefthand column and it will work just as expected.

The second line causes mail to the specific recipient root@notify.com to use the queue group named fastgroup. This line demonstrates that mail to an individual can be used in the lefthand column.

The third line illustrates your local domain, which shows that mail to your domain, your.domain, will use the queue group named localgroup.

If you omit the name of the queue group (not recommended), you will need to use the -e command-line switch with makemap to create the database. When you omit the name of the queue group the default queue group is used:

QGRP:another.your.domain
                           ↑
                 queue group name missing (not recommended)

Here, if you defined a default queue group when you declared the FEATURE(queuegroup), that group will be selected. Otherwise, the group mqueue will be selected for this domain.

Normally, the access database, described earlier, is the easiest way to select queue groups. There might be times, however, when selecting by recipient address or domain is not sufficient. Should such a situation arise, you could set up your own rule sets. But be forewarned that if you do, the FEATURE(queuegroup) cannot be used. If you try to use both, you will get the following warning every time sendmail starts to run:

WARNING: Ruleset queuegroup has multiple definitions

The first step in declaring your own rules to select queue groups is to declare a special rule set called queuegroup. You do that in your mc configuration file using the LOCAL_RULESETS macro:

LOCAL_RULESETS
Squeuegroup
               ← your rules here

The way this rule set works is simple. Any queue group for a recipient address that a rule selects is returned following the $# operator. For example, consider the following:

R $*                                          $: $>canonify $1
R $* <@some.domain>          $# somegroup

Here, mail bound for any user at some.domain will be queued in the somegroup queue group.

Normally, queuegroup rule sets are used to select queue groups based on the recipient. If you wish to select based on the sender, you can do so using rules something like the following:

LOCAL_RULESETS
Squeuegroup
R $*                     $: $>canonify $&f
R $+ <@ lists.domain.>   $# lists

First, we fetch the sender address using $&f, and pass it through the canonify rule set 3 to focus on the host part. The second rule matches any user at the domain lists.domain, and selects the lists queue group.

Because there are no more rules following the second one, this rule set returns without selecting a queue group. If the queuegroup rule set fails to select a queue group, the default queue group (mqueue) is used.

Other possible uses for the queuegroup rule set might include:

Note that there are limitations on the use of this queuegroup rule set. First, this rule set is called directly from inside sendmail, so you should not call it from inside your own rules (if you do, the selected queue group will be ignored). And second, the FEATURE(queuegroup) also uses this rule set, so you cannot share it with that feature.^[185]

As you saw in The Default Queue Group on page 409, the default queue group (mqueue) is defined by options and the command line. If any given Q configuration command is missing a given equate, that queue group inherits that property as defined by the default queue group. There are, however, properties for the default queue group which have no equivalent equates. These properties are inherited by all queue groups and cannot be overridden with a queue-group equate. They are:

V8.6 sendmail always checks the form of the qf file name for correctness. V8.7 through V8.9 sendmail also check the qf filename, but do so only if PICKY_QF_NAME_CHECK is defined when building sendmail (PICKY_HELO_CHECK on page 133). V8.10 and later no longer check the form of the qf filename for correctness.

Prior to V8.10, if the qf filename is incorrectly formed (The Queue Identifier on page 396), sendmail presumes that some other program placed the file in the queue and rejects it:

orderq: bogus qf name bogus name here

For V8.7 through V8.9, sendmail made this check only if PICKY_QF_NAME_CHECK was defined when building sendmail. This was introduced because some sites allow legitimate programs (other than sendmail) to write into sendmail’s queue. To fix this problem, either undefine PICKY_QF_NAME_CHECK when you build sendmail (if your site allows other programs to write into the queue directory), or trace down the process that is placing badly formed qf names in your queue and fix it.

Each qf file must be owned by the effective user ID under which sendmail runs (usually root). A qf file must not be group- or world-writable. If a qf file fails either test, it is considered bogus and is renamed to a Qf file. Then sendmail logs these messages:

id: bogus queue file, uid=owner, mode=perms
Losing qffile: bogus file uid in mqueue

Here, id is the identifier portion of the qf filename, owner is the uid of the user that owns the qf file, and perms are the file permissions of the qf file, printed in octal.

This problem might point to bad queue directory permissions that allow anyone (or some group) to place files there. Or it might indicate that some process other than sendmail is writing to your queue.

One form of attack against sendmail is to append additional control lines to the end of an existing qf file. V8.7 sendmail specifically checks for additional text and rejects the qf file if any is found:

SECURITY ALERT: extra data in qf: first bogus line printed here
Losing qffile: bogus queue line

V8.7 sendmail terminates its legitimate list of qf control lines by placing a dot on a line by itself. Any text following that line, including comments and blank lines, is considered an error. This can represent a serious attack against your machine or site. If you get this message, investigate at once.

Each line in a qf file must begin with a known control letter or character (The qf File Internals on page 445). If a line begins with any other character, it is considered bad, and the whole file is rejected:

readqf: qffile: line num: bad line bogus line here
Losing qffile: unrecognized line

Note that this error is to be anticipated if you go backward, from a later release to an earlier release of sendmail.

An F line in a qf file is used to save and restore envelope flag bits. Unfortunately, the first line of a Unix-style mailbox also begins with an F:

From someone@site

If a qf file’s F line begins with the five characters "From“, V8.7 and later sendmail will reject the file and log a possible attack:

SECURITY ALERT: bogus qf line bogus line here
Losing qffile: bogus queue line

This might represent a serious attack against your machine or site. If you get this message, investigate at once.

In the rare event that sendmail cannot dispose of a bounced message, it will preserve the qf file as a Qf file and log the message:

savemail: cannot save rejected email anywhere
Losing qffile: savemail panic

The sendmail program tries everything possible to avoid this state (including bouncing the message, sending it to the postmaster, and saving it to a dead.letter file). Only if all else fails will it preserve the qf file as a Qf file.

In general, this points to an alias problem with the user named postmaster or the owner of a mailing list. Such users are special. They must be able to receive email messages no matter what. They should be the names of real people, not the names of further mailing lists.

Beginning with V8.13, the -qL command-line switch allows you to view and handle Qf files. Note, however, that handling these files, without first repairing the causative problem, can be risky. One use for this new switch is to examine the mail queue to see if any lost files exist:

% mailq -qL
                /var/spool/mqueue (1 request)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
h7AJG4kr009003?     235 Sun Aug 10 13:16 <you@your.domain>
                                         <bob@other.domain>
                Total requests: 1

Here, the -qL command-line switch was used with the mailq command to see if any lost files were present. This output shows that a lost file (called Qfh7AJG4kr009003) is located in the /var/spool/mqueue directory. The "?" character following the file’s name indicates that it is a lost envelope.

This -qL switch can be combined with other queue-handling switches to further limit what can be shown.

The -v command-line switch can be used in combination with the -bp switch to cause sendmail to also print additional details about the queued messages. To begin, the usual heading shows a new item:

----Q-ID---- --Size-- -Priority- -----Q-Time----- ---------Sender/Recipient-----
----
dB928Xl04182     354     54320 Fri Mar 15 08:32 your@your.domain
                                  george@wash.dc.gov
dB928RR04181*   1972     39020 Fri Mar 15 08:45 your@your.domain
      8BITMIME   (Timed out waiting to connect to wash.dc.gov)
                                  jefferson@wash.dc.gov
dB928RR04192-     23     30001+Fri Mar 15 09:32 your@your.domain
                 (Timed out waiting to connect to wash.dc.gov)
                                  jefferson@wash.dc.gov
                               (---you---)
                                   bob

The Priority is the value from the P line (P line on page 453) in the qf file. Printing the queue does not change a message’s priority, whereas processing the queue does. See the RecipientFactor option (RecipientFactor on page 1077) for a description of how the priority is calculated.

Verbose mode also causes a + to print after the Priority (as in the third item in the example) if a warning message has been sent. See the Timeout.queuewarn option (Timeout on page 1097) for a description of how messages time out.

If any R line is preceded by a controlling user (the C line in the qf file, C line on page 447), verbose mode causes that controlling user’s name to be put in parentheses and prepended to the recipient name. The third item in the preceding example illustrates this.

Prior to V8.8 sendmail, the M line error messages were truncated to 60 characters. Beginning with V8.8, verbose mode causes the full, nontruncated text of the M line error to be printed.

Beginning with V8.12 sendmail, the -bP command-line switch can be used to print the number of messages in the queue or queues. This command-line switch relies on shared memory to gather its information, so it works only if sendmail is compiled with shared memory support. The SM_CONF_SHM compile-time macro determines whether shared memory support was included (see SM_CONF_SHM on page 142). If shared memory support is not included, use of this command-line switch will cause the following error to be printed:

Data unavailable without shared memory support

If shared memory support is compiled in, but there is a problem with it (possibly at the system level), the following error will print:

Data unavailable: shared memory not updated

Note that you will also get this error if the queue has not been processed at least once to initialize the data.

In addition to enabling shared memory using the SM_CONF_SHM m4 Build macro, you must also define a key to be used with that shared memory with the SharedMemoryKey option. To set this option in your configuration file, you could add a line such as the following to your mc configuration file:

define(`confSHARED_MEMORY_KEY',`13521')

If all goes well, the -bP command-line switch will produce output such as this:

/var/spool/mqueues/q.1/df: entries=34
/var/spool/mqueues/q.2/df: entries=51
               Total requests: 85

Here, 85 is the number of envelopes currently awaiting delivery in sendmail’s queues. But note that some shared memory timeouts can lead to an inexact count. In this latter event, the output looks like this:

Total requests: 85 (about)

If you lack shared memory support, and you are running pre-V8.12 sendmail, you can still summarize the number of messages in all queues with a simple substitute command:

% mailq -OMaxQueueRunSize=0

A single queued message has a single sender but can have many recipients. When processing a queued message, sendmail attempts to deliver it to all recipients before processing the next queued message.

The first step in processing a queued message is to lock it so that concurrent runs of sendmail do not attempt to process it simultaneously (Current-style file locking on page 398). Then the qf file is opened and read. The sender and all the recipients are gathered from the corresponding S and R lines.

For each recipient, delivery is attempted. If delivery is successful, that recipient’s address is removed from the sendmail program’s internal list of recipient addresses. If delivery fails, that address is either left in the list or bounced, depending on the nature of the error.

After all recipients have been either delivered, bounced, or left in the list, sendmail reexamines that list. If there are no recipients left in it, the message is dequeued (all the files in the queue directory that compose it are removed). If any recipients are left, each recipient results in an M line that is assigned the last error message for that recipient, and the qf file is rewritten with the list of the remaining recipients and a dot. Finally, the qf file is closed, thus freeing its lock.

Under V8 sendmail, the CheckpointInterval option (CheckpointInterval on page 983) causes checkpointing of this process. When this option has a positive value, the qf file is rewritten after that value’s number of recipients have been processed. For example, consider a mail message to five recipients. If the CheckpointInterval option is set to a value of 2, the qf file is rewritten after the first two recipients have been processed, then again after four, and again after they all have been processed. This keeps the qf file reasonably up-to-date as protection against sendmail being improperly killed or the machine crashing.

The -q command-line switch is used both to cause queues to be processed and to specify the interval between queue runs.

A typical invocation of the sendmail daemon looks like this:

/usr/sbin/sendmail -bd -q1h

Here, the sendmail program is placed into listening mode with the -bd command-line switch. The -q1h command-line switch tells it to process the queue once each hour. Note that either switch puts sendmail into the background as a daemon. The -bd switch just allows sendmail to listen for incoming SMTP connections. Consider the following:

/usr/sbin/sendmail -bd
/usr/sbin/sendmail -q1h

This runs two daemons simultaneously. The first listens for incoming SMTP connections. The second processes the queues once per hour.

The time expression following the -q is constructed from an integer followed by a letter. The letters and the meaning of each are listed in Table 11-5. Integer and letter groups can be combined—for example, 5d12h means 5 days, 12 hours. If a letter is missing, the default is minutes.

At small sites, where mail messages are rarely queued, the time interval chosen can be small to ensure that all mail is delivered promptly. An interval of 15m (15 minutes) might be appropriate.

At many sites, an interval of one hour is probably best. It is short enough to ensure that delays in delivery remain tolerable, yet long enough to ensure that queue processing does not overlap (see Persistent Queue Runners with -qp on page 434 for a way to run a persistent queue runner that avoids overlapping runs).

At large sites with huge amounts of mail and at sites that send a great deal of international mail, the interval has to be carefully tuned by observing how long it takes sendmail to process its queues and what causes that process to take a long time. Points to consider are the following:

The -q command-line switch, invoked without a time interval argument, is used to run sendmail in queue-processing mode. In this mode, sendmail processes queues once and then exits. This mode can be run interactively from the command line or in the background via cron(8).

Other command-line switches can be combined with -q to refine the way queues are processed. The -v (verbose) switch causes sendmail to print information about each message it is processing, and to process multiple queues sequentially. The -d (debugging) switch can be used to produce additional information about the queue. We’ll discuss the -v switch as it applies to the queue later in this chapter. Those -d debugging switches appropriate to the queue can be found in Table 15-3 on page 536.

V8 sendmail allows variations on -q: -qI allows you to specify a specific message identifier for processing; -qR allows you to specify specific recipient addresses for processing; and -qS allows you to specify specific sender addresses for processing.^[189]

# /usr/sbin/sendmail -q

5 * * * * /usr/sbin/sendmail -q >/dev/null 2>&1

% /usr/sbin/sendmail -v -q

Running /var/spool/mqueue/dB9JBR106687 (sequence 1 of 2)
<adams@dc.gov>... Connecting to dc.gov via ddn...
Trying 123.45.67.8... Connection timed out during user open with DC.GOV
<adams@dc.gov>... Deferred: Host DC.GOV is down

Running /var/spool/mqueue/dB9JDWt06701 (sequence 2 of 2)
<help@irs.dc.gov>... Connecting to irs.dc.gov via ddn...
Trying 123.45.67.88...  connected.
220 irs.dc.gov Sendmail 5.57/3.0 ready at Mon, 27 Jan 92 09:16:38 −0400

-qIident          ← match any queue ID that contains ident
-qRrecip          ← match any recipient address that contains recip
-qSfrom           ← match any sender address that contains from

% /usr/sbin/sendmail -qSroot -qRbiff@here

-qSroot

root
ben@groots.edu
ben@GROOTS.EDU

% /usr/sbin/sendmail -qI123 -qSroot -qR@host.edu

% mailq -qI123 -qSroot -qR@host.edu

% mailq -q\!Sroot -qR@host.edu

-qIident          ← match any queue ID that contains ident
-q!Iident         ← match any queue ID that does not contain ident
-qRrecip          ← match any recipient address that contains recip
-q!Rrecip         ← match any recipient address that does not contain recip
-qSfrom           ← match any sender address that contains from
-q!Sfrom          ← match any sender address that does not contain from

-qGgroupname

Queue group groupname unknown

% /usr/sbin/sendmail -qR@hostA.domain -qGslow

Cannot use multiple -qG options

% /usr/sbin/sendmail -q!Gmqueue
Cannot use -q!G

ETRN host

550 Parameter required

-qR@host

ETRN #groupname

459 4.5.4 Queue badname unknown

contrib/etrn.pl

% contrib/etrn.pl  your.mx.server

% nroff -man contrib/etrn.pl | more

V8.12 sendmail introduced persistent queue runners as a solution to some of the problems caused by periodic queue runners. Periodic queue runners are the result of a normal -qinterval command-line switch, or a Runners= queue-group equate. Either causes:

Persistent queue runners avoid these problems because a single process is dedicated to a queue, a queue group, or a grouping of queue groups (called a “workgroup”). A persistent queue runner is launched just like the periodic command-line queue runner, but with the addition of a p character:

-qpinterval

The p causes one or more persistent queue runners to be launched, one per queue group. One will be launched to handle your default queue group, and one more will be launched to handle each queue group defined by a QUEUE_GROUP mc option. Depending on the number of queue directories in each, these can be combined into a single workgroup. When you have many queue groups, you can end up with multiple workgroups controlling persistent runners.

Each persistent queue runner will sleep for interval. When it awakes, it reads all the files in the queues that belong to its workgroup, and sorts all the envelopes it finds into the proper order needed for delivery. After it has finished ordering the envelopes, it launches one or more regular queue runners to perform delivery using that already processed list. This can significantly reduce the disk I/O compared to that needed by periodic queue runners.

When the last of the regular queue runners has finished processing (and exited), the persistent queue runner goes back to sleep for interval.

In general, persistent queue runners are valuable only at sites that normally have queues that are very full. When a queue is normally near empty, persistent queue runners can introduce unforeseen delays. Note that a persistent queue runner will sleep again only when all of its regular queue runners have finished. One regular queue runner, delivering to a very slow site, can appear to hang, and so can cause the persistent queue runner to also appear to hang. Subsequent queue runs will be delayed until the hung site times out, allowing the persistent queue runner to sleep interval again.

At large sites, such delays will eventually smooth out due to the normal distribution of slow jobs. At small sites, such delays might be noticed and objected to. In general, persistent queue runners should be reserved for sites with full queues.

If interval is omitted, the default interval becomes 1 second:

-qp

When the default interval is used (by omitting the interval), the persistent queue runner will sleep one second between queue runs, unless the prior queue run was empty, in which instance it will sleep five seconds. If you choose the default interval, we recommend you also set the MinQueueAge option (MinQueueAge on page 1057).

If interval is specified as zero, the effect is the same as though it were omitted. If interval is negative, the following error is logged and printed and sendmail exits:

Invalid -q value

If interval is nonnumeric (if you specify O when you mean zero), the following error is logged and printed, and sendmail exits:

Invalid time unit `O'

The process that was given the -qp command-line switch is the controlling process. It could be the listening daemon (if -bd or -bD were also used), or it could be a queue processing daemon (if only -qp and other queue processing limiters were specified). The controlling process has two special properties:

If a persistent queue runner core-dumps, the following will be logged and that queue runner will not be restarted:

persistent queue runner=number core dumped, signal=signal

If a persistent queue runner exits because of a caught signal, the following is logged and that queue runner is restarted:

persistent queue runner=number died, signal=signal

If a persistent queue runner is restarted because of a SIGHUP, the following is logged:

restart queue runner=number due to signal signal

If the -dno_persistent_restart debugging command-line switch is specified, a failed persistent queue runner will not be restarted, and the following error will be logged:

persistent queue runner=number, exited

A persistent queue runner will not be restarted if it has already been restarted 10 times. Instead, the following error will be logged and that persistent queue runner marked as bad:

ERROR: persistent queue runner=number restarted too many times, queue runner lost

If this happens, examine your logs. Some nonmail-related process might be signaling your persistent queue runners, or you might have bad memory, or you might have made a mistake when building sendmail and should rebuild it, or you might have a junior system administrator who does not know how to correctly restart sendmail.

Persistent queue runners look like executing periodic queue runners in process listings:

root   22958   476  ?  S   08:43   0:00 sendmail: accepting connections
root   22947   512  ?  S   08:32   0:00 sendmail: running queue: /var/spool/mqueues/
q.1/df

Here, the first line shows the controlling process, and the second line shows a persistent queue runner. Note that even though the second entry says “running,” it might not be.

If a site is down, messages to that site can collect in the queue. If the site is expected to be down for a protracted period of time, those queued messages will begin to time out and bounce. To prevent them from bouncing, you can move them to a separate queue directory. Later, when the down site comes back up, you can process that separate queue.

There are two ways to move mail to a holding queue. One way is to simply move them to a different directory, but you cannot do that if you are using queue groups. The other way is to use queue groups, as we show later.

# mkdir /var/spool/newqueue
# chmod 700 /var/spool/newqueue

# contrib/qtool.pl /var/spool/newqueue /var/spool/mqueues/q.1

% /usr/sbin/sendmail -OQueueDirectory=/var/spool/newqueue -OTimeout.queuereturn=51d
-OTimeout.queuewarn=0 -q

A quarantined message is an envelope, containing one or more recipients, that is held in the queue pending review. It can either be an inbound or outbound envelope that, for policy or security reasons, should not be sent or delivered immediately, or not be sent or delivered as is.

For example, consider a user who has a history of sending offensive email. You might want to intercept such a user’s email on its way out, so it can be screened for words or phrases that the user has been previously warned about.

V8.13 sendmail implemented quarantining by creating a new kind of queued file. Instead of storing the envelope information in a qf file, a quarantined message has its envelope information stored in an hf file. The different file allows sendmail to process messages normally (quarantined messages are invisible) unless you specifically ask it to handle quarantined messages (make them visible).

Note that the mailstats program (The mailstats Program on page 364) automatically (without you needing to ask) includes the total count of quarantined messages in its output.

To ensure that the reason for quarantining a message is not lost, a new qf file^[191] line has been introduced. Called a q line (q line on page 453), it stores the reason the message was quarantined. In parallel, a new macro, called ${quarantine} (${quarantine} on page 841) has also been added. It is intended for use in rule sets, and contains the reason the envelope was quarantined.

Note that quarantining integrates well with all the other queuing facilities of sendmail and even works with envelope splitting.

The command line can be used to quarantine and dequarantine envelopes. V8.13 has added one new command-line switch and modified another. We will show the use of the modified switch first, and then the new one.

-qIident            ← match any queue ID that contains ident (§11.8.2.3 on page
431)
-q!Iident            ← match any queue ID that does not contain ident (§11.8.2.4
on page 432)
-qRrecip            ← match any recipient address that contains recip (§11.8.2.3
on page 431)
-q!Rrecip            ← match any recipient address that does not contain recip (§
11.8.2.4 on page 432)
-qSfrom            ← match any sender address that contains from (§11.8.2.3 on pa
ge 431)
-q!Sfrom            ← match any sender address that does not contain from (§11.8.
2.4 on page 432)
-qGname            ← match any queue group with the name name (§11.8.2.5 on page
432)
-qQreason            ← match any queue group with the name reason (§11.10.2 on
page 439)

/usr/sbin/sendmail -qQ -qGokayclients -qSbob

mailq -qQ

/usr/sbin/sendmail -qSbob@your.domain -Q"Bob resigned today"

/usr/sbin/sendmail -qQ -qSbob@your.domain -Q

# mailq -qQ
                /var/spool/mailqueue (1 request)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------
h2VJcN3M012024   875429 Thu Mar 24 16:44 bob@your.domain
     QUARANTINE: Bob resigned today
                                         fred@compeditor.domain
                Total requests: 1

key          QUARANTINE
key          QUARANTINE:reason

Connect:192.168.1.23      QUARANTINE:Bob's PC
To:your.compeditor.gov    QUARANTINE:Review mail to our compeditor
From:head.hunter.domain   QUARANTINE:Employee theft?

R $* < @ bad.site > $*         $# error $@ quarantine $: reason

LOCAL_CONFIG
HX-review: $>Xreview

LOCAL_RULESETS
SXreview
R YES           $#error $@ quarantine $: X-review held for review

Oct  9 11:26:00 your.domain sendmail[4788]: f99IPuIH004788: ruleset=check_mail,
arg1=bob@compeditor.gov, quarantine=Hold mail from compeditor.gov

Oct  23 09:25:59 monkeyboy sendmail[52314]: f99IPuIH004787: milter=DocMilter,
quarantine=Suspect application/ms-word attachment

Nov 21 09:32:13 your.domain sendmail[33522]: fALHVwAQ033522: to=<bob@your.domain>,
delay=00:00:06, mailer=local, pri=30029, quarantine=Suspect application/ms-word
attachment, stat=quarantine

#./qtool.pl -Q /var/spool/hold /var/spool/mqueue

#./qtool.pl -b -Q -e '$msg{quarantine_reason} =˜ m/Virus/'

qreason

AUTH= parameter V8.10 and later

RFC2554 describes the currently approved method for SMTP authentication. One part of this method is to add the AUTH= extension to the MAIL From: command for ESMTP. This A line in the qf file is used to store the value passed in that parameter.

If you compiled sendmail with SASL defined (SASL on page 137), this value will be the actual value passed by the AUTH=. Otherwise, it is the value of $f ($f on page 824) normalized to the local domain if $f lacks a domain.

Message body type V8.6 and later

The message body type is described under the -B command-line switch (-B on page 232). The B line in the qf file stores whatever the body type was set to, either from the command line or by the SMTP MAIL command. The two usual body types are 8BITMIME and 7BIT.

The form of the B line is:

Btype

There must be no space between the B and the type. If the type is missing, the body type becomes the character value zero. If the entire B line is missing, the default is 7BIT. If type is longer than MAXNAME as defined in conf.h (MAX... on page 120) when compiling sendmail, it is truncated to MAXNAME-1 characters when the qf file is read.

Note that the type must be either 7bit or 8bitmime. Anything else will not be detected when the qf file is read and might eventually cause the ESMTP dialog to fail:

501 <sender>... Unknown BODY type badtype

This error will be reproduced at every MX site for the recipient until a site that does not speak ESMTP is found or until the MX list is exhausted.

Set controlling user V5.62 and later

To ensure secure handling of delivery, recipient addresses that are either a file or a program require that sendmail perform delivery as the owner of the file or program rather than as the user defined by the DefaultUser option (DefaultUser on page 1000). A file address is one that begins with a / character. A program address is one that begins with a | character. Both characters are detected after quotation marks have been stripped from the address.

To prevent potential security violations, sendmail must take special precautions when addresses in the qf file result from reading a ~/.forward or :include: file. When such an address is to be placed into the qf file (whether as a recipient’s address in an R line or as an error recipient’s address in an E line), sendmail first places a C line (for Controlling user) into the file and then the recipient’s address. The C line specifies the owner of the ~/.forward or :include: file:

Cgeorge
RPF:/u/users/george/mail/archive
Cben
RPF:|/u/users/ben/bin/mailfilter

Here, when sendmail later delivers to the recipients in this qf file, it first converts its user identity to that of the user george, and then resets itself back to being root again. The same process repeats with the next recipient, except that sendmail changes from root to ben and back again. If there is no C line preceding an R line, the previous C line’s value is carried down:

Cgeorge
RPF:/u/users/george/mail/archive
RPF:|/u/users/ben/bin/mailfilter          ← controlling user is george

The form of the C line in the qf file is:

Cuser                   ← prior to V8
Cuser:eaddr             ← V8.1 through V8.7.5
Cuser:uid:gid:eaddr     ← V8.7.6 and later

The C must begin the line and be immediately followed by user, with no intervening space. If no user follows the C, any prior controlling user is cleared and the identity that is used reverts to that specified by the DefaultUser option (DefaultUser on page 1000). If present, the user is the login name of the owner of the ~/.forward or :include: file that yielded the address in the next following R or E line. If user is the name of a user who is unknown to the system, prior to V8.7.6 and prior to V8.8 the effect was the same as though it were missing. Beginning with V8.8 and V8.7.6, an unknown user causes the identity to become that of the uid and gid. Beginning with V8 sendmail, an optional eaddr might be last. If present, the eaddr gives the address to use for error messages.

There can be only one C line immediately preceding each R and E line. Two C lines in a row have the effect of the second superseding the first.

Data file directory V8.12 and later

Beginning with V8.12 sendmail, it is possible to split envelopes for more efficient delivery. When sendmail splits an envelope, the new qf file will share a df file with the prior qf file. But to ensure that each qf file has it own df file, sendmail creates a hard link to make a copy of the df file. That way, the old qf file uses the old df file, and the new qf file uses the new df file. The two df files are the same because they are hard-linked together.

A problem arises when the two qf files are saved on two different disks. Because it is not possible to hard-link across disks, the new qf file is put on the new disk, but the new df file is left on the old disk. That way, an efficient hard link can still be made, but now the new qf file and its new df file are on different disks.

The d qf-file line was introduced to allow a qf file to find its corresponding df file when that df file is on a different disk. Whenever a qf file has a corresponding df file on a different disk, that qf file will contain a d line that looks like this:

d/path

The /path must be the full pathname of the directory that contains the df part. If /path is missing, or is not a directory that was set in the configuration file, the following error is logged and the qf file is considered bad, and marked as such (Bogus qf Files on page 419):

Losing qf file: bogus queue file directory

Datafile name Obsolete as of V8.7

Beginning with V8.7, sendmail looks for its datafile (the file containing the message body) under the same name as its qf file, but with the q changed into a d. Prior to V8.7, the D line in the qf file contained the name of the file that contained the message body. If the D line was missing, there was no message body. The form of the qf file D line was:

Dfile

The D must begin the line. The file must immediately follow with no intervening space. All text, from the first character following the D to the end of the line, is taken as the name of the file. There is no default for file; either it must be present, or the entire D line must be absent.

The sendmail program opens the df file for reading. If that open fails, sendmail syslog(3)s the following error message at LOG_CRIT and continues to process the qf file:

readqf: cannot open dfAA12345

Be aware that sendmail attempts to remove the file after it has been delivered to all recipients. If sendmail is unable to remove the file, and if the LogLevel option (LogLevel on page 1040) is greater than 97, sendmail syslog(3)s the following warning at LOG_DEBUG:

file: unlink-fail #

The file is the name of the file that could not be removed. The # is the error number, as defined in /usr/include/errno.h.

The df file is opened only when processing the queue file, not when printing it. When printing the queue, the df is stated so that its size can be printed.

Send errors to V8.6 and earlier

Notification of errors often requires special handling by sendmail. When mail to a mailing list fails, for example, sendmail looks for the owner of that list. If it finds one, the owner, not the sender, receives notification of the error. To differentiate error notification addresses from ordinary sender and recipient addresses, pre-V8.7 sendmail stored error addresses separately in the qf file, one per E line. Beginning with V8.7, this E line is no longer used. Instead, sendmail uses the S line.

The form of the E line in the qf file looks like this:

Eaddr          ← V8.6 and earlier

The E must begin the line. One or more addresses can be entered on that same line. Whitespace and commas can surround the individual addresses. Note, however, that sendmail places only a single address on each E line. There can be multiple E lines. Each is processed in turn.

Each line is fully processed as it is read. That is, the line is scanned for multiple addresses. Each address that is found is alias-expanded. Each resulting new address is processed by rule sets 3 and 0 to resolve a delivery agent for each.

If an alias expands to a program or a file (text that begins with a / or | character), that text is sent out in the delivered message’s Errors-To: line in that form. This can cause confusion when the message is later processed and bounced at the receiving site.

Saved flag bits V8.1 and later

Under V8 sendmail, the Timeout.queuewarn option (Timeout on page 1097) can specify an interval to wait before notifying the sender that a message could not immediately be delivered. To keep track of whether such a notification has been sent, sendmail stores the state of its EF_WARNING envelope flag in the qf file. If that flag is set, notification has already been sent.

Error mail messages sent by sendmail can occasionally be queued, rather than immediately delivered. The Timeout.queuewarn option notification should not be sent for such mail. If such mail remains in the queue too long, it should be canceled rather than bounced. V8 sendmail saves the state of the EF_RESPONSE envelope flag in the qf file. If that flag is set, the message is an error notification.

Beginning with V8.8, sendmail also records the state of the EF_HAS8BIT flag (the message body contains 8-bit data) and the EF_DELETE_BCC flag (delete empty Bcc: headers, Bcc: on page 1152).

All envelope flags are listed in Table 15-5 on page 545. The F line is used to save envelope flags for later restoration. Its form looks like this:

Fflags

Here, the flags are any combination of those shown in Table 11-7.

Only the letters listed in the table are recognized. Other letters are silently ignored. Note that these flags might be done away with in later versions of sendmail and new flags might be added without notice.

For security protection, V8 sendmail rejects and logs the following flag sequence:

From
    ↑
    a space here

See Funny Flag Bits in qf File on page 421 for more information about this.

Header line All versions of sendmail

The lines of text that form the message header are saved to the qf file, one per H line. Any header lines added by sendmail are also saved to H lines in the qf file.

The form of the H line is:

Hdefinition

The H must begin the line, and the definition must immediately follow with no intervening space. The definition is exactly the same as, and obeys the same rules as, the H commands in the configuration file (Overview on page 1120). Beginning with V8.10, if the header lacks header flags, an empty pair of ? characters are prefixed to the definition.

When sendmail writes header lines to the qf file, it pre-expands sendmail macros (replaces expressions such as $x with their values) and preresolves conditionals ($?, $!, and $.). Beginning with V8.10, the headers in the qf file might have been rewritten by rule sets.

The order in which H lines appear in the qf file is exactly the same as the order in which they appear in the delivered message.

Inode and device information for the df file V8.7 and later

When a machine crashes under Unix, files in a directory can become detached from that directory. When this happens, those orphaned files are saved in a directory called lost+found. Because filenames are saved only in directories, orphaned files are nameless. Consequently, Unix stores them in lost+found using their inode numbers as their names.

To illustrate, consider finding these four files in lost+found after a crash:

#1528 #1200 #3124 #3125

Two of these are qf files, and two are df files. Beginning with V8.7 sendmail, the qf files contain a record of the inode numbers for their corresponding df files. That information is stored in the I line:

Imajor/minor/ino

Here, the major and minor are the major and minor device numbers for the disk device that the df file was stored on. The ino is the inode number for the df file. In our lost+found example, the following command could be run to pair up the orphaned files:

% grep "^I.*/.*/" *
#1200:I123/45/3124
#1325:I123/45/1528

This shows that the qf file #1200 has the df file #3124 and that the qf file #1325 has the df file #1528.

The sendmail program does not check the inode number in the I line against the actual inode number of the df file. Instead, the I line is generated afresh each time the qf file is processed.

When df, qf, and xf subdirectories are used, and when those subdirectories are on separate disks, a crash of one disk can leave the df or qf file intact, and the other in lost+found.

The time last processed from the queue V8.7 and later

The MinQueueAge option (MinQueueAge on page 1057) sets the length of time a queued message must remain queued before delivery can again be tried. Each time sendmail processes a qf file, it subtracts the time stored in the K line from the current time and compares the result to the MinQueueAge. If sufficient time has not passed, the rest of the processing is skipped. (Note that this test is performed only if the qf file has been processed at least once; see the N line in N line on page 452.)

The time stored in the K line looks like this:

K703531020

This number represents the date and time in seconds since January 1, 1970. Every time the qf file is processed (delivery is attempted), the K line is updated with the current time.

Why the message was queued All versions of sendmail

When a mail message is placed into the queue because of an error during the delivery attempt, the nature of that error is stored in the M line of the qf file. The error is usually prefixed with Deferred:

Deferred: reason

Delivery can be deferred until a later queue run because of a temporary lack of services. For example, the reason might be “remote host is down.”

The form of the qf file M line is:

Mmsg

The M must begin the line. It is immediately followed by the msg with no intervening space. The text of msg is everything up to the end of the line. The msg created by sendmail can include the word Deferred: followed by a reason. The envelope-specific M line should appear before the S line.

Beginning with V8.12, each recipient also has an M line preceding its R line.

If the msg is missing, sendmail simply prints a blank line rather than a reason when showing the queue with mailq or the -bp command-line switch. If the M line is missing entirely, sendmail prints nothing.

The maximum number of characters in msg is defined by MAXLINE in conf.h (MAX... on page 120). Prior to V8.12, there could be only one M line in a qf file.

Number of times tried V8.7 and later

Each time delivery is attempted for a message, the number stored in its qf file’s N line is incremented by one. This number always begins at zero.

When delivering many messages to a single host, sendmail remembers failures. If one message fails to make it all the way through an SMTP dialog, all the following messages to that same host will be deferred (not attempted during the current queue run). For those deferred messages, the number of tries is correctly incremented as though the delivery was actually attempted.

The value in this N line is used to determine whether the delay of the MinQueueAge option (MinQueueAge on page 1057) should be triggered. This value, when zero, can also be used to enable a special first-time connection timeout (Timeout.iconnect (V8.8 and later) on page 1103).

Priority when processed from queue All versions of sendmail

Not all messages need to be treated equally. Messages that have failed often, for example, tend to continue to fail. When sendmail processes the messages in its queue, it sorts them by priority and attempts to deliver those with the lowest priority value first.

When a mail message is first placed into the queue, it is given an initial priority calculated when it was first created (RecipientFactor on page 1077), which is stored in the P line:

P640561

This number in the qf file is really a cost. The lower it is, the more preferentially the message is treated by sendmail. Each time the qf file is read, the number in the P line is incremented. The size of that increment is set by the value of the RetryFactor option (RetryFactor on page 1081). If that option is negative, this logic is inverted.

The form of the qf file P line is:

Ppri

The P must begin the line. The pri is a text representation of an integer value. The pri must immediately follow the P with no intervening space. The text in pri is converted to an integer using the C-library routine atol(3). That routine allows pri to be represented in text as a signed decimal number, an octal number, or a hexadecimal number.

If pri is absent, the priority value used is that of the configuration file RetryFactor option. If the entire P line is absent, the priority value begins at zero.

There should be only one P line in any qf file. Multiple P lines cause all but the last to be ignored.

Reason an envelope was quarantined V8.13 and later

When an envelope is quarantined (Queue Quarantining on page 438) the reason is stated in this q line. The q line is not part of a qf file, but is actually in the quarantined hf file.

qreason

Here, the reason can be manually inserted using sendmail’s command line, or automatically inserted via the access database or rule sets.

The DSN ORCPT address V8.7 and later

When a mail message arrives that includes an ORCPT parameter for the ESMTP RCPT command (see RFC1891), sendmail needs to save that parameter’s information separately from the RCPT recipient address:

RCPT To:<gw@wash.dc.gov> ORCPT=rfc822;gw@wash.dc.gov
    ↑                     ↑
     recipient address   parameter's information

Not all sites understand DSN. If sendmail forwards the message to such a site, it needs to omit the ORCPT parameter. Consequently, sendmail must not store that parameter with the RCPT address.

The Q line is used to separately store the ORCPT parameter information:

Qtype;addr

The type;addr is defined by RFC1891. The sendmail program checks the syntax of addr when that information is received, but otherwise merely stores type;addr as is in the Q line.

There must be only a single Q line for each recipient R line, and each such Q line must precede its corresponding R line.

Final-Recipient DSN address V8.10 and later

When sendmail bounces a mail message, it does so using DSN (T= on page 754). The type of address and the actual address of the final original recipient are reported in the bounce message:

Final-Recipient: RFC822; nosuchuser@site.com

This Final-Recipient line shows the type of address (here RFC822) and the actual address (here nosuchuser@site.com) of the final recipient.

Beginning with V8.10, sendmail composes this address expression only once (when the message is queued) and stores it in case the message bounces. The text following the Final-Recipient: is stored in the r line in the qf file, and looks like this:

rRFC822; nosuchuser@site.com

Note that sendmail performs no checks on the text following the r. This means that invalid DSN information placed there will become the text that follows the Final-Recipient: in the bounce message.

Recipient’s address All versions of sendmail

The qf file lists all the recipients for a mail message. There can be one recipient or many. When sendmail creates the qf file, it lists each recipient address on an individual R line. The form of the R line in the qf file looks like this:

Rflags:addr

The R must begin the line. Only a single address can appear on each R line. There can be multiple R lines. Each is processed in turn.

If the colon is present and if the version of the qf file is greater than 0, the characters between the R and the colon are interpreted as flags that further define the nature of the address:

Sender’s address All versions of sendmail

Each mail message must have a sender. The sendmail program can determine the sender in four ways:

The form of the S line in the qf file looks like this:

Saddr

The S must begin the line. Exactly one address must follow on that same line. Whitespace can surround that address. There can be only one S line in the qf file.

If the addr is missing, sendmail sets the sender to be the user who ran sendmail. If that user is not known in the passwd file (or database), sendmail syslog(3)s the following message and sets the sender to be postmaster:

Who are you?

The resulting address is then processed to extract the user’s full name into $x ($x on page 851). Finally, the sender’s address is rewritten by the canonify rule set 3, the parse rule set 0, and the final rule set 4.

Under all versions of sendmail, the address in the S line will include any RFC822 comment text that appeared with the original message. Under V8.7, if the F=c flag (F=c on page 768) is set for the sender’s delivery agent, all comment text is stripped from the address.

If sendmail is compiled with USERDB defined (USERDB on page 150), the sender address can optionally be rewritten by the User Database before it is placed into the S line. Such rewriting is allowed only if the delivery agent for the sender includes the F=i flag (F=i on page 772).

Time created All versions of sendmail

To limit the amount of time a message can remain in the queue before being bounced, sendmail must know when that message was first placed in the queue. That time of first placement is stored in the T line in the qf file. For example, the following number represents the date and time in seconds since January 1, 1970:

T703531020

Each time sendmail fails to deliver a message from the queue, it checks to see whether too much time has passed. It adds the T line value to the value specified in the Timeout.queuereturn option (Timeout on page 1097). If that sum is greater than the current time, the message is bounced instead of being left in the queue.

Messages are occasionally left in the queue for longer than the normal timeout period. This might happen, for example, if a remote machine is down but you know that it will eventually be brought back up. There are two ways to lengthen the amount of time a message can remain in the queue.

The preferred way is to create a temporary separate queue directory and move the necessary queued file to that temporary holding place. When the remote site comes back up, you can later process the files in that other queue by running sendmail with an artificially long Timeout.queuereturn value (Process Alternative Queues on page 436).

A second way to extend the life of messages in the queue is to edit the qf file and change the value stored in the T line. Just add 86400 to that value for each day you want to extend. Care is required to avoid editing a file that is currently being processed by sendmail.^[194]

There is currently no plan to give sendmail the ability to rejuvenate queued messages (make old messages appear young).

The form of the T line in the qf file is:

Tsecs

The T begins the line, and the secs must immediately follow with no intervening space. The numeric text that forms secs is converted to an integer using the C-library routine atol(3). That routine allows secs to be represented in text as a signed decimal number, an octal number, or a hexadecimal number.

If secs is absent or the entire T line is absent, the time value is zero. A zero value causes the mail message to time out immediately.

There should be only one T line in any qf file. Multiple T lines cause all but the last to be ignored.

Version of the qf file V8.7 and later

As sendmail evolves, it will continue to add new abilities to the qf file. To protect old versions of sendmail from wrongly misinterpreting new configuration files, the V line has been introduced. Note that prior to V8.7 sendmail there was no V line. The V line, and qf file version numbers for more modern implementations of sendmail, are shown in Table 11-8.

If the version found in a qf file is greater than that currently supported by sendmail, the following error will be printed if in verbose mode:

Version number in qf (bad) greater than max (max)

and the following message will be logged:

Losing qf file unsupported queue file version

The bad qf file will then be turned into a Qf file (marked as lost).

DSN ENVID envelope identifier V8.7 and later

The MAIL ESMTP command can optionally be followed by an RFC1891 ENVID envelope identifier:

MAIL From: <address> ENVID=envelopeID

ENVID is used to propagate a consistent envelope identifier (distinct from the Message-ID: header, Message-ID: on page 1159) that is permanently associated with a message.

The Z line holds that ENVID envelope identifier information:

ZenvelopeID

The ENVID information needs to be held separately from the S sender line because sendmail has no way to determine in advance whether a recipient host speaks ESMTP.

There must be only a single Z line in any qf file. The ${envid} sendmail macro (${envid} on page 823) also stores the ENVID value.

Deliver-by specification V8.12 and later

Beginning with V8.12, sendmail supports the DELIVERBY SMTP extension (defined by RFC2825). If the DeliverByMin option (DeliverByMin on page 1003) was defined with a positive value, a BY= equate can follow the SMTP MAIL From: command for inbound email. The BY= equate defines the window of time during which the message should be delivered. That equate’s time value can optionally be followed by a flag that states what to do upon delivery (an r flag), or upon delivery failure (an n flag). There can also be a flag that advises sendmail to trace delivery (a v flag). The BY= information is saved in the qf file on an ! line:

!flag time

Here, the time is the time specified by the By= and the flag is a 1-byte integer that encodes a value of 1 (notify the sender upon delivery), or 2 (return the message if it cannot be delivered in time). Additionally, the flag can be OR’d with a 0x10 (trace the delivery).

This ! line appears for inbound mail only, and appears only if you have configured your mc file to define the DeliverByMin option with a positive value.

Restore macro value V8.6 and later

The sendmail program uses the $r sendmail macro ($r on page 842) to store the protocol used when sendmail first received a mail message. If the message was received by using SMTP, that protocol is smtp. Otherwise, it is NULL.

The sendmail program uses the $s sendmail macro ($s on page 844) to store the full canonical name of the sender’s machine.

The sendmail program uses the $_ sendmail macro ($_ on page 801) to store RFC1413 identd(8) information and IP source-routing information.

When sendmail creates a qf file, it saves the values of the $r, $s, and $_ sendmail macros in lines that begin with $.

The form of the $ line in the qf file looks like this:

$Xvalue
${XXX}value

The $ must begin the line, and the sendmail macro’s single-character name (the X) or multicharacter name (the {XXX}) must immediately follow with no intervening space. The sendmail macro’s name is followed (again with no intervening space) by the value of the macro.

If value is missing, the value given to the macro is NULL. If the macro name and value are missing, the macro \ is given a value of NULL. If both are present, the macro whose name is specified is given the value specified (value).

There can be multiple $ lines. The sendmail macro names to be stored in the qf file are listed in the $={persistentMacros} class ($={persistentMacros} on page 873).

Mark EOF in qf file V8.7 and later

One form of attack against sendmail involves appending information to an existing qf file. To prevent such attacks, V8.7 introduced the dot line. In a qf file, any line that begins with a single dot:

.followed by anything

is considered to mark the end of the file’s useful information. Upon encountering that dot, sendmail continues to read the qf file. If any line follows the dot line, sendmail logs the following message and changes the qf file into a Qf file (Extra Data at End of qf File on page 420):

SECURITY ALERT: extra data in qf: bogus line here