Web caching refers to the act of storing certain web resources (i.e., pages and other data files) for possible future reuse. For example, Matilda is the first person in the office each morning, and she likes to read the local newspaper online with her wake-up coffee. As she visits the various sections, the Squid cache on their office network stores the HTML pages and JPEG images. Harry comes in a short while later and also reads the newspaper online. For him, the site loads much faster because much of the content is served from Squid. Additionally, Harry’s browsing doesn’t waste the bandwidth of the company’s DSL line by transferring the exact same data as when Matilda viewed the site.

A cache hit occurs each time Squid satisfies an HTTP request from its cache. The cache hit ratio, or cache hit rate, is the percentage of all requests satisfied as hits. Web caches typically achieve hit ratios between 30% and 60%. A similar metric, the byte hit ratio, represents the volume of data (i.e., number of bytes) served from the cache.

A cache miss occurs when Squid can’t satisfy a request from the cache. A miss can happen for any number of reasons. Obviously, the first time Squid receives a request for a particular resource, it is a cache miss. Similarly, Squid may have purged the cached copy to make room for new objects.

Another possibility is that the resource is uncachable. Origin servers can instruct caches on how to treat the response. For example, they can say that the data must never be cached, can be reused only within a certain amount of time, and so on. Squid also uses a few internal heuristics to determine what should, or should not, be saved for future use.

Cache validation is a process that ensures Squid doesn’t serve stale data to the user. Before reusing a cached response, Squid often validates it with the origin server. If the server indicates that Squid’s copy is still valid, the data is sent from Squid. Otherwise, Squid updates its cached copy as it relays the response to the client. Squid generally performs validation using timestamps. The origin server’s response usually contains a last-modified timestamp. Squid sends the timestamp back to the origin server to find if the original resource has changed.

For a detailed treatment of web caching, have a look at my book Web Caching, also by O’Reilly.

In the beginning was the CERN HTTP server. In addition to functioning as an HTTP server, it was also the first caching proxy. The caching module was written by Ari Luotonen in 1994.

That same year, the Internet Research Task Force Group on Resource Discovery (IRTF-RD) started the Harvest project. It was “an integrated set of tools to gather, extract, organize, search, cache, and replicate” Internet information. I joined the Harvest project near the end of 1994. While most people used Harvest as a local (or distributed) search engine, the Object Cache component was quite popular as well. The Harvest cache boasted three major improvements over the CERN cache: faster use of the filesystem, a single process design, and caching hierarchies via the Internet Cache Protocol.

Towards the end of 1995, many Harvest team members made the move to the exciting world of Internet-based startup companies. The original authors of the Harvest cache code, Peter Danzig and Anawat Chankhunthod, turned it into a commercial product. Their company was later acquired by Network Appliance. In early 1996, I joined the National Laboratory for Applied Network Research (NLANR) to work on the Information Resource Caching (IRCache) project, funded by the National Science Foundation. Under this project, we took the Harvest cache code, renamed it Squid, and released it under the GNU General Public License.

Since that time Squid has grown in size and features. It now supports a number of cool things such as URL redirection, traffic shaping, sophisticated access controls, numerous authentication modules, advanced disk storage options, HTTP interception, and surrogate mode (a.k.a. HTTP server acceleration).

Funding for the IRCache project ended in July 2000. Today, a number of volunteers continue to develop and support Squid. We occasionally receive financial or other types of support from companies that benefit from Squid.

Looking towards the future, we are rewriting Squid in C++ and, at the same time, fixing a number of design issues in the older code that are limiting to new features. We are adding support for protocols such as Edge Side Includes (ESI) and Internet Content Adaptation Protocol (ICAP). We also plan to make Squid support IPv6. A few developers are constantly making Squid run better on Microsoft Windows platforms. Finally, we will add more and more HTTP/1.1 features and work towards full compliance with the latest protocol specification.

Squid runs on all popular Unix systems, as well as Microsoft Windows. Although Squid’s Windows support is improving all the time, you may have an easier time with Unix. If you have a favorite operating system, I’d suggest using that one. Otherwise, if you’re looking for a recommendation, I really like FreeBSD.

Squid’s hardware requirements are generally modest. Memory is often the most important resource. A memory shortage causes a drastic degradation in performance. Disk space is, naturally, another important factor. More disk space means more cached objects and higher hit ratios. Fast disks and interfaces are also beneficial. SCSI performs better than ATA, if you can justify the higher costs. While fast CPUs are nice, they aren’t critical to good performance.

Because Squid uses a small amount of memory for every cached response, there is a relationship between disk space and memory requirements. As a rule of thumb, you need 32 MB of memory for each GB of disk space. Thus, a system with 512 MB of RAM can support a 16-GB disk cache. Your mileage may vary, of course. Memory requirements depend on factors such as the mean object size, CPU architecture (32- or 64-bit), the number of concurrent users, and particular features that you use.

People often ask such questions as, “I have a network with X users. What kind of hardware do I need for Squid?” These questions are difficult to answer for a number of reasons. In particular, it’s hard to say how much traffic X users will generate. I usually find it easier to look at bandwidth usage, and go from there. I tell people to build a system with enough disk space to hold 3-7 days worth of web traffic. For example, if your users consume 1 Mbps (HTTP and FTP traffic only) for 8 hours per day, that’s about 3.5 GB per day. So, I’d say you want between 10 and 25 GB of disk space for each Mbps of web traffic.

Squid is free software and a collaborative project. If you find Squid useful, please consider contributing back to the project in one or more of the following ways:

Squid is released as free software under the GNU General Public License. This means, for example, that anyone who distributes Squid must make the source code available to you. See http://www.gnu.org/licenses/gpl-faq.html for more information about the GPL.

The main source for up-to-date information about Squid is http://www.squid-cache.org. There you can:

Given that Squid is free software, you may need to rely on the kindness of strangers for occasional assistance. The best place to do this is the squid-users mailing list. Before posting a message to the mailing list, however, you should check Squid’s FAQ document to see if your question has already been asked and answered. If neither resource provides the help you need, you can contact one of the many services offering professional support for Squid.

If you are new to Squid, the next few chapters will help you get started. First, I’ll show you how to get the code, either the original source or precompiled binaries. In Chapter 3, I go through the steps necessary to compile and install Squid on your Unix system; this chapter is important because you’ll probably need to tune your system before compiling the source code. Chapter 4 provides a very brief introduction to Squid’s configuration file. Finally, Chapter 5 explains how to run Squid.

If you’ve already had a little experience installing and running Squid, you may want to skip ahead to Chapter 6.

The Squid developers make periodic releases of the source code. Each release has a version number, such as 2.5.STABLE4. The third component starts either with STABLE or DEVEL (short for development).

As you can probably guess, the DEVEL releases tend to have newer, experimental features. They are also more likely to have bugs. Inexperienced users should not run DEVEL releases. If you choose to try a DEVEL release, and you encounter problems, please report them to the Squid maintainers.

After spending some time in the development state, the version number changes to STABLE. These releases are suitable for all users. Of course, even the stable releases may have some bugs. The higher-numbered stable versions (e.g., STABLE3, STABLE4) are likely to have fewer bugs. If you are really concerned about stability, you may want to wait for one of these later releases.

So why can’t you just copy a precompiled binary to your system and expect it to work perfectly? The primary reason is that the code needs to know about certain operating system parameters. In particular, the most important parameter is the maximum number of open file descriptors. Squid’s ./configure script (see Section 3.4) probes for these values before compiling. If you take a Squid binary built for one value and run it on a system with a different value, you may encounter problems.

Another reason is that many of Squid’s features must be enabled at compile time. If you take a binary that somebody else compiled, and it doesn’t include the code for the features that you want, you’ll need to compile your own version anyway.

Finally, note that shared libraries sometimes make it difficult to share executable files between systems. Shared libraries are loaded at runtime. This is also known as dynamic linking. Squid’s ./configure script probes your system to find out certain things about your C library functions (if they are present, if they work, etc.). Although library functions don’t usually change, it is possible that two different systems have slightly different shared C libraries. This may become a problem for Squid if the two systems are different enough.

Getting the Squid source code is really quite easy. To get it, visit the Squid home page, http://www.squid-cache.org/. The home page has links to the current stable and development releases. If you aren’t located in the United States, you can select one of the many mirror sites. The mirror sites are usually named “wwwN.CC.squid-cache.org,” where N is a number and CC is a two-letter country code. For example, www1.au.squid-cache.org is an Australian mirror site. The home page has links to the current mirror sites.

Each Squid release branch (e.g., Squid-2.5) has its own HTML page. This page has links to the source code releases and “diffs” between releases. If you are upgrading from one release to the next, you may want to download the diff file and apply the patch as described in Section 3.7. The release pages describe the new features and important changes in each version, and also have links to bugs that have been fixed.

When web access isn’t an option, you can get the source release from the ftp://ftp.squid-cache.org FTP server or one of the FTP mirror sites. For the current versions, look in the pub/squid-2/DEVEL or pub/squid-2/STABLE directories. The Squid FTP site is mirrored at many locations as well. You can use the same country-code trick to guess some mirror sites, such as ftp1.uk.squid-cache.org.

The current Squid release distributions are about 1 MB in size. After downloading the compressed tar file, you can proceed to Chapter 3.

Some Unix distributions include, or make available, precompiled Squid packages. For Linux, you can easily find Squid RPMs. Often the Squid RPM is included on Linux CD-ROMs you can buy. The FreeBSD/NetBSD/OpenBSD distributions also contain Squid in their ports and/or packages collections.

While RPMs and precompiled packages may initially save you some time, they also have some drawbacks. As I already mentioned, certain features must be enabled or disabled before you start compiling Squid. The precompiled package that you install may not have the particular feature you want. Furthermore, Squid’s ./configure script probes your operating system for certain parameters. These parameters may be configured differently on your machine on which Squid was compiled. Finally, if you want to apply a patch to Squid, you’ll either have to wait for someone to build a new RPM/package or get the source and do it yourself.

I strongly encourage you to compile Squid from the source, but the decision is yours to make.

The Concurrent Versioning System (CVS) is a nifty package that allows you to simultaneously edit and manage source code and other files. Almost every open source software project uses CVS.

You can anonymously access Squid’s CVS files (read-only) to keep your source code up to date. The nice thing about CVS is that you can easily retrieve only the changes (diffs) of your current version. Thus, it is easy to see what has changed recently. Applying the changes to your current files efficiently synchronizes your source code with the official version.

CVS uses a tree-like indexing system. The trunk of the tree is called the head branch. For Squid’s repository, this is where all new changes and features are placed. The head branch usually contains experimental and, possibly unstable, code. The stable code is typically found on other branches.

To effectively use Squid’s anonymous CVS server, you first need to understand how different versions and branches are tagged. For example, the Version 2.5 branch is named SQUID_2_5. Particular releases, which represent a snapshot in time, have longer names, such as SQUID_2_5_STABLE4. To get exactly Squid Version 2.5.STABLE4, use the SQUID_2_5_STABLE4 tag; to get the latest code on the 2.5 branch, use SQUID_2_5.

To use the Squid anonymous CVS server, you first need to set the CVSROOT environment variable:

csh% setenv CVSROOT :pserver:anoncvs@cvs.squid-cache.org:/squid

Or, for Bourne shell users:

sh$ CVSROOT=:pserver:anoncvs@cvs.squid-cache.org:/squid
sh$ export CVSROOT

You then log in to the server:

% cvs login
(Logging in to anoncvs@cvs.squid-cache.org)
CVS password:

At the prompt, enter anoncvs for the password. Now you can check out the source tree with this command:

% cvs checkout -r SQUID_2_5 -d squid-2.5 squid

The -r option specifies the revision tag to retrieve. Omitting the -r option gets you the head branch. The -d option changes the top-level directory name in which files are placed. If you omit the -d option, the top-level directory is the same as the module name. The final command-line argument (squid) is the name of the module to check out.

Once you have the Squid source tree checked out, you can run the cvs update command to update your files and synchronize with the master repository. Additional interesting commands are cvs diff, cvs log, and cvs annotate.

To learn more about CVS, visit http://www.cvshome.org/.

The Squid developers maintain a separate site, currently hosted at SourceForge, for experimental Squid features. Check it out at http://devel.squid-cache.org/. There you’ll find a number of cutting-edge development projects that haven’t yet been integrated into the official Squid code base. You can access these projects through SourceForge’s anonymous CVS server or download diff files based on the standard releases.

If you’ve been using Unix for a while, chances are that you’ve already compiled a number of other software packages. If so, you can probably quickly scan this chapter. The procedure for compiling and installing Squid is similar to many other software distributions.

To compile Squid, you need an ANSI C compiler. Don’t be too alarmed by the “ANSI” part. Chances are that if you already have a C compiler, it is compliant with the ANSI specification. The GNU C compiler (gcc) is an excellent choice and widely available. Most operating systems come with a C compiler as a part of the standard installation. The common exceptions are Solaris and HP-UX. If you’re using one of those operating systems, you might not have a compiler installed.

Ideally you should compile Squid on the same system on which it will run. Part of the installation process probes your system for certain parameters, such as the number of available file descriptors. However, if your system doesn’t have a C compiler, you may be able to compile Squid elsewhere and then copy the binaries back. If the operating systems are different, Squid may encounter some problems. Also, Squid may become confused if the two systems have different kernel configurations.

In addition to a C compiler, you’ll also need Perl and awk. awk is a standard program on all Unix systems, so you shouldn’t need to worry about it. Perl is quite common, but it may not be installed on your system by default. You may need the gzip program to uncompress the source distribution file.

Solaris users, make sure that /usr/ccs/bin is in your PATH, even if you’re using gcc. To compile Squid, you may need the make and ar programs found in that directory.

After downloading the source distribution, you need to unpack it somewhere. The particular location doesn’t really matter. You can unpack Squid in your home directory or anywhere; you’ll need about 20 MB of free disk space. Personally, I like to use /tmp. Use the tar command to extract the source directory:

% cd /tmp
% tar xzvf /some/where/squid-2.5.STABLE4-src.tar.gz
squid-2.5.STABLE4/
squid-2.5.STABLE4/CONTRIBUTORS
squid-2.5.STABLE4/COPYING
squid-2.5.STABLE4/COPYRIGHT
squid-2.5.STABLE4/CREDITS
squid-2.5.STABLE4/ChangeLog
squid-2.5.STABLE4/INSTALL
squid-2.5.STABLE4/QUICKSTART
squid-2.5.STABLE4/README
...

Some tar programs don’t have the z option, which automatically uncompresses gzip files. In that case, you’ll need to use this command:

% gzip -dc /some/where/squid-2.5.STABLE4-src.tar.gz | tar xvf -

Once the source code has been unpacked, the next step is usually to configure the source tree. However, if this is the first time you’re compiling Squid, you should make sure certain kernel resource limits are high enough; to find out how, read on.

Squid requires a fair amount of kernel resources under moderate and high loads. In particular, you may need to configure your system with a higher-than-normal number of file descriptors and mbuf clusters. The file-descriptor limit can be especially annoying. You’d be better off to increase the limit before compiling Squid.

At this point, you might be tempted to get the precompiled binaries to avoid the hassle of building a new kernel.^[1] Unfortunately, you need to make a new kernel, regardless. Squid and the kernel exchange information through data structures that must not exceed the set file-descriptor limits. Squid checks these limits at runtime and uses the safest (smallest) value. Thus, even if a precompiled binary has higher file descriptors than the kernel, the kernel value takes precedence.

To change some settings, you must build and install a new kernel. This procedure varies among different operating systems. Consult Unix System Administration Handbook (Prentice Hall) or your operating-system documentation if necessary. If you’re using Linux, you probably don’t need to recompile your kernel.

csh% limit descriptors unlimited
csh% limit descriptors
descriptors    4096

sh$ ulimit -n unlimited
sh$ ulimit -n
4096

% sysctl -a | grep maxfiles
kern.maxfiles: 8192
kern.maxfilesperproc: 4096

checking Maximum number of file descriptors we can open... 4096

options       MAXFILES=8192

#define _ _FD_SETSIZE    8192

# echo 8192 > /proc/sys/fs/file-max

sh# ulimit -Hn 8192

set rlim_fd_max = 4096

% netstat -m
196/6368/32768 mbufs in use (current/peak/max):
        146 mbufs allocated to data
        50 mbufs allocated to packet headers
103/6182/8192 mbuf clusters in use (current/peak/max)
13956 Kbytes allocated to network (56% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

options         NMBCLUSTERS=16384

% netstat -n | grep TIME_WAIT
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0      0  192.43.244.42.19583    212.67.202.80.80       TIME_WAIT
tcp4       0      0  192.43.244.42.19597    202.158.66.190.80      TIME_WAIT
tcp4       0      0  192.43.244.42.19600    207.99.19.230.80       TIME_WAIT
tcp4       0      0  192.43.244.42.19601    216.131.72.121.80      TIME_WAIT
tcp4       0      0  192.43.244.42.19602    209.61.183.115.80      TIME_WAIT
tcp4       0      0  192.43.244.42.3128     128.109.131.47.25666   TIME_WAIT
tcp4       0      0  192.43.244.42.3128     128.109.131.47.25795   TIME_WAIT
tcp4       0      0  192.43.244.42.3128     128.182.72.190.1488    TIME_WAIT
tcp4       0      0  192.43.244.42.3128     128.182.72.190.2194    TIME_WAIT

# sysctl -w net.inet.ip.portrange.last=30000

# sysctl -w net.inet.ip.portlast=49151

# sysctl -w net.inet.ip.anonportmin=10000

# echo "1024 40000" > /proc/sys/net/ipv4/ip_local_port_range

Like many other Unix software packages, Squid uses a ./configure script to learn about an operating system before compiling. The ./configure script is generated by the popular GNU autoconf program. When the script runs, it probes the system in various ways to find out about libraries, functions, types, parameters, and features that may or may not be present. One of the first things that ./configure does is look for a working C compiler. If the compiler can’t be found or fails to compile a simple test program, the ./configure script can’t proceed.

The ./configure script has a number of different options. The most important is the installation prefix. Before running ./configure, you need to decide where Squid should live. The installation prefix determines the default locations for the Squid logs, binaries, and configuration files. You can change the location for those files after installing, but it’s easier if you decide now.

The default installation prefix is /usr/local/squid. Squid puts files in seven different subdirectories under the prefix:

% ls -l /usr/local/squid
total 5
drwxr-x---  2 wessels  wheel  512 Apr 28 20:42 bin
drwxr-x---  2 wessels  wheel  512 Apr 28 20:42 etc
drwxr-x---  2 wessels  wheel  512 Apr 28 20:42 libexec
drwxr-x---  3 wessels  wheel  512 Apr 28 20:43 man
drwxr-x---  2 wessels  wheel  512 Apr 28 20:42 sbin
drwxr-x---  4 wessels  wheel  512 Apr 28 20:42 share
drwxr-x---  4 wessels  wheel  512 Apr 28 20:43 var

Squid uses the bin, etc, libexec, man, sbin, and share directories for a few, relatively small files (or other directories) that don’t change very often. The files under the var directory, however, are a different story. This is where you’ll find Squid’s log files, which may grow quite large (tens or hundreds of megabytes). var is also the default location for the actual disk cache. You may want to put var on a different partition with plenty of space. One easy way to do this is with the —localstatedir option:

% ./configure --localstatedir=/bigdisk/var

You don’t need to worry too much about pathnames when configuring Squid. You can always change the pathnames later, in the squid.conf file.

% cd squid-2.5.STABLE4
% ./configure --enable-icmp --enable-htcp

configure: error: installation or configuration problem: C compiler
cannot create executables.

% setenv CC /usr/local/bin/gcc
% ./configure ...

Once ./configure has done its job, you can simply type make to begin compiling the source code:

% make

Normally, this part goes smoothly. You’ll see a lot of lines that look like this:^[2]

source='cbdata.c' object='cbdata.o' libtool=no  depfile='.deps/cbdata.Po'
tmpdepfile='.deps/cbdata.TPo'  depmode=gcc /bin/sh ../cfgaux/depcomp  gcc -DHAVE_
CONFIG_H -DDEFAULT_CONFIG_FILE=\"/usr/local/squid/etc/squid.conf\" -I. -I. -I../
include -I. -I. -I../include -I../include     -g -O2 -Wall -c 'test -f cbdata.c ||
echo './''cbdata.c
source='client_db.c' object='client_db.o' libtool=no  depfile='.deps/client_db.Po'
tmpdepfile='.deps/client_db.TPo'  depmode=gcc /bin/sh ../cfgaux/depcomp  gcc -DHAVE_
CONFIG_H -DDEFAULT_CONFIG_FILE=\"/usr/local/squid/etc/squid.conf\" -I. -I. -I../
include -I. -I. -I../include -I../include     -g -O2 -Wall -c 'test -f client_db.c ||
echo './''client_db.c
source='client_side.c' object='client_side.o' libtool=no  depfile='.deps/client_side.Po'
tmpdepfile='.deps/client_side.TPo'  depmode=gcc /bin/sh ../cfgaux/depcomp  gcc -
DHAVE_CONFIG_H -DDEFAULT_CONFIG_FILE=\"/usr/local/squid/etc/squid.conf\" -I. -I. -I../
include -I. -I. -I../include -I../include     -g -O2 -Wall -c 'test -f client_side.c ||
echo './''client_side.c
source='comm.c' object='comm.o' libtool=no  depfile='.deps/comm.Po' tmpdepfile='.
deps/comm.TPo'  depmode=gcc /bin/sh ../cfgaux/depcomp  gcc -DHAVE_CONFIG_H -DDEFAULT_
CONFIG_FILE=\"/usr/local/squid/etc/squid.conf\" -I. -I. -I../include -I. -I. -I../
include -I../include     -g -O2 -Wall -c 'test -f comm.c || echo './''comm.c

You may see some compiler warnings. In most cases, it is safe to ignore these. If you see a lot of them or something that looks really serious, report it to the developers as described in Section 16.5.

If the compilation gets all the way to the end without any errors, you can move to the next section, which describes how to install the programs you just built.

To verify that compilation was successful, you can run make again. You should see this output:^[3]

% make
Making all in lib...
Making all in scripts...
Making all in src...
Making all in fs...
Making all in repl...
'squid' is up to date.
'client' is up to date.
'unlinkd' is up to date.
'cachemgr.cgi' is up to date.
Making all in icons...
Making all in errors...
Making all in auth_modules...

The compilation step may fail for a number of reasons, including:

Always make sure that your compiler’s header files are synchronized with the library files. The header files normally reside in /usr/include, while libraries are found in /usr/lib. Linux’s popular RPM system makes it possible to upgrade one, but not the other. If the libraries are based on different header files, Squid may not compile.

If you want to upgrade the compiler on one of the open-source BSD variants, be sure to run make world from the /usr/src directory, rather than from the /usr/src/lib or /usr/src/include directories.

Here are some common compilation problems and error messages:

If you have problems compiling Squid, check the FAQ first. You may also want to search the Squid web site (use the search box on the home page). Finally, if you’re still stuck, send email to the squid-users@squid-cache.org list.

After compiling, you need to install the programs into their permanent directories. This might require superuser privileges, to put files in the installation directories. If so, become root first:

% su
Password:
# make install

If you enable Squid’s ICMP measurement features with the —enable-icmp option, you must install the pinger program. The pinger program must be installed with superuser privileges because only root is allowed to send and receive ICMP messages. The following command installs pinger with the appropriate permissions:

# make install-pinger

After installing Squid, you should see the following directories and files listed under the installation prefix directory (/usr/local/squid by default):

After you’ve been running Squid for a while, you may find that you need to patch the source code to fix a bug or add an experimental feature. Patches are posted for important bug fixes on the squid-cache.org web site. If you don’t want to wait for the next official release, you can download and apply the patch to your source code. You will then need to recompile Squid.

To apply a patch—also sometimes called a diff—you need a program called patch. Chances are that your operating system already has the patch program. If not, you can download it from the GNU collection (http://www.gnu.org/directory/patch.html). Note that if you’re using anonymous CVS (see Section 2.4), you don’t need to worry about patching files. The CVS system does it for you automatically when you update your tree.

To apply a patch, you need to save the patch file somewhere on your system. Then cd to the Squid source directory and run the command like this:

% cd squid-2.5.STABLE4
% patch < /tmp/patch_file

By default, the patch program tells you what it’s doing as it runs. Usually this output scrolls by very quickly, unless there is a problem. You can safely ignore the warnings that say offset NNN lines. If you don’t want to see all this output, use the -s option to make patch silent.

When patch updates the source files, it creates a backup copy of the original file. For example, if you’re applying a patch to src/http.c, patch names the backup file src/http.c.orig. Thus, if you want to undo the patch after applying it, you can simply rename all the .orig files back to their former names. To use this technique successfully, it’s a good idea to remove all .orig files before applying a patch.

If patch encounters a problem, it stops and prompts you for advice. Common problems are as follows:

Sometimes patch can’t apply part or all of the diff. In these cases, you’ll see such messages as Hunk 3 of 4 failed. The failed sections are saved to files named .rej. For example, if a failure occurs while processing src/http.c, patch saves that piece of the diff to src/http.c.rej. In some cases, you may be able to fix these by hand, but it’s usually not worth the trouble. If you have a lot of “failed hunks” or .rej files, it’s a good idea to download a whole new copy of the latest source code.

After you apply a patch, you need to recompile Squid. One of the great things about make is that it only recompiles the files that have changed. But sometimes make doesn’t comprehend all the intricate dependencies, and it doesn’t rebuild enough of the files. To be safe, it’s usually a good idea to recompile everything. The best way to do this is to clean the source tree before recompiling:

% make clean
% make

Sometimes you may find it necessary to rerun ./configure. For example, if you tune your kernel parameters, you must run ./configure again so it picks up the new settings. As you read this book, you may also find that you want to use features that must be enabled with ./configure options.

To rerun ./configure with the same options, use this command:

% ./config.status --recheck

Another technique is to “touch” the config.status file, which updates its timestamp. This causes make to re-run the ./configure script before compiling the source code:

% touch config.status
% make

To add or remove ./configure options, you need to type in the whole command again. If you can’t remember the previous options, just look at the top of the config.status file. For example:

% head config.status
#! /bin/sh
# Generated automatically by configure.
# Run this file to recreate the current configuration.
# This directory was configured as follows,
# on host foo.life-gone-hazy.com:
#
# ./configure  --enable-storeio=ufs,diskd --enable-carp \
#   --enable-auth-modules=NCSA
# Compiler output produced by configure, useful for debugging
# configure, is in ./config.log if it exists.

After rerunning ./configure, you must compile and install Squid again. To be safe, it’s a good idea to run make clean first:

% make clean
% make

Recall that ./configure caches the things it discovers about your system. In some situations, you’ll want to clear this cache and start the compilation process from the very beginning. You can simply remove the config.cache file if you like. Then, the next time ./configure runs, it won’t use the previous values. You can also restore the Squid source tree to its preconfigure state with the following command:

% make distclean

This removes all object files and other files created by the ./configure and make commands.

Squid’s configuration file is relatively straightforward. It is similar in style to many other Unix programs. Each line begins with a configuration directive, followed by some number of values and/or keywords. Squid ignores empty lines and comment lines (beginning with #) when reading the configuration file. Here are some sample configuration lines:

cache_log /squid/var/cache.log

# define the localhost ACL
acl Localhost src 127.0.0.1/32

connect_timeout 2 minutes

log_fqdn on

Some directives take a single value. For these, repeating the directive with a different value overwrites the previous value. For example, there is only one connect_timeout value. The first line in the following example has no effect because the second line overwrites it:

connect_timeout 2 minutes
connect_timeout 1 hour

On the other hand, some directives are actually lists of values. For these, each occurrence of the directive adds a new value to the list. The extension_methods directive works this way:

extension_methods UNGET
extension_methods UNPUT
extension_methods UNPOST

For these list-based directives, you can also usually put multiple values on the same line:

extension_methods UNGET UNPUT UNPOST

Many of the directives have common types. For example, connect_timeout is a time specification that has a number followed by a unit of time. For example:

connect_timeout 3 hours
client_lifetime 4 days
negative_ttl 27 minutes

Similarly, a number of directives refer to the size of a file or chunk of memory. For these, you can write a size specification as a decimal number, followed by bytes, KB, MB, or GB. For example:

minimum_object_size 12 bytes
request_header_max_size 10 KB
maximum_object_size 187 MB

Another type worth mentioning is the toggle, which can be either on or off. Many directives use this type. For example:

server_persistent_connections on
strip_query_terms off
prefer_direct on

In general, the configuration file directives may appear in any order. However, the order is important when one directive makes reference to something defined by another. Access controls are a good example. An acl must be defined before it can be used in an http_access rule:

acl Foo src 1.2.3.4
http_access deny Foo

Many things in squid.conf are case-sensitive, such as directive names. You can’t write HTTP_port instead of http_port.

The default squid.conf file contains comments describing each directive, as well as the default values. For example:

#  TAG: persistent_request_timeout
#       How long to wait for the next HTTP request on a persistent
#       connection after the previous request completes.
#
#Default:
# persistent_request_timeout 1 minute

Each time you install Squid, the current default configuration file is saved as squid.conf.default in the $prefix/etc directory. Since directives change from time to time, you can refer to this file for the most up-to-date documentation on squid.conf.

The rest of this chapter is about the handful of directives you need to know before running Squid for the very first time.

As you probably know, Unix processes and files have user and group ownership attributes. You need to select a user and group for Squid. This user and group combination must have read and write access to most of the Squid-related files and directories.

I highly recommend creating a dedicated squid user and group. This minimizes the chance that someone can exploit Squid to read other files on the system. If more than one person has administrative authority over Squid, you can add them to the squid group.

Unix processes inherit their parent process’ ownership attributes. That is, if you start Squid as user joe, Squid also runs as user joe. If you don’t want Squid to run as joe, you need to change your user ID beforehand. This is typically accomplished with the su command. For example:

joe% su - squid
squid% /usr/local/squid/sbin/squid

Unfortunately, running Squid isn’t always so simple. In some cases, you may need to start Squid as root, depending on your configuration. For example, only root can bind a TCP socket to privileged ports like port 80. If you need to start Squid as root, you must set the cache_effective_user directive. It tells Squid which user to become after performing the tasks that require special privileges. For example:

cache_effective_user squid

The name that you provide must be a valid user (i.e., in the /etc/passwd file). Furthermore, note that this directive is used only when you start Squid as root. Only root has the ability to become another user. If you start Squid as joe, it can’t switch to user squid.

You might be tempted to just run Squid as root without setting cache_effective_user. If you try, you’ll find that Squid refuses to run. This, again, is due to security concerns. If an outsider were somehow able to compromise or exploit Squid, he could gain full access to your system. Although we strive to make Squid secure and bug-free, this requirement provides some extra insurance, just in case.

If you start Squid as root without setting cache_effective_user, Squid uses nobody as the default value. Whatever user ID you choose for Squid, make sure it has read access to the files installed in $prefix/etc, $prefix/libexec, and $prefix/share. The user ID must also have write access to the log files and cache directory.

Squid also has a cache_effective_group directive, but you probably don’t need to set it. By default, Squid uses the cache_effective_user’s default group (from the password file).

The http_port directive tells Squid which port number to listen on for HTTP requests. The default is port 3128:

http_port 3128

If you are running Squid as a surrogate (see Chapter 15), you should probably set this to 80.

You can instruct Squid to listen on multiple ports with additional http_port lines. This is often useful if you must support groups of clients that have been configured differently. For example, the browsers from one department may be sending requests to port 3128, while another department uses port 8080. Simply list both port numbers as follows:

http_port 3128
http_port 8080

You can also use the http_port directive to make Squid listen on specific interface addresses. When Squid is used on a firewall, it should have two network interfaces: one internal and one external. You probably don’t want to accept HTTP requests coming from the external side. To make Squid listen on only the internal interface, simply put the IP address in front of the port number:

http_port 192.168.1.1:3128

I’ll discuss all the details of Squid’s log files in Chapter 13. For now the only thing you may need to worry about is where you want Squid to put its log files. The default location is a directory named logs under the installation prefix. For example, if you don’t use the —prefix= option with ./configure, the default log file directory is /usr/local/squid/var/logs.

You need to make sure that log files are stored on a disk partition with enough space. When Squid receives a write error for a log file, it exits and restarts. The primary reason for this behavior is to grab your attention. Squid wants to make sure you don’t miss any important logging information, especially if your system is being abused or attacked.

Squid has three main log files: cache.log, access.log, and store.log. The first of these, cache.log, contains informational and debugging messages. When you start Squid the first few times, you should closely watch this file. If Squid refuses to run, the reason is probably at the end of cache.log. Under normal conditions, this log file doesn’t become large enough to warrant any special attention. Also note that if you start Squid with the -s option, the important cache.log messages are also sent to your syslog daemon. You can change the location for this log file with the cache_log directive:

cache_log /squid/logs/cache.log

The access.log file contains a single line for each client request made to Squid. On average, each line is about 150 bytes. In other words, it takes about 150 MB to log one million client requests. Use the cache_access_log directive to change the location of this log file:

cache_access_log /squid/logs/access.log

If, for some reason, you don’t want Squid to log client requests, you can specify the log file pathname as /dev/null.

The store.log file is probably not very useful to most cache administrators. It contains a record for each object that enters and leaves the cache. The average record size is typically 175-200 bytes. However, Squid doesn’t create an entry in store.log for cache hits, so it contains fewer records than access.log. Use the cache_store_log directive to change the location:

cache_store_log /squid/logs/store.log

You can easily disable store.log altogether by specifying the location as none:

cache_store_log none

If you’re not careful, Squid’s log files increase in size without limit. Some operating systems enforce a 2-GB file size limit, even if you have plenty of free disk space. Exceeding this limit results in a write error, which then causes Squid to exit. To keep log file sizes reasonable, you should create a cron job that regularly renames and archives the log files. Squid has a built-in feature to make this easy. See Section 13.7 for an explanation of log file rotation.

I’ll have a lot to say about access controls in Chapter 6. For now, I’ll cover a few controls so that more enthusiastic readers can quickly start using Squid.

Squid’s default configuration file denies every client request. You must place additional access control rules in squid.conf before anyone can use the proxy. The simplest approach is to define an ACL that corresponds to your user’s IP addresses and an access rule that tells Squid to allow HTTP requests from those addresses. Squid has many different ACL types. The src type matches client IP addresses, and the http_access rules are checked for client HTTP requests. Thus, you need to add only two lines:

acl MyNetwork src 192.168.0.0/16
http_access allow MyNetwork

The tricky part is putting these lines in the right place. The order of http_access lines is very important, but the order of acl lines doesn’t matter. You should also be aware that the default configuration file contains some important access controls. You shouldn’t change or disrupt these until you fully comprehend their significance. When you edit squid.conf for the first time, look for this comment:

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

Insert your new rules below this comment, and before the http_access deny All line.

For the sake of completeness, here is a suitable initial access control configuration, including the recommended default controls and the example earlier:

acl All src 0/0
acl Manager proto cache_object
acl Localhost src 127.0.0.1/32
acl Safe_ports port 80 21 443 563 70 210 280 488 591 777 1025-65535
acl SSL_ports 443 563
acl CONNECT method CONNECT
acl MyNetwork src 192.168.0.0/16

http_access allow Manager Localhost
http_access deny Manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow MyNetwork
http_access deny All

Hopefully, you won’t need to worry about the visible_hostname directive. However, you’ll need to set it if Squid can’t figure out the hostname of the machine on which it is running. When this happens, Squid complains and refuses to run:

% squid -Nd1
FATAL: Could not determine fully qualified hostname.  Please set 'visible_hostname'

Squid wants to be sure about its hostname for a number of reasons:

If you see the fatal message mentioned previously, you need either to fix the hostname and DNS information or explicitly configure the hostname for Squid. In most cases, it is sufficient to ensure the hostname command returns a fully qualified hostname and add an entry to /etc/hosts. If that doesn’t work, just set the visible hostname in squid.conf:

visible_hostname squid.packet-pushers.net

You should set the cache_mgr directive as a favor to your users. The value is an email address users can write to in case a problem surfaces. The cache_mgr address appears in Squid’s error messages by default. For example:

cache_mgr squid@web-cache.net

After creating the minimal configuration file, you’re more or less ready to run Squid for the first time. To do that, just follow the instructions in the next chapter.

When you’ve mastered starting and stopping Squid, you can spend some time beefing up the configuration file. You may want to add more sophisticated access controls, which you’ll find documented in Chapter 6. Since I didn’t say anything about the disk cache yet, you should also spend a fair amount of time in Chapter 7 and Chapter 8.

Before getting too far into other things, let’s look at Squid’s command-line options. Many of these you will never use and some are useful only when debugging problems:

Before trying to start Squid, you should verify that your squid.conf file makes sense. This is easy to do. Just run the following command:

% squid -k parse

If you see no output, the configuration file is valid, and you can proceed to the next step.

However, if your configuration file contains an error, Squid tells you about it:

squid.conf line 62: http_access allow okay2
aclParseAccessLine: ACL name 'okay2' not found.

Here you can see that the http_access directive on line 62 references an ACL that doesn’t exist. Sometimes the error messages are less informative:

FATAL: Bungled squid.conf line 76: memory_pools

In this case, we forgot to put either on or off after the memory_pools directive on line 76.

It’s a good idea to develop the habit of using squid -k parse every time you modify your configuration file. If you don’t bother, and your file has some errors, Squid tells you about them and refuses to start anyway. If you end up managing a number of caches, it is likely that you’ll develop some scripts to automate starting, stopping, and reconfiguring Squid. You can use this feature in your scripts to ensure that the configuration files are always valid.

Before running Squid for the first time, and whenever you add a new cache_dir, you must initialize the cache directories. The command is simply:

% squid -z

For the UFS-related storage schemes (ufs, aufs, and diskd; see Chapter 8), this command creates the subdirectories needed under each cache_dir. You don’t need to worry that Squid will wipe out your current cache directories (if any).

Ownership and permissions are a common problem at this stage. Squid runs under a certain user ID, specified with cache_effective_user in squid.conf. This user ID must have read and write permission under each cache_dir directory. If not, you’ll see a message like this:

Creating Swap Directories
FATAL: Failed to make swap directory /usr/local/squid/var/cache/00:
    (13) Permission denied

In this case, you should make sure that all components of /usr/local/squid/var/cache are accessible to the user ID given in squid.conf. The final component—the cache directory—must be writable by this user ID as well.

Cache directory initialization may take a couple of minutes, depending on the size and number of cache directories, and the speed of your disk drives. If you want to watch the progress, use the -X option:

% squid -zX

Once you’ve initialized the cache directories, you should run Squid in a terminal window with logging to stderr. This way, you can easily spot any errors or problems and make sure that Squid successfully starts. Use the -N option to keep Squid in the foreground and the -d1 option to display level 1 debugging on stderr:

% squid -N -d1

You should see output like this:

2003/09/29 12:57:52| Starting Squid Cache 
version 2.5.STABLE4 for i386-unknown-freebsd4.8...
2003/09/29 12:57:52| Process ID 294
2003/09/29 12:57:52| With 1064 file descriptors available
2003/09/29 12:57:52| DNS Socket created on FD 4
2003/09/29 12:57:52| Adding nameserver 206.107.176.2 from /etc/resolv.conf
2003/09/29 12:57:52| Adding nameserver 205.162.184.2 from /etc/resolv.conf
2003/09/29 12:57:52| Unlinkd pipe opened on FD 9
2003/09/29 12:57:52| Swap maxSize 102400 KB, estimated 7876 objects
2003/09/29 12:57:52| Target number of buckets: 393
2003/09/29 12:57:52| Using 8192 Store buckets
2003/09/29 12:57:52| Max Mem  size: 8192 KB
2003/09/29 12:57:52| Max Swap size: 102400 KB
2003/09/29 12:57:52| Rebuilding storage in /usr/local/squid/var/cache (DIRTY)
2003/09/29 12:57:52| Using Least Load store dir selection
2003/09/29 12:57:52| Set Current Directory to /usr/local/squid/var/cache
2003/09/29 12:57:52| Loaded Icons.
2003/09/29 12:57:52| Accepting HTTP connections at 0.0.0.0, port 3128, FD 11.
2003/09/29 12:57:52| Accepting ICP messages at 0.0.0.0, port 3130, FD 12.
2003/09/29 12:57:52| WCCP Disabled.
2003/09/29 12:57:52| Ready to serve requests.

If you see an error message, you need to fix it before proceeding. Be sure to check the first few lines of output for warning messages. The most common errors are file/directory permissions and configuration file syntax errors. If you see an error message that doesn’t make sense, have a look at Chapter 16 for advice and information on troubleshooting Squid. If that doesn’t help, check the Squid FAQ, or search the mailing list archives for an explanation.

Once you see the Ready to serve requests message, test Squid with a few HTTP requests. You can do this by configuring your browser to use Squid as a proxy and then open a web page. If Squid is working correctly, the page should load as quickly as it would without using Squid. Alternatively, you can use the squidclient program that comes with Squid:

% squidclient http://www.squid-cache.org/

If this works, Squid’s home page HTML file will scroll across your terminal window. Once you’re confident that Squid works okay, you can interrupt the Squid process (i.e., with Ctrl-C) and run Squid as a daemon.

Normally you’ll want to run Squid as a daemon process (i.e., not attached to your terminal window). The easiest way to do this is simply execute Squid as follows:

% squid -s

The -s option causes Squid to write important status and warning messages to syslogd. Squid uses the LOCAL4 facility and the LOG_WARNING and LOG_NOTICE priorities. Your syslog daemon may or may not actually log Squid’s messages, depending on how it is configured. These same messages are written to the cache.log file, so it is safe to omit the -s option if you prefer.

When you start Squid without the -N option (as shown earlier), Squid automatically backgrounds itself and creates a parent/child process pair. The child process is the one that does all the real work. The parent process makes sure that a child process is always running. Thus, if the child process dies unexpectedly, the parent starts another so that Squid remains in operation. You can see this parent/child process interaction by looking at your syslog messages:

Jul 31 14:58:35 zapp squid[294]: Squid Parent: child process 296 started

Here you can see that the parent is process ID 294, and the child is 296. When you look at ps output, you’ll see that the child process is listed as (squid):

% ps ax | grep squid
  294  ??  Is     0:00.01 squid -sD
  296  ??  S      0:00.27 (squid) -sD (squid)

If the child Squid process dies unexpectedly, the parent starts another. For example:

Jul 31 15:02:53 zapp squid[294]: Squid Parent: child process 296 exited due to signal 6
Jul 31 15:02:56 zapp squid[294]: Squid Parent: child process 359 started

In some situations, the child Squid process may die immediately. Rather than constantly spawning new Squid processes, the parent process gives up if the child processes won’t stay running for at least 10 seconds five times in a row:

Jul 31 15:13:48 zapp squid[455]: Squid Parent: child process 474 exited with status 1
Jul 31 15:13:48 zapp squid[455]: Exiting due to repeated, frequent failures

If this happens to you, check syslog and Squid’s cache.log for error messages.

% /usr/local/squid/sbin/squid -sD

% squid -sD

Most likely, you’ll want Squid to start automatically every time your computer boots. Different operating systems vary widely in how their boot-up scripts work. I’ll describe some common environments here, but you may need to refer to your particular operating system for specific information.

/usr/local/squid/sbin/squid -s

/usr/bin/su nobody -c '/usr/local/squid/sbin/squid -s'

#!/bin/sh
#
# this script starts and stops Squid

case "$1" in
start)
          /usr/local/squid/sbin/squid -s
          echo -n ' Squid'
          ;;
stop)
          /usr/local/squid/sbin/squid -k shutdown
          ;;
esac

echo 8192 > /proc/sys/fs/file-max
limit -HSn 8192

sq:2345:once:/usr/local/squid/sbin/squid -s

sq:2345:respawn:/usr/local/squid/sbin/squid -Ns

# init q

Some people like to run Squid in a chroot environment. This is a Unix feature that gives a process a new root filesystem directory. It provides an extra level of security in the event that Squid is compromised. If an attacker somehow gains access to the operating system through Squid, she can only access files under the chroot filesystem. The other system files, outside of the chroot tree, remain inaccessible.

The easiest way to run Squid in a chroot environment is by specifying the new root directory in the squid.conf file with this directive:

chroot /new/root/directory

The chroot environment isn’t for first-time Unix users. It is a little tricky because you must replicate a number of files underneath the new root directory. For example, if the default configuration file is normally /usr/local/squid/etc/squid.conf, and you use the chroot directive, the file must be located at /new/root/directory/usr/local/squid/etc/squid.conf. You must copy all of the files under $prefix/etc, $prefix/share, and $prefix/libexec to the chroot directory. Make sure that $prefix/var and the cache directories exist and are writable under the chroot directory as well.

Chances are that your operating system requires a number of files in the chroot directory, such as /etc/resolv.conf and /dev/null. If you use an external helper program, such as a redirector (see Chapter 11) or an authenticator (see Chapter 12), you’ll also need some shared libraries from /usr/lib. You can use the ldd utility to find out which shared libraries are required for a given program:

% ldd /usr/local/squid/libexec/ncsa_auth
/usr/local/squid/libexec/ncsa_auth:
        libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x28067000)
        libm.so.2 => /usr/lib/libm.so.2 (0x28080000)
        libc.so.4 => /usr/lib/libc.so.4 (0x28098000)

You can also use the chroot command to test helpers:

# chroot /new/root/directory /usr/local/squid/libexec/ncsa_auth
/usr/libexec/ld-elf.so.1: Shared object "libcrypt.so.2" not found

For more information on chroot, see the chroot( ) manpage on your system.

The safest way to shut down Squid is with the squid -k shutdown command:

% squid -k shutdown

This command sends the TERM signal to the running Squid process. Upon receipt of the TERM signal, Squid closes its incoming sockets so that new requests aren’t accepted. It then waits some amount of time for outstanding requests to complete. The default is 30 seconds, which you can change with the shutdown_lifetime directive.

If, for some reason, the squid.pid file is missing or unreadable, the squid -k commands don’t work. In this case, you can manually kill Squid by finding the process ID with ps. For example:

% ps ax | grep squid

If you see more than one Squid process, be sure to kill the one that shows up as (squid). For example:

% ps ax | grep squid
  294  ??  Is     0:00.01 squid -sD
  296  ??  S      0:00.27 (squid) -sD (squid)
% kill -TERM 296

After sending the TERM signal, you may want to watch the log file to double-check that Squid is shutting down:

% tail -f logs/cache.log
2003/09/29 21:49:30| Preparing for shutdown after 9316 requests
2003/09/29 21:49:30| Waiting 10 seconds for active connections to finish
2003/09/29 21:49:30| FD 11 Closing HTTP connection
2003/09/29 21:49:31| Shutting down...
2003/09/29 21:49:31| FD 12 Closing ICP connection
2003/09/29 21:49:31| Closing unlinkd pipe on FD 9
2003/09/29 21:49:31| storeDirWriteCleanLogs: Starting...
2003/09/29 21:49:32| Finished.  Wrote 253 entries.
2003/09/29 21:49:32| Took 0.1 seconds (1957.6 entries/sec).
2003/09/29 21:49:32| Squid Cache (Version 2.5.STABLE4): Exiting normally.

If you use squid -k interrupt, Squid shuts down immediately, without waiting for active requests to complete. This is equivalent to sending the INT signal with kill.

As you learn more about Squid, you’ll probably find yourself making many changes to the squid.conf file. To have the new settings take effect, you can either shut down and restart Squid, or you can reconfigure Squid while it is running.

The best way to reconfigure a running Squid process is with the squid -k reconfigure command:

% squid -k reconfigure

When you run this command, a HUP signal is sent to the running Squid process. Squid then reads and parses the squid.conf file. If the operation is successful, you’ll see this in cache.log:

2003/09/29 22:02:25| Restarting Squid Cache (version 2.5.STABLE4)...
2003/09/29 22:02:25| FD 12 Closing HTTP connection
2003/09/29 22:02:25| FD 13 Closing ICP connection
2003/09/29 22:02:25| Cache dir '/usr/local/squid/var/cache' size remains unchanged
                     at 102400 KB
2003/09/29 22:02:25| DNS Socket created on FD 5
2003/09/29 22:02:25| Adding nameserver 10.0.0.1 from /etc/resolv.conf
2003/09/29 22:02:25| Accepting HTTP connections at 0.0.0.0, port 3128, FD 9.
2003/09/29 22:02:25| Accepting ICP messages at 0.0.0.0, port 3130, FD 11.
2003/09/29 22:02:25| WCCP Disabled.
2003/09/29 22:02:25| Loaded Icons.
2003/09/29 22:02:25| Ready to serve requests.

You need to be a little careful with the reconfigure option because it’s possible to make changes that cause a fatal error. For example, note that Squid closes and reopens the incoming HTTP and ICP sockets. If you change the http_port to a port number that Squid can’t open, it exits with a fatal error message.

Certain options and directives can’t be changed while Squid is running. This includes:

Solaris users may experience a subtle problem when reconfiguring Squid. The fopen( ) call in the Solaris stdio implementation requires an unused file descriptor less than 256. The FILE structure stores the file descriptor as an 8-bit value. Normally this isn’t a problem because Squid uses raw I/O (e.g., open( )) to open cache files. However, certain tasks that occur during the reconfigure procedure use fopen( ). These may fail if the first 256 file descriptors are already allocated.

Squid writes to a number of log files unless you disable them in squid.conf. You must periodically rotate the log files to prevent them from consuming too much disk space. Squid places a lot of importance on log files and exits with an error message when it can’t write to them. To keep disk space consumption under control, use the following command in a cron job:

% squid -k rotate

For example, this crontab entry rotates the logs every 24 hours, at 4 A.M.:

0 4 * * * /usr/local/squid/sbin/squid -k rotate

This command does two things. First, it closes the currently open log files. Then, it renames the cache.log, store.log, and access.log files by appending a numeric extension. For example, cache.log becomes cache.log.0, cache.log.0 becomes cache.log.1, and so on, up to the value of the logfile_rotate option.

Squid keeps only the last logfile_rotate versions of each log file. The older versions are simply removed during the renaming process. If you want to keep more copies, you need to increase the logfile_rotate limit or write some custom scripts that move the log files to a different location.

See Section 13.7 for additional information about rotating log files.

ACL elements are the building blocks of Squid’s access control implementation. These are how you specify things such as IP addresses, port numbers, hostnames, and URL patterns. Each ACL element has a name, which you refer to when writing the access list rules. The basic syntax of an ACL element is as follows:

acl name type value1 value2 ...

For example:

acl Workstations src 10.0.0.0/16

In most cases, you can list multiple values for one ACL element. You can also have multiple acl lines with the same name. For example, the following two configurations are equivalent:

acl Http_ports port 80 8000 8080

acl Http_ports port 80
acl Http_ports port 8000
acl Http_ports port 8080

acl Foo src 172.16.44.21/255.255.255.255
acl Foo src 172.16.44.21/32
acl Foo src 172.16.44.21

acl Xyz src 172.16.55.32/255.255.255.248
acl Xyz src 172.16.55.32/28

acl Bar src 172.16.66.0/255.255.255.0
acl Bar src 172.16.66.0/24
acl Bar src 172.16.66.0

acl Foo src 127.0.0.1/8

aclParseIpData: WARNING: Netmask masks away part of the specified IP in 'Foo'

acl Foo src 127.0.0.1/32

acl Foo src 127.0.0.0/8

acl Bar src 172.16.10.0-172.16.19.0/24

acl Foo src 172.16.10.0/24
acl Foo src 172.16.11.0/24
acl Foo src 172.16.12.0/24
acl Foo src 172.16.13.0/24
acl Foo src 172.16.14.0/24
acl Foo src 172.16.15.0/24
acl Foo src 172.16.16.0/24
acl Foo src 172.16.18.0/24
acl Foo src 172.16.19.0/24

acl Squid dst www.squid-cache.org

acl Foo src 1.2.3.0/24
acl Foo src 1.2.3.4/32

WARNING: '1.2.3.4' is a subnetwork of '1.2.3.0/255.255.255.0'
WARNING: because of this '1.2.3.4' is ignored to keep splay tree searching
         predictable
WARNING: You should probably remove '1.2.3.4' from the ACL named 'Foo'

www.squid-cache.org
squid-cache.org
org

acl A dstdomain foo.com
acl B dstdomain .foo.com

acl Foo dstdomain .lrrr.org .am.lrrr.org

WARNING: '.am.lrrr.org' is a subdomain of '.lrrr.org'
WARNING: because of this '.am.lrrr.org' is ignored to keep splay tree searching predictable
WARNING: You should probably remove '.am.lrrr.org' from the ACL named 'Foo'

acl Foo dstdomain .lrrr.org
acl Bar dstdomain .am.lrrr.org

^http://

.jpg$

\.jpg$

acl Foo url_regex -i ^http://www

acl Foo port 123
acl Bar port 1-1024

ACL Types

Now we can focus on the ACL types themselves. I present them here roughly in order of decreasing importance.

src

IP addresses are the most commonly used access control elements. Most sites use IP address controls to specify clients that are allowed to access Squid and those that aren’t. The src type refers to client (source) IP addresses. That is, when an src ACL appears in an access list, Squid compares it to the IP address of the client issuing the request.

Normally you want to allow requests from hosts inside your network and block all others. For example, if your organization is using the 192.168.0.0 subnet, you can use an ACL like this:

acl MyNetwork src 192.168.0.0

If you have many subnets, you can list them all on the same acl line:

acl MyNetwork src 192.168.0.0 10.0.1.0/24 10.0.5.0/24 172.16.0.0/12

Squid has a number of other ACL types that check the client’s address. The srcdomain type compares the client’s fully qualified domain name. It requires a reverse DNS lookup, which may add some delay to processing the request. The srcdom_regex ACL is similar, but it allows you to use a regular expression to compare domain names. Finally, the src_as type compares the client’s AS number.

dst

The dst type refers to origin server (destination) IP addresses. Among other things, you can use this to prevent some or all of your users from visiting certain web sites. However, you need to be a little careful with the dst ACL. Most of the requests received by Squid have origin server hostnames. For example:

GET http://www.web-cache.com/ HTTP/1.0

Here, www.web-cache.com is the hostname. When an access list rule includes a dst element, Squid must find the IP addresses for the hostname. If Squid’s IP cache contains a valid entry for the hostname, the ACL is checked immediately. Otherwise, Squid postpones request processing while the DNS lookup is in progress. This can add significant delay to some requests. To avoid those delays, you should use the dstdomain ACL type (instead of dst) whenever possible.^[2]

Here is a simple dst ACL example:

acl AdServers dst 1.2.3.0/24

Note that one problem with dst ACLs is that the origin server you are trying to allow or deny may change its IP address. If you don’t notice the change, you won’t bother to update squid.conf. You can put a hostname on the acl line, but that adds some delay at startup. If you need many hostnames in ACLs, you may want to preprocess the configuration file and turn the hostnames into IP addresses.

myip

The myip type refers to the IP address where clients connect to Squid. This is what you see under the Local Address column when you run netstat -n on the Squid box. Most Squid installations don’t use this type. Usually, all clients connect to the same IP address, so this ACL element is useful only on systems that have more than one IP address.

To understand how myip may be useful, consider a simple company local area network with two subnets. All users on subnet-1 are programmers and engineers. Subnet-2 consists of accounting, marketing, and other administrative departments. The system on which Squid runs has three network interfaces: one on subnet-1, one on subnet-2, and the third connecting to the outbound Internet connection (see Figure 6-1).

Figure 6-1. An application of the myip ACL

When properly configured, all users on subnet-1 connect to Squid’s IP address on that subnet, and similarly, all subnet-2 users connect to Squid’s second IP address. You can use this to give the technical staff on subnet-1 full access, while limiting the administrative staff to only work-related web sites.

The ACLs might look like this:

acl Eng myip 172.16.1.5
acl Admin myip 172.16.2.5

Note, however, that with this scheme you must take special measures to prevent users on one subnet from connecting to Squid’s address on the other subnet. Otherwise, clever users on the accounting and marketing subnet can connect through the programming and engineering subnet and bypass your restrictions.

dstdomain

In some cases, you’re likely to find that name-based access controls make a lot of sense. You can use them to block access to certain sites, to control how Squid forwards requests and to make some responses uncachable. The dstdomain type is very useful because it checks the hostname in requested URLs.

First, however, I want to clarify the difference between the following two lines:

acl A dst www.squid-cache.org
acl B dstdomain www.squid-cache.org

A is really an IP address ACL. When Squid parses the configuration file, it looks up the IP address for www.squid-cache.org and stores the address in memory. It doesn’t store the name. If the IP address for www.squid-cache.org changes while Squid is running, Squid continues using the old address.

The dstdomain ACL, on the other hand, is stored as a domain name (i.e., a string), not as an IP address. When Squid checks ACL B, it uses string comparison functions on the hostname part of the URL. In this case, it doesn’t really matter if the www.squid-cache.org IP changes while Squid is running.

The primary problem with dstdomain ACLs is that some URLs have IP addresses instead of hostnames. If your goal is to block access to certain sites with dstdomain ACLs, savvy users can simply look up the site’s IP address manually and insert it into the URL. For example, these two URLs bring up the same page:

http://www.squid-cache.org/docs/FAQ/
http://206.168.0.9/docs/FAQ/

The first can be easily matched with dstdomain ACLs, but the second can’t. Thus, if you elect to rely on dstdomain ACLs, you may want to also block all requests that use an IP address instead of a hostname. See the Section 6.3.8 for an example.

srcdomain

The srcdomain ACL is somewhat tricky as well. It requires a so-called reverse DNS lookup on each client’s IP address. Technically, Squid requests a DNS PTR record for the address. The answer—a fully qualified domain name (FQDN)—is what Squid compares to the ACL value. (Refer to O’Reilly’s DNS and BIND for more information about DNS PTR records.)

As with dst ACLs, FQDN lookups are a potential source of significant delay. The request is postponed until the FQDN answer comes back. FQDN answers are cached, so the srcdomain lookup delay usually occurs only for the client’s first request.

Unfortunately, srcdomain lookups sometimes don’t work. Many organizations fail to keep their reverse lookup databases current. If an address doesn’t have a PTR record, the ACL check fails. In some cases, requests may be postponed for a very long time (e.g., two minutes) until the DNS lookup times out. If you choose to use the srcdomain ACL, make sure that your own DNS in-addr.arpa zones are properly configured and working. Assuming that they are, you can use an ACL like this:

acl LocalHosts srcdomain .users.example.com

port

Most likely, you’ll want to use the port ACL to limit access to certain origin server port numbers. As I’ll explain shortly, Squid really shouldn’t connect to certain services, such as email and IRC servers. The port ACL allows you to define individual ports, and port ranges. Here is an example:

acl HTTPports port 80 8000-8010 8080

HTTP is similar in design to other protocols, such as SMTP. This means that clever users can trick Squid into relaying email messages to an SMTP server. Email relays are one of the primary reasons we must deal with a daily deluge of spam. Historically, spam relays have been actual mail servers. Recently, however, more and more spammers are using open HTTP proxies to hide their tracks. You definitely don’t want your Squid cache to be used as a spam relay. If it is, your IP address is likely to end up on one of the many mail-relay blacklists (MAPS, ORDB, spamhaus, etc.). In addition to email, there are a number of other TCP/IP services that Squid shouldn’t normally communicate with. These include IRC, Telnet, DNS, POP, and NNTP. Your policy regarding port numbers should be either to deny the known-to-be-dangerous ports and allow the rest, or to allow the known-to-be-safe ports and deny the rest.

My preference is to be conservative and allow only the safe ports. The default squid.conf includes the following Safe_ports ACL:

acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443 563     # https, snews
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http

http_access deny !Safe_ports

This is a sensible approach. It allows users to connect to any nonprivileged port (1025-65535), but only specific ports in the privileged range. If one of your users tries to request a URL, such as http://www.lrrr.org:123/, Squid returns an access denied error message. In some cases, you may need to add additional port numbers to the Safe_ports ACL to keep your users happy.

A more liberal approach is to deny access to certain ports that are known to be particularly dangerous. The Squid FAQ includes an example of this:

acl Dangerous_ports 7 9 19 22 23 25 53 109 110 119

http_access deny Dangerous_ports

One drawback to the Dangerous_ports approach is that Squid ends up searching the entire list for almost every request. This places a little extra burden on your CPU. Most likely, 99% of the requests reaching Squid are for port 80, which doesn’t appear in the Dangerous_ports list. The list is searched for all of these requests without resulting in a match. However, integer comparison is a fast operation and should not significantly impact performance.

myport

Squid also has a myport ACL. Whereas the port ACL refers to the origin server port number, myport refers to the port where Squid receives client requests. Squid listens on different port numbers if you specify more than one with the http_port directive.

The myport ACL is particularly useful if you use Squid as an HTTP accelerator for your web site and as a proxy for your users. You can accept the accelerator requests on port 80 and the proxy requests on port 3128. You probably want the world to access the accelerator, but only your users should access Squid as a proxy. Your ACLs may look something like this:

acl AccelPort myport 80
acl ProxyPort myport 3128
acl MyNet src 172.16.0.0/22

http_access allow AccelPort         # anyone
http_access allow ProxyPort MyNet   # only my users
http_access deny ProxyPort          # deny others

method

The method ACL refers to the HTTP request method. GET is typically the most common method, followed by POST, PUT, and others. This example demonstrates how to use the method ACL:

acl Uploads method PUT POST

Squid knows about the following standard HTTP methods: GET, POST, PUT, HEAD, CONNECT, TRACE, OPTIONS, and DELETE. In addition, Squid knows about the following methods from the WEBDAV specification, RFC 2518: PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK.^[3] Certain Microsoft products use nonstandard WEBDAV methods, so Squid knows about them as well: BMOVE, BDELETE, BPROPFIND. Finally, you can configure Squid to understand additional request methods with the extension_methods directive. See Appendix A.

Note that the CONNECT method is special in a number of ways. It is the method used for tunneling certain requests through HTTP proxies (see also RFC 2817: Upgrading to TLS Within HTTP/1.1). Be especially careful with the CONNECT method and remote server port numbers. As I talked about in the previous section, you don’t want Squid to connect to certain remote services. You should limit the CONNECT method to only the HTTPS/SSL and perhaps NNTPS ports (443 and 563, respectively). The default squid.conf does this:

acl CONNECT method CONNECT
acl SSL_ports 443 563

http_access allow CONNECT SSL_ports
http_access deny CONNECT

With this configuration, Squid only allows tunneled requests to ports 443 (HTTPS/SSL) and 563 (NNTPS). CONNECT method requests to all other ports are denied.

PURGE is another special request method. It is specific to Squid and not defined in any of the RFCs. It provides a way for the administrator to forcibly remove cached objects. Since this method is somewhat dangerous, Squid denies PURGE requests by default, unless you define an ACL that references the method. Otherwise, anyone with access to the cache may be able to remove any cached object. I recommend allowing PURGE from localhost only:

acl Purge method PURGE
acl Localhost src 127.0.0.1
http_access allow Purge Localhost
http_access deny Purge

See Section 7.6 for more information on removing objects from Squid’s cache.

proto

This type refers to a URI’s access (or transfer) protocol. Valid values are the following: http, https (same as HTTP/TLS), ftp, gopher, urn, whois, and cache_object. In other words, these are the URL scheme names (RFC 1738 terminology) supported by Squid. For example, suppose that you want to deny all FTP requests. You can use the following directives:

acl FTP proto FTP
http_access deny FTP

The cache_object scheme is a feature specific to Squid. It is used to access Squid’s cache management interface, which I’ll talk about in Section 14.2. Unfortunately, it’s not a very good name, and it should probably be changed.

The default squid.conf file has a couple of lines that restrict cache manager access:

acl Manager proto cache_object
acl Localhost src 127.0.0.1

http_access allow Manager Localhost
http_access deny Manager

These configuration lines allow cache-manager requests only when they come from the localhost address. All other cache-manager requests are denied. This means that any user with an account on the Squid machine can access the potentially sensitive cache-manager information. You may want to modify the cache-manager access controls or protect certain pages with passwords. I’ll talk about that in Section 14.2.2.

time

The time ACL allows you to control access based on the time of day and the day of the week. The syntax is somewhat cryptic:

acl name [days] [h1:m1-h2:m2]

You can specify days of the week, starting and stopping times, or both. Days are specified by the single-letter codes shown in Table 6-2. Times are specified in 24-hour format. The starting time must be less than the ending time, which makes it awkward to write time ACLs that span “midnights.”

Table 6-2. Day codes for the time ACL

Code	Day
S	Sunday
M	Monday
T	Tuesday
W	Wednesday
H	Thursday
F	Friday
A	Saturday
D	All weekdays (M-F)

Tip

Days and times are interpreted with the localtime( ) function, which takes into account your local time zone and daylight savings time settings. Make sure that your computer knows what time zone it is in! You’ll also want to make sure that your clock is synchronized to the correct time.

To specify a time ACL that matches your weekday working hours, you can write:

acl Working_hours MTWHF 08:00-17:00

or:

acl Working_hours D 08:00-17:00

Let’s look at a trickier example. Perhaps you’re an ISP that relaxes access during off-peak hours, say 8 P.M. to 4 A.M. Since this time spans midnight, you can’t write “20:00-04:00.” Instead you’ll need either to split this into two ACLs or define the peak hours and use negation. For example:

acl Offpeak1 20:00-23:59
acl Offpeak2 00:00-04:00
http_access allow Offpeak1 ...
http_access allow Offpeak2 ...

Alternatively, you can do it like this:

acl Peak 04:00-20:00
http_access allow !Peak ...

Although Squid allows it, you probably shouldn’t put more than one day list and time range on a single time ACL line. The parser isn’t always smart enough to figure out what you want. For example, if you enter this:

acl Blah time M 08:00-10:00 W 09:00-11:00

what you really end up with is this:

acl Blah time MW 09:00-11:00

The parser ORs weekdays together and uses only the last time range. It does work, however, if you write it like this, on two separate lines:

acl Blah time M 08:00-10:00
acl Blah time W 09:00-11:00

ident

The ident ACL matches usernames returned by the ident protocol. This is a simple protocol, that’s documented in RFC 1413. It works something like this:

1. A user-agent (client) establishes a TCP connection to Squid.

2. Squid connects to the ident port (113) on the client’s system.

3. Squid writes a line containing the two TCP port numbers of the client’s first connection. The Squid-side port number is probably 3128 (or whatever you configured in squid.conf). The client-side port is more or less random.

4. The client’s ident server writes back the username belonging to the process that opened the first connection.

5. Squid records the username for access control purposes and for logging in access.log.

When Squid encounters an ident ACL for a particular request, that request is postponed until the ident lookup is complete. Thus, the ident ACL may add some significant delays to your users’ requests.

We recommend using the ident ACL only on local area networks and only if all or most of the client workstations run the ident server. If Squid and the client workstations are connected to a LAN with low latency, the ident ACL can work well. Using ident for clients connecting over WAN links is likely to frustrate both you and your users.

The ident protocol isn’t very secure. Savvy users will be able to replace their normal ident server with a fake server that returns any username they select. For example, if I know that connections from the user administrator are always allowed, I can write a simple program that answers every ident request with that username.

Tip

You can’t use ident ACLs with interception caching (see Chapter 9). When Squid is configured for interception caching, the operating system pretends that it is the origin server. This means that the local socket address for intercepted TCP connections has the origin server’s IP address. If you run netstat -n on Squid, you’ll see a lot of foreign IP addresses in the Local Address column. When Squid makes an ident query, it creates a new TCP socket and binds the local endpoint to the same IP address as the local end of the client’s TCP connection. Since the local address isn’t really local (it’s some far away origin server’s IP address), the bind( ) system call fails. Squid handles this as a failed ident query.

Note that Squid also has a feature to perform “lazy” ident lookups on clients. In this case, requests aren’t delayed while waiting for the ident query. Squid logs the ident information if it is available by the time the HTTP request is complete. You can enable this feature with the ident_lookup_access directive, which I’ll discuss later in this chapter.

proxy_auth

Squid has a powerful, and somewhat confusing, set of features to support HTTP proxy authentication. With proxy authentication, the client’s HTTP request includes a header containing authentication credentials. Usually, this is simply a username and password. Squid decodes the credential information and then queries an external authentication process to find out if the credentials are valid.

Squid currently supports three techniques for receiving user credentials: the HTTP Basic protocol, Digest authentication protocol, and NTLM. Basic authentication has been around for a long time. By today’s standards, it is a very insecure technique. Usernames and passwords are sent together, essentially in cleartext. Digest authentication is more secure, but also more complicated. Both Basic and Digest authentication are documented in RFC 2617. NTLM also has better security than Basic authentication. However, it is a proprietary protocol developed by Microsoft. A handful of Squid developers have essentially reverse-engineered it.

In order to use proxy authentication, you must also configure Squid to spawn a number of external helper processes. The Squid source code includes some programs that authenticate against a number of standard databases, including LDAP, NTLM, NCSA-style password files, and the standard Unix password database. The auth_param directive controls the configuration of all helper programs. I’ll go through it in detail in Chapter 12.

The auth_param directive and proxy_auth ACL is one of the few cases where their order in the configuration file is important. You must define at least one authentication helper (with auth_param) before any proxy_auth ACLs. If you don’t, Squid prints an error message and ignores the proxy_auth ACLs. This isn’t a fatal error, so Squid may start anyway, and all your users’ requests may be denied.

The proxy_auth ACL takes usernames as values. However, most installations simply use the special value REQUIRED:

auth_param ...
acl Auth1 proxy_auth REQUIRED

In this case, any request with valid credentials matches the ACL. If you need fine-grained control, you can specify individual usernames:

auth_param ...
acl Auth1 proxy_auth allan bob charlie
acl Auth2 proxy_auth dave eric frank

Tip

Proxy authentication doesn’t work with HTTP interception because the user-agent doesn’t realize it’s talking to a proxy rather than the origin server. The user-agent doesn’t know that it should send a Proxy-Authorization header in its requests. See Section 9.2 for additional details.

src_as

This type checks that the client (source) IP address belongs to a specific AS number. (See Section 6.1.1.6 for information on how Squid maps AS numbers to IP addresses.) As an example, consider the fictitious ISP that uses AS 64222 and advertises the 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16 networks. You can write an ACL like this, which allows requests from any host in the ISP’s address space:

acl TheISP src 10.0.0.0/8
acl TheISP src 172.16.0.0/12
acl TheISP src 192.168.0.0/16
http_access allow TheISP

Alternatively, you can write it like this:

acl TheISP src_as 64222
http_access allow TheISP

Not only is the second form shorter, it also means that if the ISP adds more networks, you won’t have to update your ACL configuration.

dst_as

The dst_as ACL is often used with the cache_peer_access directive. In this way, Squid can forward cache misses in a manner consistent with IP routing. Consider an ISP that exchanges routes with a few other ISPs. Each ISP operates their own caching proxy, and these proxies can forward requests to each other. Ideally, ISP A forwards cache misses for servers on ISP B’s network to ISP B’s caching proxy. An easy way to do this is with AS ACLs and the cache_peer_access directive:

acl ISP-B-AS dst_as 64222
acl ISP-C-AS dst_as 64333
cache_peer proxy.isp-b.net parent 3128 3130
cache_peer proxy.isp-c.net parent 3128 3130
cache_peer_access proxy.isb-b.net allow ISP-B-AS
cache_peer_access proxy.isb-c.net allow ISP-C-AS

These access controls make sure that the only requests sent to the two ISPs are for their own origin servers. I’ll talk further about cache cooperation in Chapter 10.

snmp_community

The snmp_community ACL is meaningful only for SNMP queries, which are controlled by the snmp_access directive. For example, you might write:

acl OurCommunityName snmp_community hIgHsEcUrItY
acl All src 0/0
snmp_access allow OurCommunityName
snmp_access deny All

In this case, an SNMP query is allowed only if the community name is set to hIgHsEcUrItY.

maxconn

The maxconn ACL refers to the number of simultaneous connections from a client’s IP address. Some Squid administrators find this a useful way to prevent users from abusing the proxy or consuming too many resources.

The maxconn ACL matches a request when that request exceeds the number you specify. For this reason, you should use maxconn ACLs only in deny rules. Consider this example:

acl OverConnLimit maxconn 4
http_access deny OverConnLimit

In this case, Squid allows up to four connections at once from each IP address. When a client makes the fifth connection, the OverConnLimit ACL is matched, and the http_access rule denies the request.

The maxconn ACL feature relies on Squid’s client database. This database keeps a small data structure in memory for each client IP address. If you have a lot of clients, this database may consume a significant amount of memory. You can disable the client database in the configuration file with the client_db directive. However, if you disable the client database, the maxconn ACL will no longer work.

arp

The arp ACL is used to check the Media Access Control (MAC) address (typically Ethernet) of cache clients. The Address Resolution Protocol (ARP) is the way that hosts find the MAC address corresponding to an IP address. This feature came about when some university students discovered that, under Microsoft Windows, they could set a system’s IP address to any value. Thus, they were able to circumvent Squid’s address-based controls. To escalate this arms race, a savvy system administrator gave Squid the ability to check the client’s Ethernet addresses.

Unfortunately, this feature uses nonportable code. If you use Solaris or Linux, you should be able to use arp ACLs. If not, you’re out of luck. The best way to find out is to add the —enable-arp-acl option when you run ./configure.

The arp ACL feature contains another important limitation. ARP is a datalink layer protocol. It works only for hosts on the same subnet as Squid. You can’t easily discover the MAC address of a host on a different subnet. If you have routers between Squid and your users, you probably can’t use arp ACLs.

Now that you know when not to use them, let’s see how arp ACLs actually look. The values are Ethernet addresses, as you would see in ifconfig and arp output. For example:

acl WinBoxes arp 00:00:21:55:ed:22
acl WinBoxes arp 00:00:21:ff:55:38

srcdom_regex

The srcdom_regex ACL allows you to use regular expression matching on client domain names. This is similar to the srcdomain ACL, which uses modified substring matching. The same caveats apply here: some client addresses don’t resolve back to domain names. As an example, the following ACL matches hostnames that begin with dhcp:

acl DHCPUser srcdom_regex -i ^dhcp

Because of the leading ^ symbol, this ACL matches the hostname dhcp12.example.com, but not host12.dhcp.example.com.

dstdom_regex

The dstdom_regex ACL is obviously similar, except that it applies to origin server names. The issues with dstdomain are relevant here, too. The following example matches hostnames that begin with www:

acl WebSite dstdom_regex -i ^www\.

Here is another useful regular expression that matches IP addresses given in URL hostnames:

acl IPaddr dstdom_regex [0-9]$

This works because Squid requires URL hostnames to be fully qualified. Since none of the global top-level domains end with a digit, this ACL matches only IP addresses, which do end with a number.

url_regex

You can use the url_regex ACL to match any part of a requested URL, including the transfer protocol and origin server hostname. For example, this ACL matches MP3 files requested from FTP servers:

acl FTPMP3 url_regex -i ^ftp://.*\.mp3$

urlpath_regex

The urlpath_regex ACL is very similar to url_regex, except that the transfer protocol and hostname aren’t included in the comparison. This makes certain types of checks much easier. For example, let’s say you need to deny requests with sex in the URL, but still possibly allow requests that have sex in their hostname:

acl Sex urlpath_regex sex

As another example, let’s say you want to provide special treatment for cgi-bin requests. You can catch some of them with this ACL:

acl CGI1 urlpath_regex ^/cgi-bin

Of course, CGI programs aren’t necessarily kept under /cgi-bin/, so you’d probably want to write additional ACLs to catch the others.

browser

Most HTTP requests include a User-Agent header. The value of this header is typically something strange like:

Mozilla/4.51 [en] (X11; I; Linux 2.2.5-15 i686)

The browser ACL performs regular expression matching on the value of the User-Agent header. For example, to deny requests that don’t come from a Mozilla browser, you can use:

acl Mozilla browser Mozilla
http_access deny !Mozilla

Before using the browser ACL, be sure that you fully understand the User-Agent strings your cache receives. Some user-agents lie about their identity. Even Squid has a feature to rewrite User-agent headers in requests that it forwards. With browsers such as Opera and KDE’s Konqueror, users can send different user-agent strings to different origin servers or omit them altogether.

req_mime_type

The req_mime_type ACL refers to the Content-Type header of the client’s HTTP request. Content-Type headers usually appear only in requests with message bodies. POST and PUT requests might include the header, but GET requests don’t. You might be able to use the req_mime_type ACL to detect certain file uploads and some types of HTTP tunneling requests.

The req_mime_type ACL values are regular expressions. To catch audio file types, you can use an ACL like this:

acl AuidoFileUploads req_mime_type -i ^audio/

rep_mime_type

The rep_mime_type ACL refers to the Content-Type header of the origin server’s HTTP response. It is really only meaningful when used in an http_reply_access rule. All other access control forms are based on aspects of the client’s request. This one is based on the response.

If you want to try blocking Java code with Squid, you might use some access rules like this:

acl JavaDownload rep_mime_type application/x-java
http_reply_access deny JavaDownload

ident_regex

You saw the ident ACL earlier in this section. The ident_regex simply allows you to use regular expressions, instead of exact string matching on usernames returned by the ident protocol. For example, this ACL matches usernames that contain a digit:

acl NumberInName ident_regex [0-9]

proxy_auth_regex

As with ident, the proxy_auth_regex ACL allows you to use regular expressions on proxy authentication usernames. For example, this ACL matches admin, administrator, and administrators:

acl Admins proxy_auth_regex -i ^admin

external_acl_type type-name [options] format 
               helper-command

/usr/local/squid/libexec/my-acl-prog.pl -X -5 /usr/local/squid/etc/datafile

external_acl_type MyAclType cache=100 %LOGIN %{User-Agent} \
    /usr/local/squid/libexec/my-acl-prog.pl -X -5 \
    /usr/local/squid/share/usernames \
    /usr/local/squid/share/useragents

acl acl-name external type-name [args ...]

acl MyAcl external MyAclType

acl name "filename"

acl Foo BadClients 1.2.3.4 1.2.3.5 1.2.3.6 1.2.3.7 1.2.3.9 ...

acl Foo BadClients "/usr/local/squid/etc/BadClients"

1.2.3.4
1.2.3.5
1.2.3.6
1.2.3.7
1.2.3.9
...

acl Simpsons ident Maggie Lisa Bart Marge Homer

acl Schmever port 80-90 101 103 107 1 2 3 9999

As I mentioned earlier, ACL elements are the first step in building access controls. The second step is the access control rules, where you combine elements to allow or deny certain actions. You’ve already seen some http_access rules in the preceding examples. Squid has a number of other access control lists:

Squid has a number of additional configuration directives that use ACL elements. Some of these used to be global settings that were modified to use ACLs to provide more flexibility.

               access_list allow|deny [!]ACLname ...

http_access allow MyClients
http_access deny !Safe_Ports
http_access allow GameSites AfterHours

access_list allow ACL1 ACL2 ACL3

acl A method http
acl B port 8080
http_access deny A B

http_access deny B A

acl A src 1.2.3.4
acl B src 5.6.7.8
http_access allow A B

acl A src 1.2.3.4 5.6.7.8
http_access allow A

acl Bob ident bob
http_access allow Bob

acl All src 0/0
acl Bob ident bob
http_access allow Bob
http_access deny All

acl All src 0/0
acl Net1 src 1.2.3.0/24
acl Net2 src 1.2.4.0/24
acl Net3 src 1.2.5.0/24
acl Net4 src 1.2.6.0/24
acl WorkingHours time 08:00-17:00

http_access allow Net1 WorkingHours
http_access allow Net2 WorkingHours
http_access allow Net3 WorkingHours
http_access allow Net4
http_access deny All

http_access allow Net4
http_access deny !WorkingHours
http_access allow Net1
http_access allow Net2
http_access allow Net3
http_access deny All

acl CacheManager proto cache_object
acl Localhost src 127.0.0.1
http_access deny CacheManager !Localhost

acl CacheManager proto cache_object
acl Localhost src 127.0.0.1
acl MyNet 10.0.0.0/24
acl All src 0/0
http_access deny CacheManager !Localhost
http_access allow MyNet
http_access deny All

http_access allow CacheManager localhost
http_access deny CacheManager
http_access allow MyNet
http_access deny All

Because access controls can be complicated, this section contains a few examples. They demonstrate some of the common uses for access controls. You should be able to adapt them to your particular needs.

acl All src 0/0
acl MyNetwork src 172.16.5.0/24 172.16.6.0/24

http_access allow MyNetwork
http_access deny All

acl All src 0/0
acl MyNetwork src 172.16.5.0/24 172.16.6.0/24
acl ProblemHost src 172.16.5.9

http_access deny ProblemHost
http_access allow MyNetwork
http_access deny All

acl PornSites url_regex "/usr/local/squid/etc/pornlist"
http_access deny PornSites

acl NotWorkRelated dstdomain "/usr/local/squid/etc/not-work-related-sites"
acl WorkingHours time D 08:00-17:30

http_access deny !WorkingHours NotWorkRelated

acl All src 0/0
acl MyNetwork src 172.16.5.0/24 172.16.6.0/24
acl NotWorkRelated dstdomain "/usr/local/squid/etc/not-work-related-sites"
acl WorkingHours time D 08:00-17:30

http_access deny !WorkingHours NotWorkRelated
http_access allow MyNetwork
http_access deny All

http_access deny !MyNetwork
http_access deny !WorkingHours NotWorkRelated
http_access Allow All

acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443 563     # https, snews
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 1025-65535  # unregistered ports

acl SSL_ports port 443 563
acl CONNECT method CONNECT

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
<additional http_access lines as necessary...>

acl Dangerous_ports 7 9 19 22 23 25 53 109 110 119
acl SSL_ports port 443 563
acl CONNECT method CONNECT

http_access deny Dangerous_ports
http_access deny CONNECT !SSL_ports
<additional http_access lines as necessary...>

auth_param basic program /usr/local/squid/libexec/ncsa_auth
    /usr/local/squid/etc/passwd

acl Authenticated proxy_auth REQUIRED
acl Admins proxy_auth Pat Jean Chris
acl Porn dstdomain "/usr/local/squid/etc/porn.domains"
acl All src 0/0

http_access allow Admins
http_access deny Porn
http_access allow Authenticated
http_access deny All

alc All src 0/0
acl OurUsers src 172.16.5.0/24
acl ChildCache src 192.168.1.1
acl SiblingCache src 192.168.3.3

http_access allow OurUsers
http_access allow ChildCache
http_access allow SiblingCache
http_access deny All

miss_access deny SiblingCache

icp_access allow ChildCache
icp_access allow SiblingCache
icp_access deny All

acl IPForHostname dstdom_regex ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$
http_access deny IPForHostname

acl All src 0/0
acl Movies rep_mime_type video/mpeg
acl MP3s rep_mime_type audio/mpeg
http_reply_access deny Movies
http_reply_access deny MP3s
http_reply_access allow All

acl LocalServers dst 172.17.1.0/24

acl LocalServers dstdomain .example.com

no_cache deny LocalServers

As your access control configuration becomes longer, it also becomes more complicated. I strongly encourage you to test your access controls before turning them loose on a production server. Of course, the first thing you should do is make sure that Squid can correctly parse your configuration file. Use the -k parse feature for this:

% squid -k parse

To further test your access controls, you may need to set up a fake Squid installation. One easy way to do that is compile another copy of the Squid source code with a different $prefix location. For example:

% tar xzvf squid-2.5.STABLE4.tar.gz
% cd squid-2.5.STABLE4
% ./configure --prefix=/tmp/squid ...
% make && make install

After installing, you need to edit the new squid.conf file and change a few directives. Change http_port if Squid is already running on the default port. For simple testing, create a single, small cache directory like this:

cache_dir ufs /tmp/squid/cache 100 4 4

If you don’t want to recompile Squid again, you can also just create a new configuration file. The drawback to this approach is that you’ll need to set all the log-file pathnames to the temporary location so that you don’t overwrite the real files.

You can easily test some access controls with the squidclient program. For example, if you have a rule that depends on the origin server hostname (dstdomain ACL), or some part of the URL (url_regex or urlpath_regex), simply enter a URI that you would expect to be allowed or denied:

% squidclient -p 4128 http://blocked.host.name/blah/blah

or:

% squidclient -p 4128 http://some.host.name/blocked.ext

Certain aspects of the request are harder to control. If you have src ACLs that block requests from outside your network, you may need to actually test them from an external host. Testing time ACLs may be difficult unless you can change the clock on your system or stay awake long enough.

You can use squidclient’s -H option to set arbitrary request headers. For example, use the following if you need to test a browser ACL.

% squidclient -p 4128 http://www.host.name/blah \
      -H 'User-Agent: Mozilla/5.0 (compatible; Konqueror/3)\r\n'

For more complicated request, with many headers, you may want to use the technique described in Section 16.4.

You might also consider developing a routine cron job that checks your ACLs for expected behavior and reports any anomalies. Here is a sample shell script to get you started:

#!/bin/sh
set -e

TESTHOST="www.squid-cache.org"

# make sure Squid is not proxying dangerous ports
#
ST=`squidclient 'http://$TESTHOST:25/' | head -1 | awk '{print $2}'`
if test "$ST" != 403 ; then
        echo "Squid did not block HTTP request to port 25"
fi


# make sure Squid requires user authentication
#
ST=`squidclient 'http://$TESTHOST/' | head -1 | awk '{print $2}'`
if test "$ST" != 407 ; then
        echo "Squid allowed request without proxy authentication"
fi


# make sure Squid denies requests from foreign IP addresses
# elsewhere we already created an alias 192.168.1.1 on one of
# the system interfaces
#
EXT_ADDR=192.168.1.1
ST=`squidclient -l $EXT_ADDR 'http://$TESTHOST/' | head -1 | awk '{print $2}'`
if test "$ST" != 403 ; then
        echo "Squid allowed request from external address $EXT_ADDR"
fi

exit 0

The cache_dir directive is one of the most important in squid.conf. It tells Squid where and how to store cache files on disk. The cache_dir directive takes the following arguments:

cache_dir scheme 
            directory 
            size 
            L1 
            L2 [options]

# newfs /dev/da1d
# newfs /dev/da2d
# mount /dev/da1d /cache0
# mount /dev/da2d /cache1

cache_dir ufs /cache0 7000 16 256
cache_dir ufs /cache1 7000 16 256

# mkdir /var/squidcache

cache_dir ufs /var/squidcache 7000 16 256

% df -k
Filesystem  1K-blocks     Used    Avail Capacity  Mounted on
/dev/da1d     3037766        8  2794737     0%    /cache0
/dev/da2d     3037766        8  2794737     0%    /cache1

% df -ik
Filesystem  1K-blocks     Used    Avail Capacity iused   ifree  %iused  Mounted on
/dev/ad0s1a    197951    57114   125001    31%    1413   52345     3%   /
/dev/ad0s1f   5004533  2352120  2252051    51%  129175 1084263    11%   /usr
/dev/ad0s1e    396895     6786   358358     2%     205   99633     0%   /var
/dev/da0d     8533292  7222148   628481    92%  430894  539184    44%   /cache1
/dev/da1d     8533292  7181645   668984    91%  430272  539806    44%   /cache2
/dev/da2d     8533292  7198600   652029    92%  434726  535352    45%   /cache3
/dev/da3d     8533292  7208948   641681    92%  427866  542212    44%   /cache4

L1 and L2

For the ufs, aufs, and diskd schemes, Squid creates a two-level directory tree underneath the cache directory. The L1 and L2 arguments specify the number of first- and second-level directories. The defaults are 16 and 256, respectively. Figure 7-1 shows the filesystem structure.

Figure 7-1. The cache directory structure for ufs-based storage schemes

Some people think that Squid performs better, or worse, depending on the particular values for L1 and L2. It seems to make sense, intuitively, that small directories can be searched faster than large ones. Thus, L1 and L2 should probably be large enough so that each L2 directory has no more than a few hundred files.

For example, let’s say you have a cache directory that stores about 7000 MB. Given a mean file size of 10 KB, you can store about 700,000 files in this cache_dir. With 16 L1 and 256 L2 directories, there are 4096 total second-level directories. 700,000 ÷ 4096 leaves about 170 files in each second-level directory.

The process of creating swap directories with squid -z, goes faster for smaller values of L1 and L2. Thus, if your cache size is really small, you may want to reduce the number of L1 and L2 directories.

Squid assigns each cache object a unique file number. This is a 32-bit integer that uniquely identifies files on disk. Squid uses a relatively simple algorithm for turning file numbers into pathnames. The algorithm uses L1 and L2 as parameters. Thus, if you change L1 and L2, you change the mapping from file number to pathname. Changing these parameters for a nonempty cache_dir makes the existing files inaccessible. You should never change L1 and L2 after the cache directory has become active.

Squid allocates file numbers within a cache directory sequentially. The file number-to-pathname algorithm (e.g., storeUfsDirFullPath( )) is written so that each group of L2 files go into the same second-level directory. Squid does this to take advantage of locality of reference. This algorithm increases the probability that an HTML file and its embedded images are stored in the same second-level directory. Some people expect Squid to spread cache files evenly among the second-level directories. However, when the cache is initially filling, you’ll find that only the first few directories contain any files. For example:

% cd /cache0; du -k
2164    ./00/00
2146    ./00/01
2689    ./00/02
1974    ./00/03
2201    ./00/04
2463    ./00/05
2724    ./00/06
3174    ./00/07
1144    ./00/08
1       ./00/09
1       ./00/0A
1       ./00/0B
...

This is perfectly normal and nothing to worry about.

cache_dir ufs /cache0 7000 16 256 read-only

cache_dir ufs /cache0 7000 16 256 max-size=1048576

The cache_swap_low and cache_swap_high directives control the replacement of objects stored on disk. Their values are a percentage of the maximum cache size, which comes from the sum of all cache_dir sizes. For example:

cache_swap_low 90
cache_swap_high 95

As long as the total disk usage is below cache_swap_low, Squid doesn’t remove cached objects. As the cache size increases, Squid becomes more aggressive about removing objects. Under steady-state conditions, you should find that disk usage stays relatively close to the cache_swap_low value. You can see the current disk usage by requesting the storedir page from the cache manager (see Section 14.2.1.39).

Note that changing cache_swap_high probably won’t have a big impact on Squid’s disk usage. In earlier versions of Squid, this parameter played a more important role; now, however, it doesn’t.

You can control both the maximum and minimum size of cached objects. Responses larger than maximum_object_size aren’t stored on disk. They are still proxied, however. The logic behind this directive is that you don’t want a really big response to take up space better utilized by many small responses. The syntax is as follows:

maximum_object_size size-specification

Here are some examples:

maximum_object_size 100 KB
maximum_object_size 1 MB
maximum_object_size 12382 bytes
maximum_object_size 2 GB

Squid checks the response size in two different ways. If the reply includes a Content-Length header, Squid compares its value to the maximum_object_size value. If the content length is the larger of the two numbers, the object becomes immediately uncachable and never consumes any disk space.

Unfortunately, not every response has a Content-Length header. In this case, Squid writes the response to disk as data comes in from the origin server. Squid checks the object size again only when the response is complete. Thus, if the object’s size reaches the maximum_object_size limit, it continues consuming disk space. Squid increments the total cache size only when it is done reading a response.

In other words, the active, or in-transit, objects don’t contribute to the cache size value Squid maintains internally. This is good because it means Squid won’t remove other objects in the cache, unless the object remains cachable and then contributes to the total cache size. However, it is also bad because Squid may run out of free disk space if the reply is very large. To reduce the chance of this happening, you should also use the reply_body_max_size directive. A response that reaches the reply_body_max_size limit is cut off immediately.

Squid also has a minimum_object_size directive. It allows you to place a lower limit on the size of cached objects. Responses smaller than this size aren’t stored on disk or in memory. Note that this size is compared to the response’s content length (i.e., the size of the reply body), which excludes the HTTP headers.

When Squid wants to store a cachable response on disk, it calls a function that selects one of the cache directories. It then opens a disk file for writing on the selected directory. If, for some reason, the open( ) call fails, the response isn’t stored. In this case, Squid doesn’t try opening a disk file on one of the other cache directories.

Squid has two of these cache_dir selection algorithms. The default algorithm is called least-load; the alternative is round-robin.

The least-load algorithm, as the name implies, selects that cache directory that currently has the smallest workload. The notion of load depends on the underlying storage scheme. For the aufs, coss, and diskd schemes, the load is related to the number of pending operations. For ufs, the load is constant. For cases in which all cache_dirs have equal load, the algorithm uses free space and maximum object sizes as tie-breakers.

The selection algorithm also takes into account the max-size and read-only options. Squid skips a cache directory if it knows the object size is larger than the limit. It also always skips any read-only directories.

The round-robin algorithm also uses load measurements. It always selects the next cache directory in the list (subject to max-size and read-only), as long as its load is less than 100%.

Under some circumstances, Squid may fail to select a cache directory. This can happen if all cache_dirs are overloaded or if all have max-size limits less than the size of the object. In this case, Squid simply doesn’t write the object to disk. You can use the cache manager to track the number of times Squid fails to select a cache directory. View the store_io page (see Section 14.2.1.41), and find the create.select_fail line.

The cache_replacement_policy directive controls the replacement policy for Squid’s disk cache. Version 2.5 offers three different replacement policies: least recently used (LRU), greedy dual-size frequency (GDSF), and least frequently used with dynamic aging (LFUDA).

LRU is the default policy, not only for Squid, but for most other caching products as well. LRU is a popular choice because it is almost trivial to implement and provides very good performance. On 32-bit systems, LRU uses slightly less memory than the others (12 versus 16 bytes per object). On 64-bit systems, all policies use 24 bytes per object.

Over the years, many researchers have proposed alternatives to LRU. These other policies are typically designed to optimize a specific characteristic of the cache, such as response time, hit ratio, or byte hit ratio. While the research almost always shows an improvement, the results can be misleading. Some of the studies use unrealistically small cache sizes. Other studies show that as cache size increases, the choice of replacement policy becomes less important.

If you want to use the GDSF or LFUDA policies, you must pass the —enable-removal-policies option to the ./configure script (see Section 3.4.1). Martin Arlitt and John Dilley of HP Labs wrote the GDSF and LFUDA implementation for Squid. You can read their paper online at http://www.hpl.hp.com/techreports/1999/HPL-1999-69.html. My O’Reilly book, Web Caching, also talks about these algorithms.

The cache_replacement_policy directive is unique in an important way. Unlike most of the other squid.conf directives, the location of this one is significant. The cache_replacment_policy value is actually used when Squid parses a cache_dir directive. You can change the replacement policy for a cache_dir by setting the replacement policy beforehand. For example:

cache_replacement_policy lru
cache_dir ufs /cache0 2000 16 32
cache_dir ufs /cache1 2000 16 32
cache_replacement_policy heap GDSF
cache_dir ufs /cache2 2000 16 32
cache_dir ufs /cache3 2000 16 32

In this case, the first two cache directories use LRU replacement, and the second two use GDSF. This characteristic of the replacement_policy directive is important to keep in mind if you ever decide to use the config option of the cache manager (see Section 14.2.1.7). The cache manager outputs only one (the last) replacement policy value, and places it before all of the cache directories. For example, you may have these lines in squid.conf:

cache_replacement_policy heap GDSF
cache_dir ufs /tmp/cache1 10 4 4
cache_replacement_policy lru
cache_dir ufs /tmp/cache2 10 4 4

but when you select config from the cache manager, you get:

cache_replacement_policy lru
cache_dir ufs /tmp/cache1 10 4 4
cache_dir ufs /tmp/cache2 10 4 4

As you can see, the heap GDSF setting for the first cache directory has been lost.

At some point you may find it necessary to manually remove one or more objects from Squid’s cache. This might happen if:

Some of these problems can be solved by forcing a reload in a web browser. However, this doesn’t always work. For example, some browsers display certain content types externally by launching another program; that program probably doesn’t have a reload button or even know about caches.

You can always use the squidclient program to reload a cached object if necessary. Simply insert the -r option before the URI:

% squidclient -r http://www.lrrr.org/junk >/tmp/foo

If you happen to have a refresh_pattern directive with the ignore-reload option set, you and your users may be unable to force a validation of the cached response. In that case, you’ll be better off purging the offending object or objects.

acl AdminBoxes src 127.0.0.1 172.16.0.1 192.168.0.1
acl Purge method PURGE
http_access allow AdminBoxes Purge
http_access deny Purge

% squidclient -m PURGE http://www.lrrr.org/junk

PURGE http://www.lrrr.org/junk HTTP/1.0
Accept: */*

% awk '{print $7}' /usr/local/squid/var/logs/access.log \
        | grep www.example.com \
        | xargs -n 1 squidclient -m PURGE

# echo '' > /usr/local/squid/var/cache/swap.state

# squid -k shutdown
# cd /usr/local/squid/var
# mv cache oldcache
# mkdir cache
# chown nobody:nobody cache
# squid -z
# squid -s
# rm -rf oldcache &

The refresh_pattern directive controls the disk cache only indirectly. It helps Squid decide whether or not a given request can be a cache hit or must be treated as a miss. Liberal settings increase your cache hit ratio but also increase the chance that users receive a stale response. Conservative settings, on the other hand, decrease hit ratios and stale responses.

You can put any number of refresh_pattern lines in the configuration file. Squid searches them in order for a regular expression match. When Squid finds a match, it uses the corresponding values to determine whether a cached response is fresh or stale. The refresh_pattern syntax is as follows:

refresh_pattern [-i] regexp 
            min 
            percent 
            max [options]

For example:

refresh_pattern -i \.jpg$ 30 50% 4320 reload-into-ims
refresh_pattern -i \.png$ 30 50% 4320 reload-into-ims
refresh_pattern -i \.htm$ 0 20% 1440
refresh_pattern -i \.html$ 0 20% 1440
refresh_pattern -i . 5 25% 2880

The regexp parameter is a regular expression that is normally case-sensitive. You can make them case-insensitive with the -i option. Squid checks the refresh_pattern lines in order; it stops searching when one of the regular expression patterns matches the URI.

The min parameter is some number of minutes. It is, essentially, a lower bound on stale responses. A response can’t be stale unless its time in the cache exceeds the minimum value. Similarly, max is an upper limit on fresh responses. A response can’t be fresh unless its time in the cache is less than the maximum time.

Responses that fall between the minimum and maximum are subject to Squid’s last-modified factor (LM-factor) algorithm. For such responses, Squid calculates the response age and the LM-factor and compares it to the percent value. The response age is simply the amount of time passed since the origin server generated, or last validated, the response. The resource age is the difference between the Last-Modified and Date headers. The LM-factor is the ratio of the response age to the resource age.

Figure 7-2 demonstrates the LM-factor algorithm. Squid caches an object that is 3 hours old (based on the Date and Last-Modified headers). With an LM-factor value of 50%, the response will be fresh for the next 1.5 hours, after which the object expires and is considered stale. If a user requests the cached object during the fresh period, Squid returns an unvalidated cache hit. For a request that occurs during the stale period, Squid forwards a validation request to the origin server.

Figure 7-2. Calculating expiration times based on LM-factor

It’s important to understand the order that Squid checks the various values. Here is a simplified description of Squid’s refresh_pattern algorithm:

The refresh_pattern directive also has a handful of options that cause Squid to disobey the HTTP protocol specification. They are as follows:

Web caches such as Squid don’t usually come right out and tell you when disk I/O is becoming a bottleneck. Instead, response time and/or hit ratio degrade as load increases. The tricky thing is that response time and hit ratio may be changing for other reasons, such as increased network latency and changes in client request patterns.

Perhaps the best way to explore the performance limits of your cache is with a benchmark, such as Web Polygraph. The good thing about a benchmark is that you can fully control the environment and eliminate many unknowns. You can also repeat the same experiment with different cache configurations. Unfortunately, benchmarking often takes a lot of time and requires spare systems that aren’t already being used.

If you have the resources to benchmark Squid, begin with a standard caching workload. As you increase the load, at some point you should see a significant increase in response time and/or a decrease in hit ratio. Once you observe this performance degradation, run the experiment again but with disk caching disabled. You can configure Squid never to cache any response (with the null storage scheme, see Section 8.7). Alternatively, you can configure the workload to have 100% uncachable responses. If the average response time is significantly better without caching, you can be relatively certain that disk I/O is a bottleneck at that level of throughput.

If you’re like most people, you have neither the time nor resources to benchmark Squid. In this case, you can examine Squid’s runtime statistics to look for disk I/O bottlenecks. The cache manager General Runtime Information page (see Chapter 14) gives you median response times for both cache hits and misses:

Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.39928  0.35832
        Cache Misses:          0.42149  0.39928
        Cache Hits:            0.12783  0.11465
        Near Hits:             0.37825  0.39928
        Not-Modified Replies:  0.07825  0.07409

For a healthy Squid cache, hits are significantly faster than misses. Your median hit response time should usually be 0.5 seconds or less. I strongly recommend that you use SNMP or another network monitoring tool to collect periodic measurements from your Squid caches (see Chapter 14). A significant (factor of two) increase in median hit response time is a good indication that you have a disk I/O bottleneck.

If you believe your production cache is suffering in this manner, you can test your theory with the same technique mentioned previously. Configure Squid not to cache any responses, thus avoiding all disk I/O. Then closely observe the cache miss response time. If it goes down, your theory is probably correct.

Once you’ve convinced yourself that disk throughput is limiting Squid’s performance, you can try a number of things to improve it. Some of these require recompiling Squid, while others are relatively simple steps you can take to tune the Unix filesystems.

First of all, you should never use RAID for Squid cache directories. In my experience, RAID always degrades filesystem performance for Squid. It is much better to have a number of separate filesystems, each dedicated to a single disk drive.

I have found four simple ways to improve UFS performance for Squid. Some of these are specific to certain operating systems, such as BSD and Linux, and may not be available on your platform:

If you’re like me, you’re probably wondering what the difference is between the async option and soft updates. One important difference is that soft update code has been designed to maintain filesystem consistency in the event of a system crash, while the async option has not. This might lead you to conclude that async performs better than soft updates. However, as I show in Appendix D, the opposite is true.

Previously, I mentioned that UFS performance, especially writing, depends on the amount of free space. Disk writes for empty filesystems are much faster than for full ones. This is one reason behind UFS’s minfree parameter and space/time optimization tradeoffs. If your cache disks are full and Squid’s performance seems bad, try reducing the cache_dir capacity values so that more free space is available. Of course, this reduction in cache size also decreases your hit ratio, but the response time improvement may be worth it. If you’re buying the components for a new Squid cache, consider getting much larger disks than you need and using only half the space.

Some operating systems support filesystems other than UFS (or ext2fs). Journaling filesystems are a common alternative. The primary difference between UFS and journaling filesystems is in the way that they handle updates. With UFS, updates are made in-place. For example, when you change a file and save it to disk, the new data replaces the old data. When you remove a file, UFS updates the directory directly.

A journaling filesystem, on the other hand, writes updates to a separate journal, or log file. You can typically select whether to journal file changes, metadata changes, or both. A background process reads the journal during idle moments and applies the actual changes. Journaling filesystems typically recover much faster from crashes than UFS. After a crash, the filesystem simply reads the journal and commits all the outstanding changes.

The primary drawback of journaling filesystems is that they require additional disk writes. Changes are first written to the log and later to the actual files and/or directories. This is particularly relevant for web caches because they tend to have more disk writes than reads in the first place.

Journaling filesystems are available for a number of operating systems. On Linux, you can choose from ext3fs, reiserfs, XFS, and others. XFS is also available for SGI/IRIX, where it was originally developed. Solaris users can use the Veritas filesystem product. The TRU64 (formerly Digital Unix) Advanced Filesystem (advfs) supports journaling.

You can use a journaling filesystem without making any changes to Squid’s configuration. Simply create and mount the filesystem as described in your operating system documentation. You don’t need to change the cache_dir line in squid.conf. Use a command like this to make a reiserfs filesystem on Linux:

# /sbin/mkreiserfs /dev/sda2

For XFS, use:

# mkfs -t xfs -f /dev/sda2

Note that ext3fs is simply ext2fs with journaling enabled. Use the -j option to mke2fs when creating the filesystem:

# /sbin/mke2fs -j /dev/sda2

Refer to your documentation (e.g., manpages) for other operating systems.

The aufs storage scheme has evolved out of the very first attempt to improve Squid’s disk I/O response time. The “a” stands for asynchronous I/O. The only difference between the default ufs scheme and aufs is that I/Os aren’t executed by the main Squid process. The data layout and format is the same, so you can easily switch between the two schemes without losing any cache data.

aufs uses a number of thread processes for disk I/O operations. Each time Squid needs to read, write, open, close, or remove a cache file, the I/O request is dispatched to one of the thread processes. When the thread completes the I/O, it signals the main Squid process and returns a status code. Actually, in Squid 2.5, certain file operations aren’t executed asynchronously by default. Most notably, disk writes are always performed synchronously. You can change this by setting ASYNC_WRITE to 1 in src/fs/aufs/store_asyncufs.h and recompiling.

The aufs code requires a pthreads library. This is the standard threads interface, defined by POSIX. Even though pthreads is available on many Unix systems, I often encounter compatibility problems and differences. The aufs storage system seems to run well only on Linux and Solaris. Even though the code compiles, you may encounter serious problem on other operating systems.

To use aufs, you must add a special ./configure option:

% ./configure --enable-storeio=aufs,ufs

Strictly speaking, you don’t really need to specify ufs in the list of storeio modules. However, you might as well because if you try aufs and don’t like it, you’ll be able to fall back to the plain ufs storage scheme.

You can also use the —with-aio-threads= N option if you like. If you omit it, Squid automatically calculates the number of threads to use based on the number of aufs cache_dirs. Table 8-1 shows the default number of threads for up to six cache directories.

After you compile aufs support into Squid, you can specify it on a cache_dir line in squid.conf:

cache_dir aufs /cache0 4096 16 256

After starting Squid with aufs enabled, make sure everything still works correctly. You may want to run tail -f store.log for a while to make sure that objects are being swapped out to disk. You should also run tail -f cache.log and look for any new errors or warnings.

2003/09/29 13:42:47| squidaio_queue_request: WARNING - Disk I/O overloading

% squidclient mgr:squidaio_counts
...

ASYNC IO Counters:
Operation       # Requests
open             15318822
close            15318813
cancel           15318813
write                   0
read             19237139
stat                    0
unlink            2484325
check_callback  311678364
queue                   0

diskd (short for disk daemons) is similar to aufs in that disk I/Os are executed by external processes. Unlike aufs, however, diskd doesn’t use threads. Instead, inter-process communication occurs via message queues and shared memory.

Message queues are a standard feature of modern Unix operating systems. They were invented many years ago in AT&T’s Unix System V, Release 1. The messages passed between processes on these queues are relatively small: 32-40 bytes. Each diskd process uses one queue for receiving requests from Squid and another queue for transmitting results back.

cache_dir diskd /cache0 7000 16 256 Q2=50

cache_dir diskd /cache0 7000 16 256 Q1=60 Q2=50

% ./configure --enable-storeio=ufs,diskd

# System V message queues and tunable parameters
options         SYSVMSG         # include support for message queues
options         MSGMNB=8192     # max characters per message queue
options         MSGMNI=40       # max number of message queue identifiers
options         MSGSEG=512      # max number of message segments per queue
options         MSGSSZ=64       # size of a message segment MUST be power of 2
options         MSGTQL=2048     # max number of messages in the system
options         SYSVSHM
options         SHMSEG=16       # max shared mem segments per process
options         SHMMNI=32       # max shared mem segments in the system
options         SHMMAX=2097152  # max size of a shared mem segment
options         SHMALL=4096     # max size of all shared memory (pages)

kernel.msgmnb=8192
kernel.msgmni=40
kernel.msgmax=8192
kernel.shmall=2097152
kernel.shmmni=32
kernel.shmmax=16777216

set msgsys:msginfo_msgmax=8192
set msgsys:msginfo_msgmnb=8192
set msgsys:msginfo_msgmni=40
set msgsys:msginfo_msgssz=64
set msgsys:msginfo_msgtql=2048
set shmsys:shminfo_shmmax=2097152
set shmsys:shminfo_shmmni=32
set shmsys:shminfo_shmseg=16

ipc:
          msg-max = 2048
          msg-mni = 40
          msg-tql = 2048
          msg-mnb = 8192
          shm-seg = 16
          shm-mni = 32
          shm-max = 2097152
          shm-max = 4096

# sysconfigdb -a -f ipc.stanza

cache_dir diskd /cache0 7000 16 256 Q1=72 Q2=64
cache_dir diskd /cache1 7000 16 256 Q1=72 Q2=64
...

2003/09/29 01:30:11| storeDiskdSend: msgsnd: (35) Resource temporarily unavailable

% squidclient mgr:diskd
...
sent_count: 755627
recv_count: 755627
max_away: 14
max_shmuse: 14
open_fail_queue_len: 0
block_queue_len: 0

             OPS SUCCESS    FAIL
   open   51534   51530       4
 create   67232   67232       0
  close  118762  118762       0
 unlink   56527   56526       1
   read   98157   98153       0
  write  363415  363415       0

The Cyclic Object Storage Scheme (coss) is an attempt to develop a custom filesystem for Squid. With the ufs-based schemes, the primary performance bottleneck comes from the need to execute so many open( ) and unlink( ) system calls. Because each cached response is stored in a separate disk file, Squid is always opening, closing, and removing files.

coss, on the other hand, uses one big file to store all responses. In this sense, it is a small, custom filesystem specifically for Squid. coss implements many of the functions normally handled by the underlying filesystem, such as allocating space for new data and remembering where there is free space.

Unfortunately, coss is still a little rough around the edges. Development of coss has been proceeding slowly over the last couple of years. Nonetheless, I’ll describe it here in case you feel adventurous.

% ./configure --enable-storeio=ufs,coss ...

cache_dir coss /cache0/coss 7000 max-size=1000000
cache_dir coss /cache1/coss 7000 max-size=1000000
cache_dir coss /cache2/coss 7000 max-size=1000000
cache_dir coss /cache3/coss 7000 max-size=1000000
cache_dir coss /cache4/coss 7000 max-size=1000000

cache_dir coss /cache0/coss 30000 max-size=1000000 block-size=2048

2003/09/29 18:51:42|  /usr/local/squid/var/cache: (21) Is a directory
FATAL: storeCossDirInit: Failed to open a coss file.

2003/09/29 18:53:38| /usr/local/squid/var/cache/coss/swap.state:
        (2) No such file or directory
FATAL: storeCossDirOpenSwapLog: Failed to open swap log.

options         VFS_AIO

cache_dir coss /cache0/coss0 1900 max-size=1000000 block-size=128
cache_dir coss /cache0/coss1 1900 max-size=1000000 block-size=128
cache_dir coss /cache0/coss2 1900 max-size=1000000 block-size=128
cache_dir coss /cache0/coss3 1900 max-size=1000000 block-size=128

Squid has a fifth storage scheme called null. As the name implies, this is more of a nonstorage scheme. Files that are “written” to a null cache_dir aren’t actually written to disk.

Most people won’t have any reason to use the null storage system. It’s primarily useful if you want to entirely disable Squid’s disk cache.^[4] You can’t simply remove all cache_dir lines from squid.conf because then Squid adds a default ufs cache_dir. The null storage system is also sometimes useful for testing and benchmarking Squid. Since the filesystem is typically the performance bottleneck, using the null storage scheme gives you an upper limit of Squid’s performance on your hardware.

To use this scheme you must first specify it on the —enable-storeio list when running ./configure:

% ./configure --enable-storeio=ufs,null ...

You can then create a cache_dir of type null in squid.conf:

cache_dir /tmp null

It may seem odd that you need to specify a directory for the null storage scheme. However, Squid uses the directory name as a cache_dir identifier. For example, you’ll see it in the cache manager output (see Section 14.2.1.39).

Squid’s storage scheme choices may seem a little overwhelming and confusing. Is aufs better than diskd? Does my system support aufs or coss? Will I lose my data if I use one of these fancy schemes? Is it okay to mix-and-match storage schemes?

First of all, if your Squid is lightly used (say, less than five requests per second), the default ufs storage scheme should be sufficient. You probably won’t see a noticeable performance improvement from the other schemes at this low request rate.

If you are trying to decide which scheme to try, your operating system may be a determining factor. For example, aufs runs well on Linux and Solaris but seems to have problems on other systems. The coss code uses functions that aren’t available on certain operating systems (e.g., NetBSD) at this time.

It seems to me that higher-performing storage schemes are also more susceptible to data loss in the event of a system crash. This is the tradeoff for better performance. For many people, however, cached data is of relatively low value. If Squid’s cache becomes corrupted due to a crash, you may find it easier to simply newfs the disk partition and let the cache fill back up from scratch. If you find it difficult or expensive to replace the contents of Squid’s cache, you probably want to use one of the slow, but reliable, filesystems and storage schemes.

Squid certainly allows you to use different filesystems and storage schemes for each cache_dir. In practice, however, this is uncommon. You’ll probably have fewer hassles if all cache directories are approximately the same size and use the same storage scheme.

Interception caching involves some network trickery, so it is helpful for you to understand what happens between the client and Squid. I’ll use Figure 9-1 and the following sample tcpdump output to explain how the packets are intercepted as they flow through your network.

Figure 9-1. How HTTP interception works

You don’t want your switch/router to intercept the connections that Squid makes to origin servers. If that happens, Squid ends up talking to itself and can’t satisfy any cache misses. The best way to avoid forwarding loops like this is to make sure that your users and Squid connect to separate interfaces on the switch/router. Whenever feasible, you should apply the interception rules to specific interfaces. Obviously, you should not enable interception on the interface that Squid uses.

Many organizations find interception caching attractive because they can’t, or would rather not, configure all their user’s web browsers. It’s probably easier to perform a little network trickery on a single switch or router than it is to configure hundreds or thousands of workstations. As with many choices we face, interception caching is really a tradeoff. It brings both benefits and drawbacks. It may make your life easier, or more difficult.

The obvious benefit of interception caching is that all HTTP requests leaving your network automatically go through Squid. You don’t need to worry about configuring any browsers or that users might disable their proxy settings. Interception caching puts you, the network administrator, in control of the HTTP traffic. You can change, add, or remove Squid caches from service without significantly interrupting your users’ web surfing.

Most of the disadvantages surrounding HTTP interception are because this technique violates the TCP/IP standards. These protocols mandate that routers (and switches) forward TCP/IP packets to the host specified by the destination IP address. Diverting the packets to a caching proxy breaks the rules. The proxy accepts diverted connections under false pretense. User agents are tricked into believing they have established a TCP connection with the origin server.

This confusion causes a serious problem with older versions of Microsoft’s Internet Explorer. The browser’s Reload button is the easiest way to refresh an HTML page. When Explorer is configured to use a caching proxy, a reload request includes a Cache-Control: no-cache header to force a cache miss (or validation) and ensure that the response is up to date. Explorer omits this header when not explicitly configured for proxying. With interception caching, Explorer thinks it is connecting to the origin server anyway, and there is no need to send this header. Squid can’t tell that the user pressed the Reload button in this case and may not validate the cached response. Squid’s ie_refresh provides a partial workaround for this bug (see Appendix A). According to Microsoft, this problem has been corrected in Explorer Version 5.5, Service Pack 1.^[1]

For similar reasons, you can’t use HTTP proxy authentication in combination with interception caching. Because the client is unaware of the proxy, it doesn’t send the necessary Proxy-Authorization header. Additionally, the 407 (Proxy Authorization Required) response code is inappropriate because the response should look like it came from the origin server, which would never send such a reply.

You also can’t use RFC 1413 ident lookups (see Section 6.1.2.11) with interception. Squid can’t bind a new TCP socket to the necessary IP address. The operating system cheats when forwarding the intercepted connection to Squid. However, it can’t cheat when Squid wants to bind a new TCP socket to the foreign IP address. The address that it wants to bind to isn’t really local, so the bind system call fails.

Interception caching is also incompatible with IP filtering designed to prevent address spoofing (See also RFC 2267: Network Ingress Filtering: Defeating Denial of Service Attacks Which Employ IP Source Address Spoofing). Consider the network shown in Figure 9-2. The router has two LAN interfaces: lan0 and lan1. The network administrator uses packet filters on the router to make sure that the internal hosts don’t transmit packets with spoofed source addresses. The router forwards only packets with source addresses corresponding to the connected networks. The packet filter rules might look something like this:

# lan0
allow ip from 172.16.1.0/24 to any via lan0
deny ip from any to any via lan0
# lan1
allow ip from 10.0.0.0/16 to any via lan1
deny ip from any to any via lan1

Figure 9-2. Interception caching breaks address spoofing filters

Now consider what happens when the router and Squid box on lan1 are configured to intercept HTTP connections coming from lan0. Squid pretends to be the origin server, which means that the TCP packets carrying response data from Squid back to the users have spoofed source addresses. These lan0 filter rules cause the router to deny these packets. To make interception caching work, the network administrator must remove the lan0 rules. This, in turn, leaves the network vulnerable to being the source of denial-of-service attacks.

As I explained in the previous section, clients must make DNS queries before opening a connection. This may be undesirable or difficult in certain firewall environments. A host whose HTTP traffic you want to intercept must be able to query the DNS. Clients that know they are using a proxy (due to manual configuration or proxy auto-configuration, for example) don’t usually try to resolve hostnames. Instead, they simply forward full URLs to Squid, and it becomes Squid’s job to look up origin server IP addresses.

Another little problem is that Squid accepts connections for any destination IP address. Consider, for example, a web site that still has a DNS entry even though the site and server have been taken down. Squid accepts the TCP connection for this bogus site. The client believes the site is up and running, because it’s connection is established. When Squid fails to connect to the origin server, it is forced to return an error message.

In case it’s not clear, HTTP interception can be tricky and difficult to get working the first time. A number of different components must all work together and be correctly configured. Furthermore, it can be difficult to recreate the entire configuration from memory. I strongly encourage you to set up a test environment before attempting this on a production system. Once you get it all working, be sure to document every little step.

Now that you know all the ins and outs of interception caching, let’s see how to actually make it work. We’ll start by configuring the network devices that will be intercepting your HTTP connections.

Inline Squid

In this configuration, you don’t need a switch or network router to intercept HTTP connections. Instead, Squid runs on a Unix system that is also your router (or perhaps bridge), as shown in Figure 9-3.

Figure 9-3. A system that combines routing and caching can easily intercept HTTP traffic

This configuration essentially skips the first three steps shown in Section 9.1. The Squid host already receives the HTTP connection packets because it is the router for your network. If you are taking this approach, feel free to skip ahead to Section 9.4.

Layer Four Switches

Many organizations use layer four switches specifically for their HTTP interception support. These products offer additional features as well, such as health checks and load balancing. I’ll only cover interception here. For information on health checks and load balancing, see O’Reilly’s Server Load Balancing and Load Balancing Servers, Firewalls, and Caches (John Wiley & Sons). The following subsections contain working-example configurations for a number of products and techniques.

Alteon/Nortel

The following configuration is from an ACEswitch 180 and Alteon’s WebOS 8.0.21. The network setup is shown in Figure 9-4.

Figure 9-4. Sample network for layer four switch interception, for Alteon and Foundry examples

Clients are connected to port 1, the connection to the Internet is via port 2, and Squid is on port 3. The following lines are the relevant output of a /cfg/dump command on the switch. You don’t necessarily need to type all of these lines. Furthermore, some of the commands may have changed for newer versions of Alteon’s software. Note that Alteon calls this feature Web Cache Redirection (WCR). Here’s the process, step by step:

1. First, you must give the Alteon switch an IP address. This seems necessary so that the switch can perform health checks with Squid:

   /cfg/ip/if 1
           ena
           addr 172.16.102.1
           mask 255.255.255.0
           broad 172.16.102.255

2. Alteon’s WCR is a feature of its Server Load Balancing (SLB) configuration. Thus, you need to enable SLB features on the switch with this command:

   /cfg/slb
           on

3. Next, you define a real server with Squid’s IP address:

   /cfg/slb/real 1
           ena
           rip 172.16.102.66

4. You must also define a group and make the real server a member:

   /cfg/slb/group 1
           health tcp
           add 1

5. The next step is to define two filters. The first filter matches HTTP connections—TCP packets with destination port 80—and redirects them to a server in group 1. The second filter matches all other packets and forwards them normally:

   /cfg/slb/filt 1
           ena
           action redir
           sip any
           smask 0.0.0.0
           dip any
           dmask 0.0.0.0
           proto tcp
           sport any
           dport http
           group 1
           rport 0
   /cfg/slb/filt 224
           ena
           action allow
           sip any
           smask 0.0.0.0
           dip any
           dmask 0.0.0.0
           proto any

6. The final step is to configure specific switch ports for SLB. On port 1, you enable client processing (this is where the clients connect), and add the two filters. On the second port you need only configure it for servers (i.e., the upstream Internet connection):

   /cfg/slb/port 1
           client ena
           filt ena
           add 1
           add 224
   /cfg/slb/port 2
           server ena

To verify that HTTP interception is configured and working correctly, you can use the commands under the /stats/slb and /info/slb menus. The /info/slb/dump command is a quick and easy way to see the entire SLB configuration:

>> Main# /info/slb/dump
Real server state:
  1: 172.16.102.66, 00:c0:4f:23:d7:05, vlan 1, port 3, health 3, up

Virtual server state:

Redirect filter state:
  1: dport http, rport 0, group 1, health tcp, backup none
    real servers:
      1: 172.16.102.66, backup none, up

Port state:
  1: 0.0.0.0, client
     filt  enabled, filters: 1 224
  2: 0.0.0.0, server
     filt disabled, filters: empty
  3: 0.0.0.0
     filt disabled, filters: empty

In this output, notice that the switch says Squid is reachable via port 3 and that the health checks show Squid is up. You can also see that filter 1 has been applied to port 1, where the clients connect. In the Port state section, port 1 is designated as a place where clients connect, and port 2 is similarly marked as a server port.

The /stats/slb/real command shows a handful of statistics for the real server (i.e., Squid):

>> Main# /stats/slb/real 1
------------------------------------------------------------------
Real server 1 stats:
Health check failures:                0
Current sessions:                    41
Total sessions:                     760
Highest sessions:                    55
Octets:                               0

Most of the statistics relate to the number of sessions (i.e., TCP connections). The Total sessions counter should increase if you execute the command again.

Lastly, the /stats/slb/group command shows almost the same information:

>> Main# /stats/slb/group 1
------------------------------------------------------------------
Real server group 1 stats:
                      Current      Total  Highest
Real IP address      Sessions   Sessions Sessions           Octets
---- --------------- -------- ---------- --------  ---------------
   1 172.16.102.66         65       2004       90                0
---- --------------- -------- ---------- --------  ---------------
                           65       2004       90                0

This output would be more interesting if there was more than one real server in the group.

Foundry

The configuration in the following example comes from a ServerIron XL, running software version 07.0.07T12. As before, clients are on port 1, the Internet link is on port 2, and Squid is on port 3. However, that matters less for this particular configuration because you can enable HTTP interception globally. Foundry’s name for interception caching is Transparent Cache Switching (TCS). Refer back to Figure 9-4 for this example.

The first step is to give the switch an IP address so it can perform health checks:

ip address 172.16.102.1 255.255.255.0

Foundry allows you to enable or disable TCS on particular ports. However, for the sake of simplicity, let’s enable it globally:

ip policy 1 cache tcp http global

In this line, cache is a keyword that corresponds to the TCS feature. The next line defines a web cache. I’ve given it the name squid1 and told the switch its IP address:

server cache-name squid1 172.16.102.66

The final step is to add the web cache to a cache group:

server cache-group 1
 cache-name squid1

If you’re having problems getting the Foundry switch to divert connections, have a look at the show cache-group output:

ServerIron#show cache-group

Cache-group 1 has 1 members Admin-status = Enabled Active = 0
Hash_info: Dest_mask = 255.255.255.0 Src_mask = 0.0.0.0

Cache Server Name                Admin-status Hash-distribution
squid1                           6            3

HTTP Traffic  From <-> to  Web-Caches

Name: squid1          IP: 172.16.102.66    State: 6   Groups =   1

                                   Host->Web-cache       Web-cache->Host
           State   CurConn TotConn Packets    Octets     Packets    Octets
Client     active  441     12390   188871     15976623   156962     154750098
Web-Server active  193     11664   150722     151828731  175796     15853612
Total              634     24054   339593     167805354  332758     170603710

Some of this output is cryptic, but you can tell interception is working by repeating the command and watching the counters increase.

The show server real command provides almost the same information:

ServerIron#show server real squid1
Real Servers Info

Name : squid1                                       Mac-addr: 00c0.4f23.d705
IP:172.16.102.66   Range:1    State:Active          Wt:1     Max-conn:1000000
Src-nat (cfg:op):(off:off)    Dest-nat (cfg:op):(off:off)
squid1 is a TRANSPARENT CACHE in groups   1
Remote server   : No          Dynamic : No      Server-resets:0
Mem:server: 02009eae Mem:mac: 045a3714

Port    State    Ms CurConn TotConn Rx-pkts  Tx-pkts  Rx-octet   Tx-octet   Reas
----    -----    -- ------- ------- -------  -------  --------   --------   ----
http    active   0  855     29557   379793   471713   373508204  39425322   0
default active   0  627     28335   425106   366016   38408994   368496301  0

Server  Total       1482    57892   804899   837729   411917198  407921623  0

Finally, you can use the show logging command to see if the switch believes Squid is up or down:

ServerIron#show logging
...
00d00h11m51s:N:L4 server 172.16.102.66 squid1 port 80 is up
00d00h11m49s:N:L4 server 172.16.102.66 squid1 port 80 is down
00d00h10m21s:N:L4 server 172.16.102.66 squid1 port 80 is up
00d00h10m21s:N:L4 server 172.16.102.66 squid1 is up

Note that the ServerIron thinks the server is running on port 80. As you’ll see later, my examples have Squid running on port 3128. The packet filtering rules actually change the packet’s destination port from 80 to 3128. This has some interesting consequences for health checks, which I address later in Section 9.3.2.5.

Extreme Networks

In this example, the hardware is a Summit1i, and the software is Version 6.1.3b11. Once again, the clients are on port 1, the Internet link is on port 2, and Squid is on port 3. The network configuration is shown in Figure 9-5.

Figure 9-5. Sample network for intercepting with a router, for the Extreme and Cisco policy routing examples

The Extreme switch can intercept HTTP connections only for packets that it routes between subnets. In other words, if you use the Extreme switch in layer two mode (with a single VLAN), you can’t divert traffic to Squid. To make HTTP interception work, you must configure separate VLANs for users, Squid, and the Internet:

configure Default delete port 1-8

create vlan Users
configure Users ip 172.16.102.1 255.255.255.192
configure Users add port 1

create vlan Internet
configure Internet ip 172.16.102.129 255.255.255.192
configure Internet add port 2

create vlan Squid
configure Squid ip 172.16.102.65 255.255.255.192
configure Squid add port 3

The next step is to enable and configure routing in the switch:

enable ipforwarding
configure iproute add default 172.16.102.130

Lastly, you configure the switch to redirect HTTP connections to Squid:

create flow-redirect http tcp destination any ip-port 80 source any
configure http add next-hop 172.16.102.66

Cisco Arrowpoint

The following configuration is based on notes from an old test I ran. However, I don’t have access to an arrowpoint switch now and can’t verify that these lines are correct.

circuit VLAN1
  ip address 172.16.102.1 255.255.255.0

service pxy1
  type transparent-cache
  ip address 172.16.102.66
  port 80
  protocol tcp
  active

owner foo
  content bar
    add service pxy1
    protocol tcp
    port 80
    active

A comment on HTTP servers and health checks

I’ve set up these examples so that the router/switch forwards packets without changing the destination TCP port. The packet filtering rules that I’ll cover in Section 9.4 change the destination port. An interesting problem arises when you also run an HTTP server on the Squid box.

To run an HTTP server on port 80 while running Squid on port 3128, your packet filter configuration must have a special rule that accepts TCP connections for the HTTP server. Otherwise, the connection gets diverted to Squid. The special rule is simple to construct. If the destination port is 80, and the destination address is the server’s, accept the packet normally. All the intercepted packets have foreign destination addresses, so they won’t match the special rule.

However, when the router/switch makes an HTTP health check, it connects to the server’s IP address. Thus, the health-check packet matches the special rule and isn’t diverted to Squid. The router/switch is checking the health of the wrong server. If the HTTP server is down, but Squid is up (or vice versa), the health check will be wrong.

If you find yourself in this situation, you have a few options:

• Don’t run an HTTP server on the Squid host.

• Add a specific packet filtering rule that diverts TCP health check connections from the router/switch to Squid.

• Configure your router/switch to change the destination port to 3128.

• Disable layer four health checks.

access-list 110 deny tcp host 172.16.102.66 any eq www
access-list 110 permit tcp any any eq www

access-list 110 permit tcp 10.102.0.0 0.0.255.255 any eq www

route-map proxy-redirect permit 10
 match ip address 110
 set ip next-hop 172.16.102.66

interface Ethernet0/0
 ip policy route-map proxy-redirect

router#show route-map proxy-redirect
route-map proxy-redirect, permit, sequence 10
  Match clauses:
    ip address (access-lists): 110
  Set clauses:
    ip next-hop 172.16.102.66
  Policy routing matches: 730 packets, 64649 bytes

ip wccp version 1
ip wccp web-cache

interface Ethernet0/1
 ip address 172.16.102.129 255.255.255.192
 ip wccp web-cache redirect out

! don't re-intercept connections coming from Squid:
access-list 112 deny   tcp host 172.16.102.66 any eq www

! don't intercept this broken web site
access-list 112 deny   tcp any 192.16.8.7 255.255.255.255 eq www

! allow other HTTP traffic
access-list 110 permit tcp any any eq www

ip wccp web-cache redirect-list 112

router#show ip wccp web-cache
Global WCCP information:
    Router information:
        Router Identifier:                   172.16.102.129
        Protocol Version:                    1.0

    Service Identifier: web-cache
        Number of Cache Engines:             1
        Number of routers:                   1
        Total Packets Redirected:            1424
        Redirect access-list:                -none-
        Total Packets Denied Redirect:       0
        Total Packets Unassigned:            0
        Group access-list:                   -none-
        Total Messages Denied to Group:      0
        Total Authentication failures:       0

router#show ip wccp web-cache detail
WCCP Cache-Engine information:
        IP Address:            172.16.102.66
        Protocol Version:      0.4
        State:                 Usable
        Initial Hash Info:     00000000000000000000000000000000
                               00000000000000000000000000000000
        Assigned Hash Info:    FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
                               FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
        Hash Allotment:        256 (100.00%)
        Packets Redirected:    1424
        Connect Time:          00:17:40

You must enable certain networking features in your operating system to make interception caching work. First, you need to enable IP packet forwarding. This allows the operating system to receive packets with foreign destination addresses. Second, you must enable and configure optional code in the kernel that redirects the foreign packets to Squid.

net.ipv4.ip_forward = 1

o  General setup
     Networking support (CONFIG_NET=y)
     Sysctl support (CONFIG_SYSCTL=y)
o  Networking options
     Network packet filtering (CONFIG_NETFILTER=y)
     TCP/IP networking (CONFIG_INET=y)
     Netfilter Configuration
       Connection tracking (CONFIG_IP_NF_CONNTRACK=y)
       IP tables support (CONFIG_IP_NF_IPTABLES=y)
       Full NAT (CONFIG_IP_NF_NAT=y)
       REDIRECT target support (CONFIG_IP_NF_TARGET_REDIRECT=y)
o  File systems
     /proc filesystem support (CONFIG_PROC_FS=y)

o Networking options
    Fast switching (CONFIG_NET_FASTROUTE=n)

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128

iptables -t nat -I PREROUTING -i eth0 -p tcp -d 172.16.102.66 --dport 80 -j ACCEPT

/sbin/service iptables save

% gcc -Wall -D_ _KERNEL_ _ -I/usr/src/linux-2.4.7-10/include  \
  -DMODULE -DMODVERSIONS -DEXPORT_SYMBAB \
  -include /usr/src/linux-2.4.7-10/include/linux/modversions.h \
  -O2 -c ip_wccp.c

# insmod ip_wccp.o

# iptables -A INPUT -p gre -s 172.16.102.65 -j ACCEPT
# iptables -A INPUT -p gre -j DROP

net.inet.ip.forwarding=1

options          IPFIREWALL
options          IPFIREWALL_FORWARD

/sbin/ipfw add allow tcp from 172.16.102.66 to any out
/sbin/ipfw add allow tcp from any 80 to any out
/sbin/ipfw add fwd 127.0.0.1,3128 tcp from any to any 80 in
/sbin/ipfw add allow tcp from any 80 to 172.16.102.66 in

/sbin/ipfw add allow tcp from any to 172.16.102.66 80 in

firewall_enable="YES"

pseudo-device   gre

# ifconfig gre0 create
# ifconfig gre0 172.16.102.66 172.16.102.65 netmask 255.255.255.255 up
# ifconfig gre0 tunnel 172.16.102.66 172.16.102.65
# route delete 172.16.102.65

/sbin/ipfw add allow gre from 172.16.102.65 to 172.16.102.66

net.inet.ip.forwarding=1

rdr inet proto tcp from any to any port = www -> 127.0.0.1 port 3128
pass out proto tcp from 172.16.102.66 to any
pass out proto tcp from any port = 80 to any
pass in proto tcp from any port = 80 to 172.16.102.66

pf=YES

net.inet.gre.allow=1
net.inet.gre.wccp=1

# ifconfig gre0 172.16.102.66 172.16.102.65 netmask 255.255.255.255 up
# ifconfig gre0 tunnel 172.16.102.66 172.16.102.65
# route delete 172.16.102.65

pass in proto gre from 172.16.102.65 to 172.16.102.66

net.inet.ip.forwarding=1

rdr fxp0 0/0 port 80 -> 172.16.102.66 port 3128 tcp

If you are using Linux 2.4 and iptables, you should probably use the —enable-linux-netfilter option when you run (or re-run) ./configure. It enables some Linux-specific code so that Squid can find the IP address of the origin server from where the request was originally sent. Squid normally gets the origin server name (and/or address) from the Host header. The —enable-linux-netfilter feature is necessary only for requests that don’t have a Host header. Statistics show that almost all requests have the Host header, so you may actually be able to get by without the —enable-linux-netfilter option.

If you are using the IPFilter package (with NetBSD, Solaris, and others), you should use the —enable-ipf-transparent option for the same reason. On OpenBSD, you should use the —enable-pf-transparent option. Each time you run ./configure you must recompile Squid, as described in Section 3.8.

After you get the ./configure options figured out, and Squid recompiled, you can edit squid.conf. As a starting point, make sure the following directives are defined with the given values:

httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_uses_host_header on
httpd_accel_with_proxy on
httpd_accel_single_host off

The httpd_accel_host directive is the key. It instructs Squid to accept HTTP requests with partial URIs. The httpd_accel_uses_host_header directive is enabled so that Squid uses the Host header to reconstruct full URIs. The virtual keyword instructs Squid to put the origin server’s IP address in the URL when the Host header is absent.

The httpd_accel_with_proxy directive controls whether or not Squid accepts both HTTP server (partial URI) requests, and proxy (full URI) requests. It should probably be enabled for interception caching. Squid may still work if httpd_accel_with_proxy is disabled as long as none of your clients are explicitly configured for Squid as a proxy.

The httpd_accel_single_host directive is normally disabled, but it was enabled by default in some earlier versions of Squid. I’ve listed it here to make sure that it is disabled for interception caching.

If you are intercepting more than just port 80, you may want to set httpd_accel_port to 0. See Appendix A for more information.

If you’re not using WCCP, you should be ready to start sending intercepted traffic to Squid. Give it a try by surfing the Web with your browser or by making some test requests with squidclient. If you are using WCCP, there is just one more step that you must complete.

wccp_router 172.16.102.65
wccp_version 4

wccp_version 3

HTTP interception is complicated because many different devices must all work correctly together. To help you track down problems, here’s a trouble-shooting check list:

If you suspect this problem, use the squidclient program to make some simple HTTP requests. For example, this command makes an HTTP request directly to the origin server:

% /usr/local/squid/bin/squidclient -p 80 -h slashdot.org /

If this command succeeds, you should see a bunch of ugly HTML from the Slashdot site on your screen. You can then try the same request through Squid:

% /usr/local/squid/bin/squidclient -r -p 3128 -h 127.0.0.1 http://slashdot.org/

Again, you should see some HTML on your screen. If not check for error messages in cache.log. If you see forwarding loop errors, you need to reconfigure your router/switch so that it allows Squid’s outgoing connections to pass without being intercepted.

Caching hierarchy is the name generally given to a collection of caches (or proxies) that forward requests to one another. We say that the members of the hierarchy are neighbors or peers.

Neighbor caches have either a parent or sibling relationship. Topologically, parent caches are one level up in the hierarchy, while siblings are on the same level. The real difference is that parents can forward cache misses for their children. Siblings, on the other hand, aren’t allowed to forward cache misses. This means that, before sending a request to a sibling, the originator should know that it will be a cache hit. Intercache protocols like ICP, HTCP, and Cache Digests can predict cache hits in neighbors. CARP, however, can’t.

Sometimes, cache hierarchies aren’t really hierarchical. Consider, for example, a group of five sibling caches. Because there are no parents or children, there is no sense of up or down. In this case, you could call it a cache mesh, or even an array, instead of a hierarchy.

A neighbor cache improves performance by providing some extra fraction of requests as cache hits. In other words, some of the requests that are misses in your cache may be hits in the neighbor cache. If your cache can download these neighbor hits faster than from the origin server, the hierarchy should improve performance overall. The downside is that neighbor caches usually provide only a small percentage of requests as hits. About 5%, or maybe 10% if you’re lucky, of your requests that are cache misses will be hits in a neighbor. In some cases, this small benefit doesn’t justify the hassle of joining a hierarchy. In other cases, such as networks with poor or overutilized connectivity, hierarchies definitely improve performance for end users.

If you use Squid inside a firewalled network, you may need to configure the firewall proxy as a parent. In this case, Squid forwards every request to the firewall because it can’t connect directly to outside origin servers. If you have some origin servers inside the firewall, you can instruct Squid to connect to them directly.

You can also use a hierarchy to send web traffic in different directions. This is sometimes called application-layer routing, or more recently, content routing. Consider, for example, a large organization with two Internet connections. Perhaps the second connection costs less, or has higher latency, than the other. This organization may want to use the second connection for low-priority traffic, such as downloading binaries, audio and video files, or other kinds of large transfers. Or, perhaps they want to send all HTTP traffic over one link, and non-HTTP traffic over the other. Or, perhaps certain users’ traffic should go through the low-priority connection, while premium customers get to use the more expensive link. You can accomplish any of these scenarios with a hierarchy of caching proxies.

Trust is one of the most important issues for the members of a cache hierarchy. You must trust your neighbors to serve correct, unmodified responses. You must trust them with sensitive information, such as the URIs requested by your users. You must trust that they maintain secure and up-to-date systems to minimize the chances of unauthorized access and denials of service.

Another problem with hierarchies is the way that they normally propagate errors. When a neighbor cache experiences an error, such as an unreachable server, it generates an HTML page that explains the error and its origin. Your users may become confused if they get errors from neighbor caches outside the immediate organization. If the problem persists, they’ll have a hard time finding an administrator who can help them.

Sibling relationships are subject to special problem, known as false hits. This occurs when Squid sends a request to a sibling, believing it will be a cache hit, but the sibling is unable to satisfy the request without contacting the origin server. False hits happen in a number of circumstances, but usually with a low probability. Furthermore, Squid and other HTTP proxies have features for automatically retrying such requests so that the user isn’t even aware of the problem.

A forwarding loop is another problem sometimes seen in cache hierarchies. It occurs when Squid forwards a request somewhere, but that request comes back to Squid again, as shown in Figure 10-1.

Figure 10-1. A forwarding loop

Forwarding loops typically happen when two caches consider each other parents. If you have such an arrangement, make sure that you use the cache_peer_access directive to prevent loops. For example, if the neighbor’s IP address is 192.168.1.1, the following lines ensure Squid won’t cause a forwarding loop:

acl FromNeighbor src 192.168.1.1
cache_peer_access the.neighbor.name deny FromNeighbor

Forwarding loops can also occur with HTTP interception, especially if the interception device is on the path between Squid and an origin server.

Squid detects forwarding loops by looking for its own hostname in the Via header. You may actually get false forwarding loops if two cooperating caches have the same hostname. The unique_hostname directive is useful in this situation. Note that if the Via header is filtered out (e.g., with headers_access), Squid can’t detect forwarding loops.

The cache_peer directive defines your neighbor caches and tells Squid how to communicate with them:

cache_peer hostname 
            type 
            http-port 
            icp-port [options]

The first argument is the neighbor’s hostname, or IP address. You can safely use hostnames here because Squid doesn’t block while resolving them. In fact, Squid periodically resolves the hostname in case the IP address changes while Squid is running. Neighbor hostnames must be unique: you can’t have two neighbors with the same name, even if they have different ports.

The second argument specifies the type of neighbor cache. The choices are: parent, sibling, or multicast. Parent and sibling are straightforward. I’ll talk about multicast in Section 10.6.3.

The third argument is the neighbor’s HTTP port number. It should correspond to the neighbor’s http_port (or equivalent) setting. You must always specify a nonzero HTTP port number.

The fourth argument specifies either the ICP or HTCP port number. By default, Squid uses ICP to query other caches. That is, Squid sends ICP queries to the neighbor on the port given here. If you add the htcp option, Squid sends HTCP queries to this port instead. The default ICP port is 3130, and the default HTCP port is 4827. Make sure that you change the port number if you add the htcp option. Setting this port number to zero disables both ICP and HTCP. However, you should instead (or also) use the no-query option to disable these protocols.

2003/09/29 01:13:46| Detected DEAD Sibling: bo2.us.ircache.net/3128/3130

2003/09/29 01:13:49| Detected REVIVED Sibling: bo2.us.ircache.net/3128/3130

This request could not be forwarded to the origin server or to any
parent caches.  The most likely cause for this error is that:

  * The cache administrator does not allow this cache to make
    direct connections to origin servers, and
  * All configured parent caches are currently unreachable.

neighbor_type_domain neighbor.host.name 
               relationship [!]domain ...

cache_peer squid.uk.web-cache.net sibling 3128 3130
neighbor_type_domain squid.uk.web-cache.net parent .uk

Many people who use hierarchical caching need to control or limit requests that Squid sends to its neighbors. Squid has seven different directives that affect request routing: cache_peer_access, cache_peer_domain, never_direct, always_direct, hierarchy_stoplist, nonhierarchical_direct, and prefer_direct.

cache_peer A-parent.my.org parent 3128 3130
cache_peer B-parent.my.org parent 3128 3130
acl FTP proto FTP
acl HTTP proto HTTP
cache_peer_access A-parent allow FTP
cache_peer_access B-parent allow HTTP

cache_peer A-parent.my.org parent 3128 3130
acl DayTime time 07:00-18:00
cache_peer_access A-parent.my.org deny DayTime