Squid: The Definitive Guide

Chapter 16. Debugging and Troubleshooting

No matter how hard the Squid developers try to be perfect, you may encounter some problems with Squid. These problems range from misbehaving clients and servers to fatal bugs in the Squid code. In this chapter, I’ll talk about various ways you can debug these problems.

Some Squid problems may require you to turn on debugging. In most cases, you’ll want to increase the debugging levels for specific parts of the code. I’ll describe how to find out what those debugging sections are and how to change the settings. Also, I’ll talk about the importance of providing detailed debugging when reporting bugs.

Finally, you may experience fatal bugs in the Squid code. These can result in segmentation violations, aborts, assertions, and core dumps. The core dump is a useful debugging aid. With a debugger, such as gdb, you can generate a process stack trace and send it to the developers for assistance.

If you suspect you have a Squid bug, but aren’t sure, check with the squid-users mailing list or one of the other resources described in Section 1.6.

Some Common Problems

Before discussing debugging in general, I’ll mention a few specific problems that commonly arise.

“Failed to make swap directory”

Failed to make swap directory /var/spool/cache: (13) Permission denied

This happens when you run squid -z, and the Squid user ID doesn’t have write permission to the /var/spool directory. Remember that if you start Squid as root and don’t add a cache_effective_user line, Squid runs as the user nobody by default. Thus, your solution may be to simply run:

# chown nobody:nobody /var/spool

“Address already in use”

commBind: Cannot bind socket FD 10 to *:3128: Address already in use

This message appears when the bind( ) system call fails because the requested port is already opened by another application. Usually, this happens when you try to start a second instance of Squid when the first one is still running. If you see this error message, use ps to see if Squid is already running.

Squid uses the SO_REUSEADDR socket option, so that the bind( ) call should succeed even if there are some leftover sockets in the TIME_WAIT state. If you get the message, even though Squid isn’t already running, your operating system may be buggy or especially finicky. Rebooting your system is one way to get around this problem.

Another possibility to consider is that the port (e.g., 3128) is currently being used by a different application. If you suspect this, you can use the lsof program (ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof) to find which application is listening on the port. FreeBSD users can use sockstat instead.

“Could not determine fully qualified hostname”

FATAL: Could not determine fully qualified hostname.  Please set 'visible_hostname'

You’ll see this if Squid can’t figure out its own fully qualified domain name. Here is the algorithm Squid uses:

If you told Squid to bind the HTTP port to a specific interface address, Squid attempts a reverse DNS lookup of that address. If successful, the answer is used.
Squid calls the gethostname( ) function, and then attempts to resolve its IP address with gethostbyname( ). If successful, Squid uses the official hostname string returned by the latter function.

If neither technique works, Squid exits with the fatal message shown earlier. In this case, you must tell Squid its hostname with the visible_hostname directive. For example:

visible_hostname my.host.name

“DNS name lookup tests failed”

By default, Squid makes a few DNS queries before starting. This ensures that your DNS servers are reachable and functioning properly. If these tests fail, you’ll see the following message in cache.log and/or syslog:

FATAL: ipcache_init: DNS name lookup tests failed

If you use Squid on an intranet, Squid may be unable to query its standard list of hostnames. You can specify your own hostnames with the dns_testnames directive. Squid considers the DNS test successful as soon as it receives any reply.

If you want to skip the DNS tests altogether, simply use the -D command-line option when starting Squid:

% squid -D ...

“Illegal character in hostname”

urlParse: Illegal character in hostname 'super_bikes.tripod.com'

By default, Squid checks the characters in the hostname part of URLs and complains if it finds nonstandard characters. According to RFCs 1034 and 1035, names must consist of the letters A-Z, the digits 0-9, and a hyphen (-). The underscore (_) is one of the most problematic characters.

Squid validates hostnames because, in some cases, DNS resolvers behave differently with respect to illegal characters. For example:

% host super_bikes.tripod.com
super_bikes.tripod.com has address 209.202.196.70

% ping super_bikes.tripod.com
ping: cannot resolve super_bikes.tripod.com: Unknown server error

Rather than return the Unknown server error message, Squid checks the hostname first. It can then tell the user when the hostname contains illegal characters.

Some DNS resolvers do work with underscores and other nonstandard characters. If you’d prefer that Squid not check hostnames, use the —disable-hostname-checks option when running ./configure. If you want to allow underscores as the only exception, use the —enable-underscores option.

“Running out of filedescriptors”

WARNING! Your cache is running out of filedescriptors

The above message appears when Squid uses up all available file descriptors. If this happens under normal conditions, you need to increase the kernel’s file-descriptor limits and recompile Squid. See Section 3.3.1.

You might also see this message if Squid is the target of a denial-of-service attack. Someone may be intentionally, or unintentionally, sending Squid hundreds or thousands of requests at once. If this is the case, you can probably add a packet-filtering rule to block incoming TCP connections from the offending address(es). If the attack is distributed or using a spoofed source address, you’ll have a harder time stopping it.

Forwarding loops (see Section 10.2) might also consume all of Squid’s file descriptors, but only if Squid can’t detect the loop. The Via header contains the hostname of all proxies that have seen a particular request. Squid looks for its own hostname in the header, and, if found, reports the loop. If, for some reason, the Via header is filtered from outgoing or incoming HTTP requests, Squid can’t detect the loop. In this case, all file descriptors are quickly consumed by the same request going through Squid over and over.

“icmpRecv: Connection refused”

You’ll see the following message if the pinger program isn’t correctly installed:

icmpRecv: recv: (61) Connection refused

Most likely, the pinger program exits immediately because it doesn’t have permission to open a raw ICMP socket. Because the process isn’t running, Squid receives an I/O error when trying to talk to it. To alleviate the problem, go to the source directory and, as root, type:

# make install-pinger

If successful, you should find that the pinger program has the following file ownership and permission settings:

# ls -l /usr/local/squid/libexec/pinger
-rws--x--x  1 root  squid  140728 Sep 16 19:58 /usr/local/squid/libexec/pinger

Squid Becomes Slow After Running for Some Time

Most likely, Squid is competing with other processes, or with itself, for memory on your system. When the Squid process no longer fits entirely in memory, the operating system is forced to read and write areas of memory to and from the swap space. This has a drastic effect on Squid’s performance.

To validate this theory, check the Squid process size with utilities such as top and ps. Also check Squid’s own page fault counter, as described in Section 14.2.1.24. Once you’ve identified memory consumption as the problem, try the following steps to reduce Squid’s memory usage:

Reduce the value of cache_mem and read Appendix B.
Turn off memory pooling with this option:
```
memory_pools off
```
Reduce the size of the disk cache by lowering the size of one or more cache directories. See Section 7.1.

Debugging Access Controls

If you’re having no luck getting your access controls to work properly, here’s a little tip that might help. Edit your squid.conf file and set the debug_options line to this:

debug_options ALL,1 33,2

Then, reconfigure Squid:

% squid -k reconfigure

Now, Squid writes a message to cache.log for each client request and another for each reply. The messages contain the request method, URI, whether the request/reply is allowed or denied, and the name of the last ACL that matched it. For example:

2003/09/29 20:22:05| The request
    GET http://images.slashdot.org:80/topics/topicprivacy.gif is ALLOWED,
    because it matched 'localhost'
2003/09/29 20:22:05| The reply for
    GET http://images.slashdot.org/topics/topicprivacy.gif is ALLOWED,
    because it matched 'all'

Knowing the name of the ACL doesn’t always tell you the corresponding http_access line, but it gets you pretty close. If necessary, you can replicate your acl lines and give them unique names so that a given ACL name appears on only one http_access rule.

Debugging via cache.log

You already know from Section 13.1 that cache.log contains various operational messages Squid thinks are important enough to tell you about. We also refer to these as debugging messages. You can use the debug_options directive to control the verbosity of messages that appear in cache.log. By increasing the debugging levels, you’ll see more detailed messages that may help you understand what Squid is doing. For example:

debug_options ALL,1 11,3 20,3

Every debugging message in the Squid source code has two numeric attributes: a section and a level. Sections range from 0 to 100, and levels range from 0 to 10. In general, section numbers correspond to particular components of the source code. In other words, all the messages within a single source file have the same section number. In some cases, multiple files use the same debugging section. This tends to happen when a source file becomes too large and is split into smaller chunks.

The top of each source file has line that mentions the debugging section. It looks like this:

 * DEBUG: section 9     File Transfer Protocol (FTP)

I don’t expect you to look at the source files to find the section numbers. The same information appears here in Table 16-1.

Table 16-1. Debugging section numbers for the debug_options directive

Number	Description	Source file(s)
0	Client Database	client_db.c
1	Startup and Main Loop	main.c
2	Unlink Daemon	unlinkd.c
3	Configuration File Parsing	cache_cf.c
4	Error Generation	errorpage.c
5	Socket Functions	comm.c
5	Socket Functions	comm_select.c
6	Disk I/O Routines	disk.c
7	Multicast	multicast.c
8	Swap File Bitmap	filemap.c
9	File Transfer Protocol (FTP)	ftp.c
10	Gopher	gopher.c
11	Hypertext Transfer Protocol (HTTP)	http.c
12	Internet Cache Protocol	icp_v2.c
12	Internet Cache Protocol	icp_v3.c
13	High Level Memory Pool Management	mem.c
14	IP Cache	ipcache.c
15	Neighbor Routines	neighbors.c
16	Cache Manager Objects	cache_manager.c
17	Request Forwarding	forward.c
18	Cache Manager Statistics	stat.c
19	Store Memory Primitives	stmem.c
20	Storage Manager	store.c
20	Storage Manager Client-Side Interface	store_client.c
20	Storage Manager Heap-Based Replacement	repl/heap/store_heap_replacement.c
20	Storage Manager Logging Functions	store_log.c
20	Storage Manager MD5 Cache Keys	store_key_md5.c
20	Storage Manager Swapfile Metadata	store_swapmeta.c
20	Storage Manager Swapin Functions	store_swapin.c
20	Storage Manager Swapout Functions	store_swapout.c
20	Store Rebuild Routines	store_rebuild.c
21	Misc Functions	tools.c
22	Refresh Calculation	refresh.c
23	URL Parsing	url.c
24	WAIS Relay	wais.c
25	MIME Parsing	mime.c
26	Secure Sockets Layer Proxy	ssl.c
27	Cache Announcer	send-announce.c
28	Access Control	acl.c
29	Authenticator	auth/basic/auth_basic.c
29	Authenticator	auth/digest/auth_digest.c
29	Authenticator	authenticate.c
29	NTLM Authenticator	auth/ntlm/auth_ntlm.c
30	Ident (RFC 1413)	ident.c
31	Hypertext Caching Protocol	htcp.c
32	Asynchronous Disk I/O	fs/aufs/async_io.c
33	Client-Side Routines	client_side.c
34	Dnsserver Interface	dns.c
35	FQDN Cache	fqdncache.c
37	ICMP Routines	icmp.c
38	Network Measurement Database	net_db.c
39	Cache Array Routing Protocol	carp.c
40	Referer Logging	referer.c
40	User-Agent Logging	useragent.c
41	Event Processing	event.c
42	ICMP Pinger Program	pinger.c
43	AIOPS	fs/aufs/aiops.c
44	Peer Selection Algorithm	peer_select.c
45	Callback Data Registry	cbdata.c
45	Callback Data Registry	leakfinder.c
46	Access Log	access_log.c
47	Store COSS Directory Routines	fs/coss/store_dir_coss.c
47	Store Directory Routines	fs/aufs/store_dir_aufs.c
47	Store Directory Routines	fs/diskd/store_dir_diskd.c
47	Store Directory Routines	fs/null/store_null.c
47	Store Directory Routines	fs/ufs/store_dir_ufs.c
47	Store Directory Routines	store_dir.c
48	Persistent Connections	pconn.c
49	SNMP Interface	snmp_agent.c
49	SNMP Support	snmp_core.c
50	Log File Handling	logfile.c
51	File Descriptor Functions	fd.c
52	URN Parsing	urn.c
53	AS Number Handling	asn.c
54	Interprocess Communication	ipc.c
55	HTTP Header	HttpHeader.c
56	HTTP Message Body	HttpBody.c
57	HTTP Status-Line	HttpStatusLine.c
58	HTTP Reply (Response)	HttpReply.c
59	Auto-Growing Memory Buffer with printf	MemBuf.c
60	Packer: A Uniform Interface to Store Like Modules	Packer.c
61	Redirector	redirect.c
62	Generic Histogram	StatHist.c
63	Low Level Memory Pool Management	MemPool.c
64	HTTP Range Header	HttpHdrRange.c
65	HTTP Cache Control Header	HttpHdrCc.c
66	HTTP Header Tools	HttpHeaderTools.c
67	String	String.c
68	HTTP Content-Range Header	HttpHdrContRange.c
69	HTTP Header: Extension Field	HttpHdrExtField.c
70	Cache Digest	CacheDigest.c
71	Store Digest Manager	store_digest.c
72	Peer Digest Routines	peer_digest.c
73	HTTP Request	HttpRequest.c
74	HTTP Message	HttpMsg.c
75	WHOIS Protocol	whois.c
76	Internal Squid Object handling	internal.c
77	Delay Pools	delay_pools.c
78	DNS Lookups; interacts with lib/rfc1035.c	dns_internal.c
79	Squid-Side DISKD I/O Functions	fs/diskd/store_io_diskd.c
79	Storage Manager COSS Interface	fs/coss/store_io_coss.c
79	Storage Manager UFS Interface	fs/ufs/store_io_ufs.c
80	WCCP Support	wccp.c
82	External ACL	external_acl.c
83	SSL Accelerator Support	ssl_support.c
84	Helper Process Maintenance	helper.c

Debugging levels are assigned such that more important messages have smaller values and less important messages have higher values. Level is for very important messages, while level 10 is for those that are relatively unimportant. Other than that, there are no strict guidelines or requirements. Developers are generally free to choose which debugging levels are appropriate.

The debug_options directive determines which messages appear in cache.log. Its syntax is:

debug_options section,level 
            section,level ...

The default setting is ALL,1 such that Squid prints any debugging message with level 0 or 1. If you want to make even less output appear in cache.log, you can set debug_options to ALL,0.

If you want to see additional debugging for a particular component, simply add the appropriate section and level to the end of the debug_options list. For example, this line adds level 5 debugging for the FTP server-side code:

debug_options ALL,1 9,5

As with other configuration directives, you can change debug_options, then send Squid the reconfigure signal:

% squid -k reconfigure

Note that the debug_options parameters are processed sequentially, and a later value can override an earlier one. This is of particular concern if you use the ALL keyword. Consider this example:

debug_options 9,5 20,9 4,2 ALL,1

In this case, the final value overwrites all of the preceding settings because ALL,1 sets the debugging level to 1 for all sections.

Selecting appropriate debugging sections and levels is sometimes quite difficult, especially for novice Squid users. Many of the more detailed debugging messages are meaningful only to developers and those familiar with the source code. Inexperienced Squid users are likely to find many of the debugging messages meaningless and overwhelming. Furthermore, you may have difficulty isolating the debugging for a particular request or event if Squid is relatively busy. The higher debugging levels are often more useful if you can test Squid with one request at a time.

You must also be particularly careful about running Squid with high debugging levels for a long amount of time. If Squid is busy, the cache.log file grows very quickly and may eventually consume all free space on its partition. If this happens, Squid exits with a fatal message. Another concern is that performance may degrade significantly. Due to the high number of debugging messages, Squid devotes a lot of CPU resources to formatting and printing strings. It also consumes a lot of disk bandwidth writing them all to cache.log.

Core Dumps, Assertions, and Stack Traces

If you are unlucky, Squid may experience a fatal error while running. These sorts of errors come in three flavors: assertions, bus errors, and segmentation violations.

An assertion is a sanity check in the source code. It is a tool, used by developers, to make sure that some condition is always true before proceeding. If the condition is false, the program exits and creates a core file so that the developer can analyze the situation. Here is a typical example:

int some_array[100];

void
some_func(int idx)
{
        ...
        assert(idx < 100);
        some_array[idx]++;
        ...
}

Here, the assertion makes sure that the value of the array index is within the bounds of the array. It would be an error to access array elements greater than (or equal to) 100. If, somehow, the value of idx isn’t less than 100, the program prints a message like this when it runs:

assertion failed: filename.c:123: "idx < 100"

If this happens with Squid, you’ll see an “assertion failed” message in cache.log. In addition, your operating system should create a core file, which is helpful in the post-mortem analysis. I’ll explain what to do with a core file at the end of this section.

A bus error is “a fatal failure in the execution of a machine language instruction resulting from the processor detecting an anomalous condition on its bus.”^[1] They typically occur when the processor attempts an operation on a nonaligned memory address. You are, perhaps, more likely to see a bus error on a 64-bit processor system, such as the Alpha and some SPARC CPUs. Fortunately, they are easy to fix.

Segmentation violation errors are, unfortunately, more common and sometimes harder to fix. A “SEGV” usually occurs when the process tries to access an invalid memory area. It might be a NULL pointer or a memory address outside the scope of the process. They are particularly difficult to track down when the cause (the bug) and effect (the SEGV) are separated in time.

By default, Squid traps bus errors and segmentation violations, and attempts a clean shutdown when they occur. You’ll see something like this in cache.log:

FATAL: Received Bus Error...dying.
2003/09/29 23:18:01| storeDirWriteCleanLogs: Starting...

In most cases, Squid is able to write clean versions of the swap.state files. Just before exiting, Squid calls abort( ) to create a core file. The core file may help you, or other developers, track down and fix the bug.

A core file is generally more useful when it is created immediately following the error, rather than calling the clean shutdown procedure first. You can tell Squid not to trap bus errors and segmentation violations with the -C command line option:

% squid -C ...

Note that some operating systems use the filename core, while others prepend the process name (i.e., squid.core). Once you have the core file, use a debugger to get a stack trace. gdb is the GNU debugger—a companion to the GNU C compiler. If you don’t have gdb, try running dbx or adb instead. Here’s how you can use gdb to get a stack trace:

% gdb /usr/local/squid/sbin/squid /path/to/squid.core
...
Core was generated by 'squid'.
Program terminated with signal 6, Abort trap.
...

Then, type where to print the stack trace:

(gdb) where
#0  0x28168b54 in kill ( ) from /usr/lib/libc.so.4
#1  0x281aa0ce in abort ( ) from /usr/lib/libc.so.4
#2  0x80a2316 in death (sig=10) at tools.c:301
#3  0xbfbfffac in ?? ( )
#4  0x80abe0a in storeDiskdSend (mtype=4, sd=0x82101e0, id=1214000,
    sio=0x9e90a10, size=4096, offset=-1, shm_offset=0)
    at diskd/store_io_diskd.c:485
#5  0x80ab726 in storeDiskdWrite (SD=0x82101e0, sio=0x9e90a10,
    buf=0x13e94000 "...", size=4096, offset=-1, free_func=0)
    at diskd/store_io_diskd.c:251
#6  0x809d2fb in storeWrite (sio=0x9e90a10, buf=0x13e94000 "...",
    size=4096, offset=-1, free_func=0) at store_io.c:89
#7  0x80a1c2d in storeSwapOut (e=0xc5a7800) at store_swapout.c:259
#8  0x809b667 in storeAppend (e=0xc5a7800, buf=0x810f9a0 "...", len=57344)
    at store.c:533
#9  0x807873b in httpReadReply (fd=134, data=0xc343590) at http.c:642
#10 0x806492f in comm_poll (msec=10) at comm_select.c:445
#11 0x8084404 in main (argc=2, argv=0xbfbffa8c) at main.c:742
#12 0x804a465 in _start ( )

As you can see, the stack trace prints the name of each function, its arguments, and the source code filenames and line numbers. This information is extremely useful when tracking down bugs. In some cases, however, it isn’t sufficient. You might be asked to execute additional commands in the debugger, such as printing the value of a variable from within a certain function:

(gdb) frame 4
#4  0x80abe0a in storeDiskdSend (mtype=4, sd=0x82101e0, id=1214000,
    sio=0x9e90a10, size=4096, offset=-1, shm_offset=0)
    at diskd/store_io_diskd.c:485
485         x = msgsnd(diskdinfo->smsgid, &M,
 msg_snd_rcv_sz, IPC_NOWAIT);
(gdb) set print pretty
(gdb) print M
$2 = {
  mtype = 4,
  id = 1214000,
  seq_no = 7203103,
  callback_data = 0x9e90a10,
  size = 4096,
  offset = -1,
  status = -1,
  shm_offset = 0
}

After you’ve reported a bug, try to keep the core file around for a few days, in case you need additional information from it.

Can’t Find the Core File?

core files are written in the process’ current directory. By default, Squid doesn’t change its current directory at startup. Thus, your core file, if any, should be written in the directory in which Squid was started. You won’t find a core file if the filesystem doesn’t have enough free space or if the process owner doesn’t have write permission in the directory. You can use the coredump_dir directive to make Squid use a specific location—somewhere with plenty of space and sufficient permissions.

Process resource limits may also prevent the creation of a core file. One of the process limit parameters is the size of the core dump file. Usually, most systems set this to “unlimited” by default. You can check the current limit from your shell with the limits or ulimit commands. Note, however, that your shell’s limit might be different than the Squid process limit, especially when Squid is started automatically at boot time. If you suspect process limits prevent generation of a core file, try this:

csh% limit coredumpsize unlimited
csh% squid -NCd1

On FreeBSD, a sysctl parameter controls whether or not the operating system generates a core file for processes that call setuid( ) and/or setgid( ). Squid uses those functions if you start it as root. To get a core dump, then, you must tell the kernel to create the core file with this command:

# sysctl kern.sugid_coredump=1

See the sysctl.conf manpage for information on how to set the variable automatically when your system boots.

Replicating Problems

Occasionally you may encounter a certain request or origin server that seems not to work with Squid. You can use the following technique to determine if the problem lies with Squid, the client, or the origin server. The trick is to capture the HTTP request, then replay it in different ways until you identify the problem.

Capturing the HTTP request means getting more than just the URL. You also need the request method, HTTP version number, and all of the request headers. One way to capture the request is by enabling full debugging in Squid for a short time. On the Squid box, type:

% squid -kdebug

Then, go to the web browser and issue the request. Squid should receive the request almost immediately. After a few seconds, go back to the Squid box and issue the same command:

% squid -kdebug

Now your cache.log file should contain the client’s request. If your Squid is busy, the cache.log will contain a lot of requests, so you’ll have to search for it. It looks something like this:

2003/09/29 10:37:40| parseHttpRequest: Method is 'GET'
2003/09/29 10:37:40| parseHttpRequest: URI is 'http://squidbook.org/'
2003/09/29 10:37:40| parseHttpRequest: Client HTTP version 1.1.
2003/09/29 10:37:40| parseHttpRequest: req_hdr = {
User-Agent: Mozilla/5.0 (compatible; Konqueror/3)
Pragma: no-cache
Cache-control: no-cache
Accept: text/*, image/jpeg, image/png, image/*, */*
Accept-Encoding: x-gzip, gzip, identity
Accept-Charset: iso-8859-1, utf-8;q=0.5, *;q=0.5
Accept-Language: en
Host: squidbook.org

Note that Squid prints the components of the first line separately. You’ll have to manually reassemble them like this:

GET http://squidbook.org/ HTTP/1.1

Another way to capture the full request is with a utility such as netcat or socket ( http://www.jnickelsen.de/socket/). Start the socket program listening on some port, then configure the browser to use that port as the proxy address. When you make the request again, socket prints the HTTP request:

% socket -s 8080
GET http://squidbook.org/ HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; Konqueror/3)
Pragma: no-cache
Cache-control: no-cache
Accept: text/*, image/jpeg, image/png, image/*, */*
Accept-Encoding: x-gzip, gzip, identity
Accept-Charset: iso-8859-1, utf-8;q=0.5, *;q=0.5
Accept-Language: en
Host: squidbook.org

Finally, you can also use a network packet capture utility, such as tcpdump or ethereal. After capturing a few packets with tcpdump, you can then use tcpshow to view them:

# tcpdump -w tcpdump.log -c 10 -s 1500 port 80
# tcpshow -noHostNames -noPortNames < tcpdump.log | less
...
Packet 4
TIME:   08:39:29.593051 (0.000627)
LINK:   00:90:27:16:AA:75 -> 00:00:24:C0:0D:25 type=IP
  IP:   10.0.0.21 -> 206.168.0.6 hlen=20 TOS=00 dgramlen=304 id=4B29
        MF/DF=0/1 frag=0 TTL=64 proto=TCP cksum=15DC
 TCP:   port 2074 -> 80 seq=0481728885 ack=4107144217
        hlen=32 (data=252) UAPRSF=011000 wnd=57920 cksum=EB38 urg=0
DATA:   GET / HTTP/1.0.
        Host: www.ircache.net.
        Accept: text/html, text/plain, application/pdf, application/
        postscript, text/sgml, */*;q=0.01.
        Accept-Encoding: gzip, compress.
        Accept-Language: en.
        Negotiate: trans.
        User-Agent: Lynx/2.8.1rel.2 libwww-FM/2.14.
        .

Note that tcpshow prints a period where the data contains a newline character.

Once you’ve captured a request, save it to a file. Then you can replay it through Squid with netcat or socket:

% socket squidhost 3128 < request | less

If the response looks normal, the problem might be with the user-agent. Otherwise, you can change various things to isolate the problem. For example, if you see some funny-looking HTTP headers, delete them from the request and try it again. You may also find it useful to try the request directly with the origin server, instead of going through Squid. To do that, remove the http://host.name/ from the request and send it to the origin server:

% cat request
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; Konqueror/3)
Pragma: no-cache
Cache-control: no-cache
Accept: text/*, image/jpeg, image/png, image/*, */*
Accept-Encoding: x-gzip, gzip, identity
Accept-Charset: iso-8859-1, utf-8;q=0.5, *;q=0.5
Accept-Language: en
Host: squidbook.org

% socket squidbook.org 80 < request | less

When working with HTTP in this manner, you might find it useful to refer to RFC 2616 and O’Reilly’s HTTP: The Definitive Guide.

Reporting a Bug

If your Squid version is more than a few months old, you should probably update it before reporting any bugs. Chances are that others noticed the same bug, and it may already be fixed.

If you discover a legitimate bug in Squid, please enter it into the Squid bug tracking database: http://www.squid-cache.org/bugs/. This is currently a “bugzilla” database, which requires you to create an account. You will receive updates as the bug is processed by Squid developers.

If you are new at reporting bugs, please take the time to read “How to Report Bugs Effectively,” by Simon Tatham (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).

When reporting a bug, be sure to include the following information:

Squid version number. If the bug happens with more than one version, include the other versions as well.
Your operating system name and version.
Whether the bug happens every time or occasionally.
A good description of exactly what happens. Phrases such as “it doesn’t work,” and “the request fails” are essentially useless to bug fixers. Be very specific.
A stack trace in the case of an assertion, bus error, or segmentation violation.

Remember that Squid developers are generally unpaid volunteers, so be patient. Critical bugs have more priority over minor annoyances.

Exercises

Use tcpdump or ethereal to capture some real HTTP requests. Save them to a file and replay the requests through Squid. Feel free to modify or delete some of the HTTP headers.
Try to make Squid run out of file descriptors.
Run tail -f cache.log and start Squid with debug_options set to ALL,3. If that is too overwhelming, try ALL,2.
Force Squid to generate a core file by sending each of the following signals: SIGBUS, SIGSEGV, and SIGABRT. Find the core file and use gdb or another debugger to get a stack trace.