Installing Varnish

Varnish is supported on the following operating systems:

• Linux

• FreeBSD

• Solaris

You can get it to work on other UNIX-like systems (OS X, OpenBSD, NetBSD, and Windows with Cygwin), but there’s no official support for those.

The supported Linux distributions are:

• Ubuntu

• Debian

• Red Hat

• CentOS

You can easily install Varnish using the package manager of your operating system, but you can also compile Varnish from source.

apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/ubuntu/ trusty varnish-4.1" \
     >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish

apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.1"\
    >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish

yum install epel-release
rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.1.el7.rpm
yum install varnish

yum install epel-release
rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.1.el6.rpm
yum install varnish

Configuring Varnish

Now that you have Varnish installed on your system, it’s time to configure some settings so that you can start using it.

Varnish has a bunch of startup options that allow you to configure the way you interact with it. These options are located in a configuration file and assigned to the varnishd program at startup time. Here are some examples of typical startup options:

• The address and port on which Varnish processes its incoming HTTP requests

• The address and port on which the Varnish CLI runs

• The location of the VCL file that holds the caching policies

• The location of the file that holds the secret key, used to authenticate with the Varnish CLI

• The storage backend type and the size of the storage backend

• Jailing options to secure Varnish

• The address and port of the backend that Varnish will interact with

sudo service varnish reload

sudo cp /lib/systemd/system/varnish.service /etc/systemd/system

sudo systemctl daemon-reload

sudo service varnish reload

usage: varnishd [options]
    -a address[:port][,proto]    # HTTP listen address and port (default: *:80)
                                 #   address: defaults to loopback
                                 #   port: port or service (default: 80)
                                 #   proto: HTTP/1 (default), PROXY
    -b address[:port]            # backend address and port
                                 #   address: hostname or IP
                                 #   port: port or service (default: 80)
    -C                           # print VCL code compiled to C language
    -d                           # debug
    -F                           # Run in foreground
    -f file                      # VCL script
    -h kind[,hashoptions]        # Hash specification
                                 #   -h critbit [default]
                                 #   -h simple_list
                                 #   -h classic
                                 #   -h classic,<buckets>
    -i identity                  # Identity of varnish instance
    -j jail[,jailoptions]        # Jail specification
                                 #   -j unix[,user=<user>][,ccgroup=<group>]
                                 #   -j none
    -l vsl[,vsm]                 # Size of shared memory file
                                 #   vsl: space for VSL records [80m]
                                 #   vsm: space for stats counters [1m]
    -M address:port              # Reverse CLI destination
    -n dir                       # varnishd working directory
    -P file                      # PID file
    -p param=value               # set parameter
    -r param[,param...]          # make parameter read-only
    -S secret-file               # Secret file for CLI authentication
    -s [name=]kind[,options]     # Backend storage specification
                                 #   -s malloc[,<size>]
                                 #   -s file,<dir_or_file>
                                 #   -s file,<dir_or_file>,<size>
                                 #   -s file,<dir_or_file>,<size>,<granularity>
                                 #   -s persistent (experimental)
    -T address:port              # Telnet listen address and port
    -t TTL                       # Default TTL
    -V                           # version
    -W waiter                    # Waiter implementation
                                 #   -W epoll
                                 #   -W poll

DAEMON_OPTS="-a :80 \
             -a :81,PROXY \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,3g \
             -j unix,user=www-data \"

DAEMON_OPTS="-a :80 \
             -a :81,PROXY \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,3g \
             -j unix,user=www-data \
             -l 100m,10m \
             -t 60 \
             -p feature=_++esi_disable_xml_check \
             -p connect_timeout=5 \
             -p first_byte_timeout=10 \
             -p between_bytes_timeout=2"

What About TLS/SSL?

Transport Layer Security (TLS), also referred to as Secure Sockets Layer (SSL), is a set of cryptographic protocols that are used to encrypt data communication over the network. In a web context, TLS and SSL are the “S” in HTTPS. TLS ensures that the connection is secured by encrypting the communication and establishing a level of trust by issuing certificates.

During the last couple of years, TLS has become increasingly popular to the point that non-encrypted HTTP traffic will no longer be considered normal in a couple of years. Security is still a hot topic in the IT industry, and nearly every brand on the internet wants to show that they are secure and trustworthy by offering HTTPS on their sites. Even Google Search supposedly gives HTTPS websites a better page rank.

Varnish does not natively include TLS support because encryption is hard and it is not part of the project’s core business. Varnish is all about caching and leaves the crypto to the crypto experts.

The trick with TLS on Varnish is to terminate the secured connection before the traffic reaches Varnish. This means adding a TLS/SSL offloader to your setup that terminates the TLS connection and communicates over HTTP with Varnish.

The downside is that this also adds another layer of complexity to your setup and another system that can fail on you. Additionally, it’s a bit harder for the web server to determine the origin IP address. Under normal circumstances, Varnish should add the value of the X-Forwarded-For HTTP request header sent by the TLS offloader and store that value in its own X-Forwarded-For header. That way, the backend can still retrieve the origin IP.

In Varnish 4.1, PROXY protocol support was added. The PROXY protocol is a small protocol that was introduced by HAProxy, the leading open source load-balancing software. This PROXY protocol adds a small preamble to the TCP connection that contains the IP address of the original client. This information is transferred along and can be interpreted by Varnish. Varnish will use this value and automatically add it to the X-Forwarded-For header that it sends to the backend.

I wrote a detailed blog post about this, and it contains more information about both the HAProxy and the Varnish setup.

Additionally, the PROXY protocol implementation in Varnish uses this new origin IP information to set a couple of variables in VCL:

• It sets the client.ip variable to the IP address that was sent via the PROXY protocol

• It sets the server.ip variable to the IP address of the server that accepted the initial connection

• It sets the local.ip variable to the IP address of the Varnish server

• It sets the remote.ip variable to the IP address of the machine that sits in front of Varnish

HAProxy is not the only TLS offloader that supports PROXY. Varnish Software released Hitch, a TLS proxy that terminates the TLS connection and communicates over HTTP with Varnish. Whereas HAProxy is primarily a load balancer that offers TLS offloading, Hitch only does TLS offloading. HAProxy also wrote a blog post about the subject that lists a set of PROXY-protocol ready projects. Depending on your use case and whether you need load balancing in your setup, you can choose either HAProxy or a dedicated TLS proxy. Varnish Plus, the advanced version of Varnish, developed by Varnish Software, offers TLS/SSL support on both the server and the client side. The TLS/SSL proxy in Varnish Plus is tightly integrated with Varnish and helps improve website security without relying on third-party solutions.

Conclusion

Don’t let all these settings scare you—they’re just proof that Varnish is an incredibly flexible tool with lots of options and settings that can be tuned.

If you’re a sysadmin, I hope I have inspired you to try tuning some of these settings. If you’re not, just remember that Varnish can easily be installed with a package manager of your Linux distribution and hardly requires any tuning to be up and running.

At the bare minimum, have a look at the setting in “Network binding” if you want Varnish to process HTTP traffic on port 80.

Idempotence

Varnish will only cache resources that are requested through an idempotent HTTP verb, which are HTTP verbs that do not change the state of the resource. To put it simply, Varnish will only cache requests using the following methods:

• GET

• HEAD

And that makes perfect sense: if you issue a request using POST or PUT, the method itself implies that a change will happen. In that respect, caching wouldn’t make sense because you would be caching stale data right from the get-go.

So if Varnish sees a request coming in through, let’s say, POST, it will pass the request to the backend and will not cache the returned response.

For the sake of completeness, these are the HTTP verbs/methods that Varnish can handle:

• GET (can be cached)

• HEAD (can be cached)

• PUT (cannot be cached)

• POST (cannot be cached)

• TRACE (cannot be cached)

• OPTIONS (cannot be cached)

• DELETE (cannot be cached)

All other HTTP methods are considered non-RFC2616 compliant and will completely bypass the cache.

State

Now that you know about idempotence and how HTTP request methods shouldn’t change the state of the resource, let’s look at other mechanisms in HTTP that can control state. I’m not talking about global state, but more specifically about user-specific data. There are two ways to keep track of state for users:

• Authorization headers

• Cookies

Whenever Varnish sees one of these, it will pass the request off to the backend and not cache the response. This happens because when an authentication header or a cookie is sent, it implies that the data will differ for each user performing that request.

If you decide to cache the response of a request that contains an authentication header or cookie, you would be serving a response tailored to the first user that requested it. Other users will see it, too, and the response could potentially contain sensitive or irrelevant information.

But let’s face it: cookies are our main instrument to keep track of state, and websites that do not uses cookies are hard to come by. Unfortunately, the internet uses too many cookies and often for the wrong reasons.

We use cookies to establish sessions in our application. We can also use cookies to keep track of language, region, and other preferences. And then there are the tracking cookies that are used by third parties to “spy” on us.

In terms of HTTP, cookies appear both in the request and the response process. It is the backend that sets one or more cookies by issuing a Set-Cookie response header. The client receives that response and stores the cookies in its local cookie store.

As you can see in the example below, a cookie is a set of key-value pairs, delimited by an ampersand.

Set-Cookie: language=en&country=us

When a client has stored cookies for a domain, it will use a Cookie request header to send the cookies back to the server upon every subsequent request. The cookies are also sent for requests that do not require a specific state (e.g., static files).

Cookie: language=en&country=us

This two-step process is how cookies are set and announced. Just remember the difference between Cookie and Set-Cookie. The first is a request header; the second is a response header.

As mentioned, Varnish doesn’t like to cache cookies. Whenever it sees a request with a Cookie header, the request will be passed to the backend and the response will not be cached.

When a request does not contain a cookie but the response includes a Set-Cookie header, Varnish will not store the result in cache.

Expiration

HTTP has a set of mechanisms in place to decide when a cached object should be removed from cache. Objects cannot live in cache forever: you might run out of cache storage (memory or disk space) and Varnish will have to evict items using an LRU strategy to clear space. Or you might run into a situation where the data you are serving is stale and the object needs to be synchronized with a new response from the backend.

Expiration is all about setting a time-to-live. HTTP has two different kinds of response headers that it uses to indicate that:

Expires

An absolute timestamp that represents the expiration time.

Cache-control

The amount of seconds an item can live in cache before becoming stale.

Expires: Sat, 09 Sep 2017 14:30:00 GMT

Cache-control: public, max-age=3600, s-maxage=86400

Conditional Requests

Expiration is a valuable mechanism for updating the cache. It’s based on the concept of checking the freshness of an object at set intervals. These intervals are defined by the time-to-live and are processed by Varnish. The end user doesn’t really have a say in this.

After the expiration, both the headers and the payload are transmitted and stored in cache. This could be a very resource-intensive matter and a waste of bandwidth, especially if the requested data has not changed in that period of time.

Luckily, HTTP offers a way to solve this issue. Besides relying on a time-to-live, HTTP allows you to keep track of the validity of a resource. There are two separate mechanisms for that:

• The Etag response header

• The Last-Modified response header

HTTP/1.1 200 OK
Host: localhost
Etag: 7c9d70604c6061da9bb9377d3f00eb27
Content-type: text/html; charset=UTF-8

Hello world output

GET /if_none_match.php HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-None-Match: 7c9d70604c6061da9bb9377d3f00eb27

HTTP/1.0 304 Not Modified
Host: localhost
Etag: 7c9d70604c6061da9bb9377d3f00eb27

<?php
$etag = md5(__FILE__.filemtime(__FILE__));
header('Etag: ' . $etag);
if (isset($_SERVER['HTTP_IF_NONE_MATCH'])
&&  $_SERVER['HTTP_IF_NONE_MATCH'] == $etag) {
    header('HTTP/1.0 304 Not Modified');
    exit;
}
sleep(5);
?>
<h1>Etag example</h1>
<?php
echo date("Y-m-d H:i:s").'<br />';

HTTP/1.1 200 OK
Host: localhost
Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT
Content-type: text/html; charset=UTF-8

Hello world output

GET /if_last_modified.php HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT

HTTP/1.0 304 Not Modified
Host: localhost
Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT

GET /if_last_modified.php HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT

HTTP/1.1 200 OK
Host: localhost
Last-Modified: Fri, 22 Jul 2016 11:00:23 GMT
Content-type: text/html; charset=UTF-8

Some other hello world output

<?php
header('Last-Modified: ' .
gmdate('D, d M Y H:i:s', filemtime(__FILE__)) . ' GMT');
if (isset($_SERVER['HTTP_IF_MODIFIED_SINCE']) &&
    strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE']) >= filemtime(__FILE__))
{
    header('HTTP/1.0 304 Not Modified');
    exit;
}
sleep(5);
?>
<h1>Last-Modified example</h1>
<?php
echo date("Y-m-d H:i:s").'<br />';

Cache Variations

In general, an HTTP resource is public and has the same value for every consumer of the resource. If data is user-specific, it will, in theory, not be cacheable. However, there are exceptions to this rule and HTTP has a mechanism for this.

HTTP uses the Vary header to perform cache variations. The Vary header is a response header that is sent by the backend. The value of this header contains the name of a request header that should be used to vary on.

A very common example is language detection based on the Accept-Language request header. Your browser will send this header upon every request. It contains a set of languages or locales that your browser supports. Your application can then use the value of this header to determine the language of the output. If the desired language is not exposed in the URL or through a cookie, the only way to know is by using the Accept-Language header.

If no vary header is set, the cache (either the browser cache or any intermediary cache) has no way to identify the difference and stores the object based on the first request. If that first request was made in Dutch, all other users will get output in Dutch—regardless of the browser language—for the duration of the cache lifetime.

That is a genuine problem, so in this case, the application returns a Vary header containing Accept-Language as its value. Here’s an example:

The browser language is set to Dutch:

GET / HTTP/1.1
Host: localhost
Accept-Language: nl

The application sets a Vary header that instructs the cache to keep a separate version of the cached object based on the Accept-Language value of the request.

HTTP/1.1 200 OK
Host: localhost
Vary: Accept-Language

Hallo, deze pagina is in het Nederlands geschreven.

The cache knows there is a Dutch version of this resource and will store it separately, but it will still link it to the cached object of the main resource. When the next request is sent from a browser that only supports English, the cached object containing Dutch output will not be served. A new backend request will be made and the output will be stored separately.

Varnish respects the Vary header and adds variations to the cache on top of the standard identifiers. The typical identifiers for a cached object are the hostname (or the IP if no hostname was set) and the URL.

When Varnish notices a cache variation, it will create a cache object for that version. Cache variations can expire separately, but when the main object is invalidated, the variations are gone, too.

Varnish Built-In VCL Behavior

Now that we know how Varnish deals with HTTP, we can summarize how Varnish behaves right out of the box. Here’s a set of questions we can ask ourselves:

1. When is a request considered cacheable in Varnish?

2. When does Varnish completely bypass the cache?

3. How does Varnish identify an object?

4. When does Varnish cache an object?

5. What happens if an object is not stored in cache?

6. How long does Varnish cache an object?

Sounds mysterious, huh? Let me provide answers and allow me to explain how Varnish respects HTTP best practices.

Conclusion

When you’re up and running and sending your HTTP traffic through Varnish, there will be a certain behavior that will impact the cacheability of your website.

This behavior does not reflect arbitrary rules and policies that were defined by Varnish itself. Varnish respects conventional HTTP best practices that were defined in industry-wide, accepted RFCs.

Even if you don’t add any VCL code, the best practices will make sure that your website is properly cached, assuming that your code respects the best practices as well.

An additional advantage is that the cacheability of your website and the portability of the caching behavior can go beyond the scope of Varnish. You can swap out Varnish for another kind of reverse proxy, or even a CDN.

At this point you will know what a Cache-control header is and how it compares to an Expires header. You’ll have a pretty solid idea how to leverage those headers to control the cacheability of your pages. By now, you’re no stranger to Cache variations and conditional requests.

Finally and most importantly: you can only cache GET or HEAD requests, because they are idempotent. Nonidempotent requests like, for example, POST, PUT, and DELETE cannot be cached.

Hooks and Subroutines

VCL is not the kind of language where you start typing away in an empty file or within a main method; it actually restricts you and only allows you to hook into certain aspects of the Varnish execution flow. This execution flow is defined in a finite state machine.

The hooks represent specific stages of the Varnish flow. The behavior of Varnish in these stages is expressed through various built-in subroutines. You define a subroutine in your VCL file, extend the caching behavior in that subroutine, and issue a reload of that VCL file to enable that behavior.

Every subroutine has a fixed set of return statements that represent a state change in the flow.

sub remove_ga_cookies {
    # Remove any Google Analytics based cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");
}

include "custom_subroutines.vcl";

sub vcl_recv {
    call remove_ga_cookies;
}

Return Statements

Whereas the VCL subroutines represent the different states of the state machine, the return statement within each subroutine allows for state changes.

If you specify a valid return statement in a subroutine, the corresponding action will be executed and a transition to the corresponding state will happen. As mentioned before: when you don’t specify a return statement, the execution of the subroutine will continue and Varnish will fall back on the built-in VCL.

Here’s a list of valid return statements:

hash

Look the object up in cache.

pass

Pass the request off to the backend, but don’t cache the result.

pipe

Pass the request off to the backend and bypass any caching logic.

synth

Stop the execution and immediately return synthetic output. This returns statement takes an HTTP status code and a message.

purge

Evict the object and its variants from cache. The URL of the request will be used as an identifier.

fetch

Pass the request off to the backend and try to cache the response.

restart

Restart the transaction and increase the req.restarts counter until max_restarts is reached.

deliver

Send the response back to the client.

miss

Synchronously refresh the object from the backend, despite a hit.

lookup

Use the hash to look an object up in cache.

abandon

Abandon a backend request and return a HTTP 503 (backend unavailable) error.

The execution flow

In Chapter 3, I talked about the built-in VCL and in the previous section I listed a set of subroutines and return statements. It’s time to put all the pieces of the puzzle together and compose the execution flow of Varnish.

In Chapter 1, I referred to the finite state machine that Varnish uses. Let’s have a look at it and see how Varnish transitions between states and what causes these transitions.

Figure 4-1 shows a simplified flowchart that explains the execution flow.

Figure 4-1. The simplified flow of execution in Varnish

We can split up the flow into two parts:

1. Backend fetches (the gray box)

2. Request and response handling (the rest of the flowchart)

We have now reached a point where the subroutines start making sense. To summarize, let’s repeat some of the important points of the execution flow:

• Every session starts in vcl_recv.

• Cache lookups happen in vcl_hash.

• Non-cacheable requests are directly passed to the backend in vcl_pass. Responses are not cached.

• Items that were found in cache are handled by vcl_hit.

• items that were not found are handled by vcl_miss.

• Cache misses or passed requests are fetched from the backend via vcl_backend_fetch.

• Backend responses are handled by vcl_backend_response.

• When a backend fetch fails, the error is handled by vcl_backend_error.

• Valid responses that were cached, passed, or missed are delivered by vcl_deliver.

At this point you know the basic vocabulary we’ll use to refer to the different stages of the finite state machine. Now it’s time to learn about the VCL syntax and the VCL objects in order to modify HTTP requests and responses and in order to transition to other stages of the flow.

VCL Syntax

If you want to hook into the Varnish execution flow and extend the subroutines, you’d better know the syntax. Well, let’s talk syntax.

The full VCL reference manual can be found on the Varnish website.

sub vcl_recv {
    if(req.method == "PURGE" || req.method == "BAN") {
        return(purge);
    }
    if(req.method != "GET" && req.method != "HEAD") {
        return(pass);
    }
    if(req.url ~ "^/products/[0-9]+/"){
        set req.http.x-type = "product";
    }
}

sub vcl_recv {
    if(req.url == "/"){
        return(pass);
    } elseif(req.url == "/test") {
        return(synth(200,"Test succeeded"));
    } else {
        return(pass);
    }
}

sub vcl_recv {
    // Single line of out-commented VCL.
    # Another way of commenting out a single line.
    /*
        Multi-line block of commented-out VCL.
    */
}

sub vcl_recv {
    set req.http.x-test = "testing 123";
    set req.http.x-test-long = {"testing '123', or even "123" for that matter"};
    set req.http.x-test-long-newline = {"testing '123',
or even "123"
for that matter"};
}

sub vcl_recv {
    return(synth(200,"All good"));
}

sub vcl_recv {
    return(synth(200,200));
}

sub vcl_backend_response {
    if(beresp.http.set-cookie) {
        set beresp.uncacheable = true;
    }
}

sub vcl_backend_response {
    set beresp.ttl = 1h;
}

sub vcl_backend_response {
    set beresp.ttl = 1.5h;
}

sub vcl_recv {
    if(req.url ~ "^/products/[0-9]+/"){
        set req.http.x-type = "product";
    }
}

sub vcl_hash {
    if(req.http.Cookie
    ~ "language=(nl|fr|en|de|es)"){
        hash_data(regsub(req.http.Cookie,
        "^.*;? ?language=(nl|fr|en|de|es)( ?|;| ;).*$","\1"));
    }
}

sub vcl_recv {
  set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");
}

sub vcl_hash {
    if(req.http.Cookie ~ "language=(nl|fr|en|de|es)"){
        hash_data(regsub(req.http.Cookie,
        "^.*;? ?language=(nl|fr|en|de|es)( ?|;| ;).*$","\1"));
    }
}

sub vcl_recv {
    if(req.method == "BAN") {
        ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
        return(synth(200, "Ban added"));
    }
}

sub vcl_recv {
    return(synth(201,"I created something"));
}

sub vcl_backend_error {
    set beresp.http.Content-Type = "text/html; charset=utf-8";
    synthetic("An error occured: " + beresp.reason + "<br />");
    synthetic("HTTP status code: " + beresp.status + "<br />");
    return(deliver);
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    synthetic("Message of the day: " + resp.reason + "<br />");
    synthetic("HTTP status code: " + resp.status + "<br />");
    return(deliver);
}

sub vcl_recv {
    include "someFile.vcl";
}

if ((req.method != "GET" && req.method != "HEAD")
|| req.http.Authorization || req.http.Cookie) {
    return (pass);
}

import std;
sub vcl_recv {
    set req.url = std.querysort(req.url);
}

Backends and Health Probes

All the VCL we covered so far has been restricted to the hooks that allow us to change Varnish’s behavior. But let’s not forget that Varnish is all about caching backend responses. So it’s about time we deal with the VCL aspect of backends.

In general, it looks like this:

backend name {
    .attribute = "value";
}

Here’ a list of supported backend attributes:

host

A mandatory attribute that represents the hostname or the IP of the backend.

port

The backend port that will be used. The default is port 80.

connect_timeout

The amount of time Varnish waits for a connection with the backend. The default value is 3.5 seconds.

first_byte_timeout

The amount of time Varnish waits for the first byte to be returned from the backend after the initial connection. The default value is 60 seconds.

between_bytes_timeout

The amount of time Varnish waits between each byte to ensure an even flow of data. The default value is 60 seconds.

max_connections

The total amount of simultaneous connections to the backend. When the limit is reached, new connections are dropped. Make sure your backend can handle this amount of connections.

probe

The backend probe that will be used to check the health of the backend. It could be defined inline, or linked to.

Let’s throw in a code example:

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 2s;
    .first_byte_timeout = 5s;
    .between_bytes_timeout = 1s;
    .max_connections = 150;
}

In the preceding example, a connection is made to the local machine on port 8080. We’re willing to wait two seconds for an initial connection. Once the connection is established, we’re going to wait up to five seconds to get the first byte. After that we want to receive bytes with a regular frequency. We’re willing to wait one second between each byte. We will allow up to 150 simultaneous connections to the backend.

If any of the criteria is not met, a backend error is thrown and the execution is passed to vcl_backend_error with an HTTP 503 status code.

Without the use of a backend probe, an unhealthy backend can only be spotted when the connection fails. A so-called backend probe will poll the backend on a regular basis and check its health based on certain assertions. Probes can be defined similarly to backends.

In general, it looks like this:

probe name {
    .attribute = "value";
}

Here’s a list of supported probe attributes:

url

The url that is requested by the probe. This defaults to /.

request

A custom HTTP request that can be sent to the backend.

expected_response

The expected HTTP status code that is returned by the backend. This defaults to 200.

timeout

The amount of time the probe waits for a backend response. The default value is 2 seconds.

interval

How often the probe is run. The default value is 5 seconds.

window

The amount of polls that are examined to determine the backend health. The default value is 8.

threshold

The amount of polls in the window that should succeed before a backend is considered healthy. This defaults to 3.

initial

The amount of initial backend polls it takes when Varnish starts before a backend is considered healthy. Defaults to threshold - 1.

Here’s an example of a backend probe that is defined inline:

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = {
        .url = "/";
        .expected_response = 200;
        .timeout = 1s;
        .interval = 1s;
        .window = 5;
        .threshold = 3;
        .initial = 2;
    }
}

This is what happens: the backend tries to connect to the local host on port 8080. There is a backend probe available that determines the backend health. The probe polls the backend on port 8080 on the root URL. In order for a poll to succeed, an HTTP response with a 200 status code is expected, and this response should happen within a second.

Every second, the probe will poll the backend and the result of five polls is used to determine health. Of those five polls, at least three should succeed before we consider the backend to be healthy. At startup time, this should be two polls.

We can also define a probe explicitly, name it, and reuse that probe for multiple backends. Here’s a code example that does that:

probe myprobe {
    .url = "/";
    .expected_response = 200;
    .timeout = 1s;
    .interval = 1s;
    .window = 5;
    .threshold = 3;
    .initial = 2;
}

backend default {
    .host = "my.primary.backend.com";
    .probe = myprobe;
}

backend backup {
    .host = "my.backup.backend.com";
    .probe = myprobe;
}

This example has two backends, both running on port 80, but on separate machines. We use the myprobe probe to check the health of both machine, but we only define the probe once.

In Chapter 6 we’ll cover some more advanced backend and probing topics.

Access Control Lists

Access control lists, or ACLs as we like to call them, are language constructs in VCL that contain IP addresses, IP ranges, or hostnames. An ACL is named and IP addresses can be matched to them in VCL.

ACLs are mostly used to restrict access to certain parts of your content or logic based on the IP address. The match can be done by using the match operator (~) in an if-clause.

Here’s a code example:

acl allowed {
    "localhost";    # myself
    "192.0.2.0"/24; # and everyone on the local network
    ! "192.0.2.23"; # except for one specific IP
}

sub vcl_recv {
    if(!client.ip ~ allowed) {
        return(synth(403,"You are not allowed to access this page."));
    }
}

In the preceding example, only local connections are allowed, or connections that come from the 192.0.2.0/24 IP range, with one exception: connections from 192.0.2.23 aren’t allowed. No other IP addresses are allowed access, either—they will get an HTTP 403 error if they do try.

In Chapter 5, we’ll be using ACLs to restrict access to the cache invalidation mechanism.

VCL Variables

You’re probably already familiar with req.url and client.ip. Yes, these are VCL variables, and let me tell you, there are a lot of them.

Here’s a list of the different variable objects:

bereq

The backend request data structure.

beresp

The backend response variable object.

client

The variable object that contains information about the client connection.

local

Information about the local TCP connection.

now

Information about the current time.

obj

The variable object that contains information about an object that is stored in cache.

remote

Information about the remote TCP connection. This is either the client or a proxy that sits in front of Varnish.

req

The request variable object.

req_top

Information about the top-level request in a tree of ESI requests.

resp

The response variable object.

server

Information about the Varnish server.

storage

Information about the storage engine.

Table 4-1 lists a couple of useful variables and explains what they do. For a full list of variables, go to the variables section in the VCL part of the Varnish documentation site.

Varnish’s Built-In VCL

Remember “Varnish Built-In VCL Behavior” in which I talked about the built-in VCL behavior of Varnish? Now that we know how VCL works, we can translate that behavior into a full-blown VCL file.

Even if you don’t register a VCL file or if your VCL file only contains a backend definition, Varnish will behave as follows:¹

/*-
 * Copyright (c) 2006 Verdens Gang AS
 * Copyright (c) 2006-2015 Varnish Software AS
 * All rights reserved.
 *
 * Author: Poul-Henning Kamp <phk@phk.freebsd.dk>
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 * SUCH DAMAGE.
 *

 *
 * The built-in (previously called default) VCL code.
 *
 * NB! You do NOT need to copy & paste all of these functions into your
 * own vcl code, if you do not provide a definition of one of these
 * functions, the compiler will automatically fall back to the default
 * code from this file.
 *
 * This code will be prefixed with a backend declaration built from the
 * -b argument.
 */

vcl 4.0;

#######################################################################
# Client side

sub vcl_recv {
    if (req.method == "PRI") {
	/* We do not support SPDY or HTTP/2.0 */
	return (synth(405));
    }
    if (req.method != "GET" &&
      req.method != "HEAD" &&
      req.method != "PUT" &&
      req.method != "POST" &&
      req.method != "TRACE" &&
      req.method != "OPTIONS" &&
      req.method != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }

    if (req.method != "GET" && req.method != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
        /* Not cacheable by default */
        return (pass);
    }
    return (hash);
}

sub vcl_pipe {
    # By default Connection: close is set on all piped requests, to stop
    # connection reuse from sending future requests directly to the
    # (potentially) wrong backend. If you do want this to happen, you can undo
    # it here.
    # unset bereq.http.connection;
    return (pipe);
}

sub vcl_pass {
    return (fetch);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_purge {
    return (synth(200, "Purged"));
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        // A pure unadultered hit, deliver it
        return (deliver);
    }
    if (obj.ttl + obj.grace > 0s) {
        // Object is in grace, deliver it
        // Automatically triggers a background fetch
        return (deliver);
    }
    // fetch & deliver once we get the result
    return (miss);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_deliver {
    return (deliver);
}

/*
 * We can come here "invisibly" with the following errors: 413, 417 & 503
 */
sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic( {"<!DOCTYPE html>
<html>
  <head>
    <title>"} + resp.status + " " + resp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + resp.status + " " + resp.reason + {"</h1>
    <p>"} + resp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + req.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"} );
    return (deliver);
}

#######################################################################
# Backend Fetch

sub vcl_backend_fetch {
    return (fetch);
}

sub vcl_backend_response {
    if (beresp.ttl <= 0s ||
      beresp.http.Set-Cookie ||
      beresp.http.Surrogate-control ~ "no-store" ||
      (!beresp.http.Surrogate-Control &&
        beresp.http.Cache-Control ~ "no-cache|no-store|private") ||
      beresp.http.Vary == "*") {
        /*
        * Mark as "Hit-For-Pass" for the next 2 minutes
        */
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

sub vcl_backend_error {
    set beresp.http.Content-Type = "text/html; charset=utf-8";
    set beresp.http.Retry-After = "5";
    synthetic( {"<!DOCTYPE html>
<html>
  <head>
    <title>"} + beresp.status + " " + beresp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
    <p>"} + beresp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + bereq.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"} );
    return (deliver);
}

#######################################################################
# Housekeeping

sub vcl_init {
}

sub vcl_fini {
    return (ok);
}

As a quick reminder, this is what the preceding code does:

• It does not support the PRI method and throws an HTTP 405 error when it is used.

• Request methods that differ from GET, HEAD, PUT, POST, TRACE, OPTIONS, and DELETE are not considered valid and are piped directly to the backend.

• Only GET and HEAD requests can be cached, other requests are passed to the backend and will not be served from cache.

• When a request contains a cookie or an authorization header, the request is passed to the backend and the response is not cached.

• If at this point the request is not passed to the backend, it is considered cacheable and a cache lookup key is composed.

• A cache lookup key is a hash that is composed using the URL and the hostname or IP address of the request.

• Objects that aren’t stale are served from cache.

• Stale objects that still have some grace time are also served from cache.

• All other objects trigger a miss and are looked up in cache.

• Backend responses that do not have a positive TTL are deemed uncacheable and are stored in the hit-for-pass cache.

• Backend responses that send a Set-Cookie header are also considered uncacheable and are stored in the hit-for-pass cache.

• Backend responses with a no-store in the Surrogate-Control header will not be stored in cache either.

• Backend responses containing no-cache, no-store, or private in the Cache-control header will not be stored in cache.

• Backend responses that have a Vary header that creates cache variations on every request header are not considered cacheable.

• When objects are stored in the hit-for-pass cache, they remain in that blacklist for 120 seconds.

• All other responses are delivered and stored in cache.

A Real-World VCL File

The VCL file that you see in “Varnish’s Built-In VCL” represents the desired behavior of Varnish. In an ideal world, this VCL code should suffice to make any website bulletproof. The reality is that modern day websites, applications and APIs don’t conform 100% to these rules.

The built-in VCL assumes that cacheable websites do not have cookies. Let me tell you: websites without cookies are few and far between. The following VCL file deals with these real-world cases and will dramatically increase your hit rate.

There are plenty of good VCL templates out there. I could also write one myself, but I’d just be reinventing the wheel. In the spirit of open source, I’d much rather showcase one of the most popular VCL templates out there.

The author of the VCL file is Mattias Geniar, a fellow Belgian, a fellow member of the hosting industry, a friend, and a true Varnish ambassador. Go to his GitHub repository to see the code.

This VCL template primarily sanitizes the request, optimizes backend connections, facilitates purges, adds caching stats, and adds ESI support.

Conclusion

By now you should know the syntax, functions, different language constructs, return types, variable objects, and execution flow of VCL. Use this chapter as a reference when in doubt.

Don’t forget that the Varnish Cache project has a pretty decent documentation site. If you don’t find the answer to your question in this book, you’ll probably find it there.

At this point, I expect you to be comfortable with the VCL syntax and be able to read and interpret pieces of VCL you come across. In Chapter 7, we’ll dive deeper into some common scenarios in which custom VCL is required.

Caching for Too Long

Throughout this book, I’ve always kept the best interests of the website owner, developer, and sysadmin in mind. The reality is that the site, API, and application are primarily services that the end user consumes. In the end, it’s all about the end user.

• Why do we want to make the site fast? For the user!

• Why do we want to keep the site available? For the user!

• Why do we cache? So that the user has a good experience!

Caching data for too long would mess with the integrity of the data, giving the user a bad experience when up-to-date output is important. This is especially the case for news websites.

We already talked about the use of Cache-control and Expires headers. It’s important to estimate the right time-to-live and set the right values for these headers. The more accurate the time-to-live, the better the balance.

Unfortunately, in many cases the data will be out-of-date even before the object expires. Setting it to a lower value could jeopardize the health and responsiveness of your backend. Talk about being stuck between a rock and a hard place!

Worry not—Varnish has your back! Varnish offers various mechanisms to evict objects from the cache based on certain criteria. By accessing these eviction mechanisms from your code, you can actively invalidate objects, even if they haven’t expired. That way your breaking news will be correctly displayed on the front page of your website, even if the object still has two hours to live according to the time-to-live.

The Varnish documentation has a page dedicated to cache invalidation. Have a look if you’re interested.

Purging

Purging is the easiest way to invalidate the cache. In the following example, you can see that in VCL you can perform a return (purge) from within the vcl_recv subroutine. This will explicitly evict the object from cache. The object will be identified by the criteria set in vcl_hash, so by default that is the hostname and the URL. Memory is freed up immediately and cache variations are also evicted.

acl purge {
    "localhost";
    "192.168.55.0"/24;
}

sub vcl_recv {
    # allow PURGE from localhost and 192.168.55...

    if (req.method == "PURGE") {
        if (!client.ip ~ purge) {
            return(synth(403,"Not allowed."));
        }
        return (purge);
    }
}

There’s a bit more housekeeping involved when you want to do it right: the preceding example protects you from unauthorized invalidations by enforcing an ACL. Only purges from localhost or from the 192.168.55.0/24 subnet are allowed.

And then there’s the PURGE request method that is checked. By requesting the resource with PURGE instead of GET, you’re basically telling Varnish that this HTTP request is not a regular data retrieval request, but a purging request.

You can implement a purge call anywhere in your code and you’ll typically use an HTTP client that is supported by your programming language or framework. In many cases, that client will be cURL-based. Here’s a purging example using the cURL binary:

curl -XPURGE http://example.com/some/page

This example uses the -X parameter in cURL to set the request method. As expected, we’re setting it to PURGE and setting the URL to http://example.com/some/page. That’s the resource we’re removing from cache.

Banning

Purging is easy: it uses the object’s hash, it evicts just that one object, and it can be executed with a simple return(purge).

But when you have a large number of purges to perform or you’re not exactly sure which resources are stale, exact URL invalidations might feel restrictive to you. A pattern-based invalidation mechanism would solve that problem, and banning does just that.

Banning should not be an unknown concept to you; in “Ban”, we talked about the ban function that executes these bans.

Basically, bans use a regular expression match to mark objects that should be removed from cache. These marked objects are put on the so-called ban list. Banning does not remove items from cache immediately and hence does not free up any memory directly.

Bans are checked when an object is hit and executed accordingly based on the ban list. There’s also a so-called ban lurker background thread that checks for bans that match against any variable of the obj object.

Here’s a basic BAN example. It does exactly the same thing as the PURGE example, but adds URL pattern-matching capabilities:

acl ban {
    "localhost";
    "192.168.55.0"/24;
}

sub vcl_recv {
    if (req.method == "BAN") {
        if (!client.ip ~ ban) {
            return(synth(403, "Not allowed."));
        }
        ban("req.http.host == " + req.http.host +
            " && req.url ~ " + req.url);
        return(synth(200, "Ban added"));
    }
}

acl ban {
    "localhost";
    "192.168.55.0"/24;
}

sub vcl_backend_response {
  set beresp.http.x-host = bereq.http.host;
  set beresp.http.x-url = bereq.url;
}

sub vcl_deliver {
  unset resp.http.x-host;
  unset resp.http.x-url;
}

sub vcl_recv {
    if (req.method == "BAN") {
        if (!client.ip ~ ban) {
            return(synth(403, "Not allowed."));
        }
        ban("obj.http.x-host == " + req.http.host +
            " && obj.http.x-url ~ " + req.url);
        return(synth(200, "Ban added"));
    }
}

acl ban {
    "localhost";
    "192.168.55.0"/24;
}

sub vcl_backend_response {
  set beresp.http.x-host = bereq.http.host;
  set beresp.http.x-url = bereq.url;
}

sub vcl_deliver {
  unset resp.http.x-host;
  unset resp.http.x-url;
}

sub vcl_recv {
    if (req.method == "BAN") {
        if (!client.ip ~ ban) {
            return(synth(403, "Not allowed."));
        }
        if(req.http.x-ban-regex) {
            ban("obj.http.x-host == " + req.http.host + "
            && obj.http.x-url ~ " + req.http.x-ban-regex);
        } else {
            ban("obj.http.x-host == " + req.http.host + "
            && obj.http.x-url == " + req.url);
        }
        return(synth(200, "Ban added"));
    }
}

curl -XBAN http://example.com/ -H"x-ban-regex: ^/product/[0-9]+/details"

curl -XBAN http://example.com/product/121/details

varnishadm ban.list

Present bans:
0xb75096d0 1318329475.377475    10      obj.http.x-host == example.com
&& obj.http.x-url ~ ^/product/[0-9]+/details
0xb7509610 1318329470.785875    20C     obj.http.x-host == example.com
&& obj.http.x-url ~ ^/category

Banning from the Command Line

The previous two sections approached the execution of cache invalidation from an HTTP perspective, meaning that your regular requests and your purge/ban requests all pass through the same channel. I mentioned the upside: it’s very easy to code in VCL and just as easy to implement in your backend.

There are also some downsides:

• You have to write additional VCL code, which adds complexity.

• There is no uniform way of implementing banning and purging in Varnish; your application will depend on the invalidation implementation in VCL.

• Although the ACLs provide a level of security, there is no isolation from a networking perspective.

Luckily, Varnish offers an admin console that allows you to issue ban statements. This can be done locally or remotely through the varnishadm program. In “CLI address binding”, I mentioned how you can configure your Varnish to accept remote connections on the admin interface.

varnishadm is just a client that connects to the Varnish CLI socket. You can also make a TCP connection to this socket and issue ban commands directly from within your code. Have a look at the documentation page on the Varnish Command Line Interface to learn more about the commands, remote connections, and the authentication protocol.

Here’s an example of our product detail invalidation, but this time using the CLI:

varnishadm> ban obj.http.x-host == example.com && obj.http.x-url ~^/product/