The Expires Header

The Expires header is a pretty straight forward one: you just set the date and time when an object should be considered stale. This is a response header that is sent by the backend.

Here’s an example of such a header:

Expires: Sat, 09 Sep 2017 14:30:00 GMT

The Cache-Control Header

The Cache-control header defines the time-to-live in a relative way: instead of stating the time of expiration, Cache-control states the amount of seconds until the object expires. In a lot of cases, this is a more intuitive approach: you can say that an object should only be cached for an hour by assigning 3,600 seconds as the time-to-live.

This HTTP header has more features than the Expires header: you can set the time to live for both clients and proxies. This allows you to define distinct behavior depending on the kind of system that processes the header; you can also decide whether to cache and whether to revalidate with the backend.

Cache-control: public, max-age=3600, s-maxage=86400

The preceding example uses three important keywords to define the time-to-live and the ability to cache:

public

Indicates that both browsers and shared caches are allowed to cache the content.

max-age

The time-to-live in seconds that must be respected by the browser.

s-maxage

The time-to-live in seconds that must be respected by the proxy.

It’s also important to know that Varnish only respects a subset of the Cache-control syntax. It will only respect the keywords that are relevant to its role as a reverse caching proxy:

• Cache-control headers sent by the browser are ignored

• The time-to-live from an s-maxage statement is prioritized over a max-age statement

• Must-revalidate and proxy-revalidate statements are ignored

• When a Cache-control response header contains the terms private, no-cache, or no-store, the response is not cached

Expiration Precedence

Varnish respects both Expires and Cache-control headers. In the Varnish Configuration Language, you can also decide what the time-to-live should be regardless of caching headers. And if there’s no time-to-live at all, Varnish will fall back to its hardcoded default of 120 seconds.

Here’s the list of priorities that Varnish applies when choosing a time-to-live:

1. If beresp.ttl is set in the VCL, use that value as the time-to-live.

2. Look for an s-maxage statement in the Cache-control header.

3. Look for a max-age statement in the Cache-control header.

4. Look for an expires header.

5. Cache for 120 seconds under all other circumstances.

ETag

An Etag is an HTTP response header that is either set by the web server or your application. It contains a unique value that corresponds to the state of the resource.

A common strategy is to create a unique hash for that resource. That hash could be an md5 or a sha hash based on the URL and the internal modification date of the resource. It could be anything as long as it’s unique.

HTTP/1.1 200 OK
Host: localhost
Etag: 7c9d70604c6061da9bb9377d3f00eb27
Content-type: text/html; charset=UTF-8

Hello world output

As soon as a browser sees this Etag, it stores the value. Upon the next request, the value of the Etag will be sent back to the server in an If-None-Match request header.

GET /if_none_match.php HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-None-Match: 7c9d70604c6061da9bb9377d3f00eb27

The server receives this If-None-Match header and checks if the value differs from the Etag it’s about to send.

If the Etag value is equal to the If-None-Match value, the web server or your application can return an HTTP/1.1 304 Not Modified response header to indicate that the value hasn’t changed.

HTTP/1.0 304 Not Modified
Host: localhost
Etag: 7c9d70604c6061da9bb9377d3f00eb27

When you send a 304 status code, you don’t send any payload, which can dramatically reduce the amount of bytes sent over the wire. The browser receives the 304 and knows that it can still output the old data.

If the If-None-Match value doesn’t match the Etag, the web server or your application will return the full payload, accompanied by the HTTP/1.1 200 OK response header and, of course, the new Etag.

This is an excellent way to conserve resources. Whereas the primary goal is to reduce bandwidth, it will also help you to reduce the consumption of memory, CPU cycles, and disk I/O if you implement it the right way.

Here’s an implementation example. It’s just some dummy script that, besides proving my point, serves no real purpose. It’s written in PHP because PHP is my language of choice. The implementation is definitely not restricted to PHP. You can implement this in any server-side language you like.

<?php
$etag = md5(__FILE__.filemtime(__FILE__));
header('Etag: ' . $etag);
if (isset($_SERVER['HTTP_IF_NONE_MATCH'])
&&  $_SERVER['HTTP_IF_NONE_MATCH'] == $etag) {
    header('HTTP/1.0 304 Not Modified');
    exit;
}
sleep(5);
?>
<h1>Etag example</h1>
<?php
echo date("Y-m-d H:i:s").'<br />';

Last-Modified

ETags aren’t the only way to do conditional requests; there’s also an alternative technique based on the Last-Modified response header. The client will then use the If-Modified-Since request header to validate the freshness of the resource.

The approach is similar:

1. Let your web server or application return a Last-Modified response header

2. The client stores this value and uses it as an If-Modified-Since request header upon the next request

3. The web server or application matches this If-Modified-Since value to the modification date of the resource

4. Either an HTTP/1.1 304 Not Modified or a HTTP/1.1 200 OK is returned

The benefits are the same: reduce the bytes over the wire and load on the server by avoiding the full rendering of output.

The starting point in the following example is the web server (or the application) returning a Last-Modified response header:

HTTP/1.1 200 OK
Host: localhost
Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT
Content-type: text/html; charset=UTF-8

Hello world output

The browser stores the Last-Modified value and uses it as an If-Last-Modified in the next request:

GET /if_last_modified.php HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT

The resource wasn’t modified, a 304 is returned, and the Last-Modified value remains the same:

HTTP/1.0 304 Not Modified
Host: localhost
Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT

The browser does yet another conditional request:

GET /if_last_modified.php HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT

The resource was modified in the meantime and a full 200 is returned, including the payload and a new Last-Modified_header.

HTTP/1.1 200 OK
Host: localhost
Last-Modified: Fri, 22 Jul 2016 11:00:23 GMT
Content-type: text/html; charset=UTF-8

Some other hello world output

Time for another implementation example for conditional requests, this time based on the Last-Modified header. Again, it’s dummy code, written in PHP:

<?php
header('Last-Modified: ' .
gmdate('D, d M Y H:i:s', filemtime(__FILE__)) . ' GMT');
if (isset($_SERVER['HTTP_IF_MODIFIED_SINCE']) &&
    strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE']) >= filemtime(__FILE__))
{
    header('HTTP/1.0 304 Not Modified');
    exit;
}
sleep(5);
?>
<h1>Last-Modified example</h1>
<?php
echo date("Y-m-d H:i:s").'<br />';

Just like in the previous implementation example, we fake the delay caused by heavy load and use a sleep statement to make the application seem slower than it really is.

How Varnish Deals with Conditional Requests

When Varnish spots an If-Modified-Since or If-None-Match header in the request, it keeps track of the Last-Modified timestamp and/or the Etag. Regardless of whether or not Varnish has the object in cache, a 304 status code will be returned if the Last-Modified or the Etag header matches.

From a client point of view, Varnish reduces the amount of bytes over the wire by returning the 304.

On the other hand, Varnish also supports conditional requests when it comes to backend communication: when an object is considered stale, Varnish will send If-Modified-Since and If-None-Match headers to the backend if the previous response from the backend contained either a Last-Modified timestamp or an Etag.

When the backend returns a 304 status code, Varnish will not receive the body of that response and will assume the content hasn’t changed. As a consequence, the stale data will have been revalidated and will no longer be stale. The Age response header will be reset to zero and the object will live in cache in accordance to the time-to-live that was set by the web server or the application.

Typically, stale data is revalidated by Varnish, but there is a VCL variable that allows you to manipulate that behavior: the beresp.keep variable decides how long stale objects will be returned while performing a conditional request. It’s basically an amount of time that is added to the time-to-live. This allows Varnish to perform the conditional requests asynchronously without the client noticing any delays. The beresp.keep variable works independently from the beresp.grace variable.

When Is a Request Considered Cacheable?

When Varnish receives a request, it has to decide whether or not the response can be cached or even served from cache. The rules are simple and based on idempotence and state.

A request is cacheable when:

• The request method is GET or HEAD

• There are no cookies being sent by the client

• There is no authorization header being sent

When these criteria are met, Varnish will look the resource up in cache and will decide if a backend request is needed, or if the response can be served from cache.

When Does Varnish Completely Bypass the Cache?

If a request is not cacheable, the request is passed, a backend connection is made and the result is stored in the hit-for-pass cache. An example of this is a POST request.

But all of this happens under the assumption that the request method is a valid one that complies to RFC2616. Other request methods will not be processed by Varnish and will be piped to the backend.

When Varnish goes in to pipe mode, it opens a TCP connection to the backend, transmits the original request and immediately returns the response. There’s no further processing of the request or response.

Here’s a list of valid request methods according to the built-in VCL:

• GET

• HEAD

• PUT

• POST

• DELETE

• TRACE

• OPTIONS

All other request methods will be piped to the backend.

How Does Varnish Identify an Object?

Once we decide that an object is cacheable, we need a way to identify the object in order to retrieve it from cache. A hash key is composed of several values that serve as a unique identifier.

1. If the request contains a Host header, the hostname will be added to the hash.

2. Otherwise, the IP address will be added to the hash.

3. The URL of the request is added to the hash.

Based on that hash, Varnish will retrieve the object from cache.

When Does Varnish Cache an Object?

If an object is not stored in cache or when it’s considered stale, a backend connection is made. Based on the backend response, Varnish will decide if the returned object will be stored in cache or if the cache is going to be bypassed.

A response will be stored in cache when:

• The time-to-live is more than zero.

• The response doesn’t contain a Set-Cookie header.

• The Cache-control header doesn’t contain the terms no-cache, no-store, or private.

• The Vary header doesn’t contain *, meaning vary on all headers.

What Happens if an Object Is Not Stored in Cache?

If after the backend response Varnish decides that an object will not be stored in cache, it puts the object on a “blacklist”—the so-called hit-for-pass cache.

For a duration of 120 seconds, the next requests will immediatly connect with the backend, directly serving the response, without attempting to store the response in cache.

After 120 seconds, upon the next request, the response can be re-evaluated and a decision can be made whether or not to store the object in cache.

How Long Does Varnish Cache an Object?

Once an object is stored in cache, a decision must be made on the time-to-live. I mentioned this before, but there’s a list of priorities that Varnish uses to decide which value it will use as the TTL.

Here’s the prioritized list:

1. If beresp.ttl is set in the VCL, use that value as the time-to-live.

2. Look for an s-maxage statement in the Cache-control header.

3. Look for a max-age statement in the Cache-control header.

4. Look for an Expires header.

5. Cache for 120 seconds under all other circumstances.

When the object is stored in the hit-for-pass cache, it is cached for 120 seconds, unless you change the value in VCL.