Chapter 10. Taking It to the Next Level

By now, we’ve established a level of understanding:

You know what Varnish is and where it fits in
You know how to install and configure Varnish
You understand some of the HTTP best practices that Varnish respects
You can interprete VCL snippets and write some of your own
You know how to invalidate (parts of) the cache
You’re familiar with the load-balancing capabilities of Varnish
You’re able to leverage some of the Varnish logging and monitoring tools
You know how to adapt your application or your VCL to real-world scenarios

So what’s next? This chapter will point you in the right direction if you want to take it to the next level. I’ll address some specific topics that are beyond the scope of this book and refer to useful resources.

We’ll talk about RESTful services, VMODs, the future of the Varnish project, and ways to get help if you have questions.

What About RESTful Services?

Throughout this book, I’ve been focusing on websites and web applications. But in this day and age, service-oriented architectures have become normal. The idea that software isn’t only consumed by humans but also by other systems has become mainstream. The fact that your application has a feed or an API doesn’t seem that exotic anymore. Nowadays, the application usually is a REST API and the GUI is just a piece of frontend code that consumes it.

That being said, RESTful services are very easy to cache with Varnish. As a matter of fact, Roy Fielding, who introduced REST in his PhD dissertation, defined REST as “an architectural style to describe the design and development of the architecture for the modern web.”

In his dissertation, he applies REST to the HTTP protocol and describes his experiences. He also refers to several data elements that can be used to identify and cache resources.

So in essence, if you write true RESTful services, you are already applying the best practices and you don’t need a lot of VCL changes. The advice from Chapter 3 surely applies here, as well!

Patch Support

You will need to add explicit support for the HTTP PATCH method. Varnish 4.1 only supports the HTTP methods that were defined in RFC 2616 and PATCH is not part of that specification. PATCH support was added in RFC 2068, you’ll need to provide the following piece of VCL to make it work:

if (req.method != "GET" &&
    req.method != "HEAD" &&
    req.method != "PUT" &&
    req.method != "POST" &&
    req.method != "TRACE" &&
    req.method != "OPTIONS" &&
    req.method != "PATCH" &&
    req.method != "DELETE") {
    return (pipe);
}

Authentication

If access to your RESTful API requires authentication, you can either use the authorization header or use a VMOD for alternative forms of authentication. We’ll talk about VMODs in “Extending Varnish’s Behavior with VMODs”.

But if you want simple basic authentication on your API without having to connect to the backend, you could use vmod_basicauth.

Invalidation

Letting your RESTful API return stale content is just as risky as it would be on websites and web applications. Therefore, it is equally important to have a correct invalidation strategy.

There’s obvious low-hanging fruit like invalidating individual resources that are called using POST, DELETE, PATCH, and PUT.

Let’s say you perform a PUT as in this example:

PUT /user/1

You’re basically performing a full update on the /users/1 resource. So if you’re trying to retrieve that resource using GET as in the following, you need to make sure it is invalidated:

GET /user/1

If you remember the built-in VCL, you’ll know that GET calls are cached and PUT calls are not. In your PUT logic, you can purge or ban that resource.

But it gets more complicated: there are aggregate resources that could also contain information about the updated resource. Take, for example, the following GET call:

GET /user

This /user resource also needs to be invalidated as it contains a collection of users that includes user 1.

In summary, as RESTful APIs are data-driven, it is usually important that the data is consistent. Make sure you have a decent cache-invalidation strategy. If eventual consistency is acceptable, you can rely on the expiration of your resources without having to worry about explicit invalidation.

Extending Varnish’s Behavior with VMODs

As you will remember from Chapter 4, the VCL syntax is quite extensive and offers many ways to influence the behavior of Varnish. But depending on your use case, you might encounter some limitations—maybe VCL doesn’t support a feature you need.

In previous versions of Varnish, the use of inline C-code was promoted. Because Varnish compiles VCL into C code, it was quite easy to process additional C code. But as you can imagine, the slightest of errors would crash Varnish entirely.

Note

In Varnish 4.1, inline C is still supported, but not advised. If you want to use custom C code, you need to enable vcc_allow_inline_c as a startup option.

So what’s the alternative? Varnish now promotes the use of Varnish modules, also known as VMODs. A VMOD is also written in C, but instead of directly extending the behavior of Varnish, a VMOD is offered as a shared library that can be called from your VCL code. You could say that VMODs are modules that enrich the VCL syntax with additional features.

Finding and Installing VMODs

As mentioned before, there is an official VMOD directory on the Varnish website. You can also write your own VMODs if you need to. The Varnish community offers some basic VMODs that are installed as one big collection. They’re up on GitHub if you’re interested.

Enabling VMODs

It’s not because you compiled and installed the VMODs in the right directory that they are immediately enabled. You still have to import them into your VCL file.

Here’s how you import the standard VMOD:

import std;

Depending on the kind of VMOD, you can then start using its functions within the VCL subroutines. If your VMOD requires further initialzation, you can do this in vcl_init.

Remember this example from Chapter 6?

vcl 4.0;

import directors;

backend web001 {
  .host = "web001.example.com";
  .probe = healthcheck;
}

backend web002 {
  .host = "web002.example.com";
  .probe = healthcheck;
}

sub vcl_init {
    new loadbalancing = directors.round_robin();
    loadbalancing.add_backend(web001);
    loadbalancing.add_backend(web002);
}

sub vcl_recv {
    set req.backend_hint = loadbalancing.backend();
}

In that example, we needed to initialize a director by setting its distribution algorithm and adding backends. We did this in vcl_init.

VMODs That Are Shipped with Varnish

If you installed Varnish, you already have two VMODs at your disposal:

vmod_std that offers a set of standard functions
vmod_directors that offers load-balancing capabilities

Everything beyond that needs to be installed separately.

Need Help?

Read the book and feel like you have all the tools you need to setup a Varnish stack with a killer hit rate? Good for you! This book doesn’t have the ambition or arrogance to be the Varnish bible. There are still things to be learned.

But if you’re stuck and need help, here are several available resources:

varnish-misc mailing list is open for questions
There is a #varnish IRC channel on irc.linpro.no. People there tend to be friendly and helpful
There is a Varnish tag on StackOverflow where you can ask questions
VCL-related questions can be asked on the varnish-vcl tag on StackOverflow
If you’re a #dta kind of person and you just want to throw it on Twitter, you can also direct your questions to @varnishcache

If you need support, training, or professional services, Varnish Software is the company you need. Varnish Software is the main sponsor of the Varnish project and employs a lot of Varnish core contributors. They’re the one-stop shop when it comes to commercial products and services on top of Varnish.

The Future of the Varnish Project

By the time you’re reading this, Varnish version 5 will be out and about, but that does not mean this book is outdated. The adoption rate for the broad public is usually quite slow. There are still tons of Varnish setups out there that use version 3.

Varnish 5 features a couple of changes, but nothing that is too worrying. The VCL syntax remains unchanged.

The Varnish team has now switched from a feature-based release schedule to a time-based release schedule. This means you can expect a new version of Varnish every six months. The main new feature in Varnish 5 is experimental support for HTTP/2. It’s not yet production-ready, but it can handle HTTP/2 traffic if you enable the http2 feature flag.

There are also some other features and improvements. I wrote a blog post about it—check it out!

With the two annual releases we can expect, it will be easier for users to see the direction Varnish is heading in. I expect HTTP/2 to remain the main talking point. I expect a lot of improvements by the next release when it comes to HTTP/2 support.

The future of Varnish is bright. Stay tuned, and keep an eye out for new releases!

Previous Chapter

9. What Does This Mean for Your Business?

Next Chapter

Index