docker exec

First, let’s take a look at the easiest and best way to get inside a running container. The dockerd server and docker command-line tool support remotely executing a new process in a running container via the docker exec command. So let’s start up a container in background mode, and then enter it using docker exec and invoking a shell. The command you invoke doesn’t have to be a shell: it’s possible to run individual commands inside the container and see their results outside it using docker exec. But if you want to get inside the container to look around, a shell is the easiest way to do that.

To run docker exec, we’ll need our container’s ID, like we did previously when we inspected it. For this demo, we listed out the containers and our ID is 589f2ad30138. We can now use that to get inside the container with docker exec. The command line for that, unsurprisingly, looks a lot like the command line for docker run. We request an interactive session and a pseudo-TTY with the -i and -t flags:

$ docker exec -i -t 589f2ad30138 /bin/bash
root@589f2ad30138:/#

Note that we got a command line back that tells us the ID of the container we’re running inside. That’s pretty useful for keeping track of where we are. We can now run a normal Linux ps to see what else is running inside our container. We should see our other bash process that we backgrounded earlier when starting the container.

root@589f2ad30138:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 23:13 ?        00:00:00 /bin/bash
root         9     0  1 23:14 ?        00:00:00 /bin/bash
root        17     9  0 23:14 ?        00:00:00 ps -ef

nsenter

Part of the core util-linux package from kernel.org is nsenter, short for “Namespace Enter,” which allows you to enter any Linux namespace. In Chapter 11, we’ll go into more detail on namespaces. But they are the core of what makes a container a container. Using nsenter, therefore, we can get into a Docker container from the server itself, even in situations where the dockerd server is not responding and we can’t use docker exec. nsenter can also be used to manipulate things in a container as root on the server that would otherwise be prevented by docker exec, for example. This can be really useful when you are debugging. Most of the time, docker exec is all you need, but you should have nsenter in your tool belt.

Most Linux distributions ship with the util-linux package that contains nsenter. But not all of them ship with one that is new enough to have nsenter itself installed, because it’s a recent addition to the package. Ubuntu 14.04, for example, still has util-linux from 2012. Ubuntu 16.04, however, has the newest package. If you are on a distribution that does not have it, the easiest way to get ahold of nsenter is to install it via a third-party Docker container.

This works by pulling a Docker image from the Docker Hub registry and then running a specially crafted Docker container that will install the nsenter command-line tool into /usr/local/bin. This might seem strange at first, but it’s a clever way to allow you to install nsenter to any Docker server remotely using nothing more than the docker command.

This code shows how we install nsenter to /usr/local/bin on your Docker server:

$ docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter
Unable to find image 'jpetazzo/nsenter' locally
Pulling repository jpetazzo/nsenter
9e4ef84f476a: Download complete
511136ea3c5a: Download complete
71d9d77ae89e: Download complete
Status: Downloaded newer image for jpetazzo/nsenter:latest
Installing nsenter to /target
Installing docker-enter to /target

Unlike docker exec, which can be run remotely, nsenter requires that you run it on the server itself. The README in the GitHub repo explains how to set this up to work over SSH automatically if you want to do that. For our purposes, we’ll log in to our Docker server and then invoke the command from there. In any case, like with docker exec, we need to have a container running. You should still have one running from earlier. If not, go back and start one, and then log into your server.

docker exec is pretty simple, but nsenter is a little inconvenient to use. It needs to have the PID of the actual top-level process in your container. That’s less than obvious to find and requires a few steps. Luckily there’s a convenience wrapper installed by that Docker container we just ran, called docker-enter, which takes away the pain. But before we jump to the convenience wrapper, let’s run nsenter by hand so you can see what’s going on.

First we need to find out the ID of the running container, because nsenter needs to know that to access it. This is the same as previously shown for docker inspect and docker exec:

$ docker ps
CONTAINER ID  IMAGE          COMMAND      ...  NAMES
3c4f916619a5  ubuntu:latest  "/bin/bash"  ...  grave_goldstine

The ID we want is that first field, 3c4f916619a5. Armed with that, we can now find the PID we need. We do that like this:

$ PID=$(docker inspect --format \{{.State.Pid\}} 3c4f916619a5)

This will store the PID we care about into the PID environment variable. We need to have root privilege to do what we’re going to do. So you should either su to root or use sudo on the command line. Now we invoke nsenter:

$ sudo nsenter --target $PID --mount --uts --ipc --net --pid
root@3c4f916619a5:/#

$ docker run -it --privileged --pid=host debian nsenter \
    --target $PID --mount --uts --ipc --net --pid

If the end result looks a lot like docker exec, that’s because it does almost exactly the same thing under the hood!

There are a lot of command-line options there; what they’re doing is telling nsenter which parts of the container we need access to. Generally you want all of them, so you might expect that to be the default, but it’s not, so we specify them all.

Back at the beginning of this section, we mentioned that there is a convenience wrapper called docker-enter that you install by running the installation Docker container. Having now seen the mechanism involved with running nsenter, you can now appreciate that if you actually just want to enter all the namespaces for the container and skip several steps, you can do this:

$ sudo docker-enter 3c4f916619a5 /bin/bash
root@3c4f916619a5:/#

Much easier and a lot less to remember. It’s still not anywhere nearly as convenient as docker exec, but in some situations it will be all you have if the Docker daemon is not responding, so it’s a very worthwhile tool to have available.

docker volume

Docker supports a volume subcommand that makes it possible to list all of the volumes stored in your root directory and then discover additional information about them, including where they are physically stored on the server.

These volumes are not bind-mounted volumes, but special data containers that are useful for persisting data.

If we run a normal docker command that bind mounts a directory, we’ll notice that it does not create any Docker volumes.

$ docker volume ls
DRIVER              VOLUME NAME

$ docker run -d -v /tmp:/tmp ubuntu:latest sleep 120
6fc97c50fb888054e2d01f0a93ab3b3db172b2cd402fc1cd616858b2b5138857

$ docker volume ls
DRIVER              VOLUME NAME

However, you can easily create a new volume with a command like this:

$ docker volume create my-data

If you then list all your volumes, you should see something like this:

# docker volume ls
DRIVER              VOLUME NAME
local               my-data

# docker volume inspect my-data

[
    {
        "CreatedAt": "2018-07-14T21:01:05Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/my-data/_data",
        "Name": "my-data",
        "Options": {},
        "Scope": "local"
    }
]

Now you can start a container with this data volume attached to it, by running the following:

 $ docker run --rm \
     --mount source=my-data,target=/app \
     ubuntu:latest touch /app/my-persistent-data

That container created a file in the data volume and then immediately exited.

If we now mount that data volume to a different container, we will see that our data is still there.

$ docker run --rm \
    --mount source=my-data,target=/app \
    fedora:latest ls -lFa /app/persistent-data

-rw-r--r-- 1 root root 0 Jul 14 21:05 /app/my-persistent-data

And finally, you can delete the data volume when you are done with it by running:

$ docker volume rm my-data

my-data

Error response from daemon: unable to remove volume:
    remove my-data: volume is in use - [
    d0763e6e8d79e55850a1d3ab21e9d...,
    4b40d52978ea5e784e66ddca8bc22...]

These commands should help you to explore your containers in great detail. Once we’ve explained namespaces more in Chapter 11, you’ll get a better understanding of exactly how all these pieces interact and combine to create a container.

docker logs

We’ll start with the simplest Docker use case: the default logging mechanism. There are limitations to this mechanism, which we’ll explain in a minute, but for the simple case it works well, and it’s very convenient. If you are running Docker in development, this is probably the only logging strategy you’ll use there. This logging method has been there from the very beginning and is well understood and supported. The mechanism is the json-file method. The docker logs command exposes most users to this.

As is implied by the name, when you run the default json-file logging plug-in, your application’s logs are streamed by the Docker daemon into a JSON file for each container. This lets us retrieve logs for any container at any time.

We can display some logs from our example container running nginx with the following command:

$ docker logs 3c4f916619a5
2017/11/20 00:34:56 [notice] 12#0: using the "epoll" ...
2017/11/20 00:34:56 [notice] 12#0: nginx/1.0.15
2017/11/20 00:34:56 [notice] 12#0: built by gcc 4.4.7 ...
2017/11/20 00:34:56 [notice] 12#0: OS: Linux 3.8.0-35-generic

This is nice because Docker allows you to get the logs remotely, right from the command line, on demand. That’s really useful for low-volume logging.

The actual files backing this logging are on the Docker server itself, by default in /var/lib/docker/containers/<container_id>/ where the <container_id> is replaced by the actual container ID. If you take a look at one of those files, you’ll see it’s a file with each line representing a JSON object. It will look something like this:

{"log":"2018-02-04 23:58:51,003 INFO success: running.\r\n",
"stream":"stdout",
"time":"2018-02-04T23:58:51.004036238Z"}

That log field is exactly what was sent to stdout on the process in question; the stream field tells us that this was stdout and not stderr; and the precise time that the Docker daemon received it is provided in the time field. It’s an uncommon format for logging, but it’s structured rather than just a raw stream, which is beneficial if you want to do anything with the logs later.

Like a logfile, you can also tail the Docker logs live with docker logs -f:

$ docker logs -f 3c4f916619a5
nginx stderr | 2017/11/20 00:34:56 [notice] 12#0: using the "epoll" ...
nginx stderr | 2017/11/20 00:34:56 [notice] 12#0: nginx/1.0.15
nginx stderr | 2017/11/20 00:34:56 [notice] 12#0: built by gcc 4.4.7 ...
nginx stderr | 2017/11/20 00:34:56 [notice] 12#0: OS: Linux 3.8.0-35-generic

This looks identical to the usual docker logs, but the client then blocks, waiting on and displaying any new logs to appear, much like the Linux command line tail -f.

For single-host logging, this mechanism is pretty good. Its shortcomings are around log rotation, access to the logs remotely once they’ve been rotated, and disk space usage for high-volume logging. Despite being backed by a JSON file, this mechanism actually performs well enough that most production applications can log this way if that’s the solution that works for you. But if you have a more complex environment, you’re going to want something more robust, and with centralized logging capabilities.

More Advanced Logging

For those times when the default mechanism isn’t enough—and at scale it’s probably not—Docker also supports configurable logging backends. This list of plug-ins is constantly growing. Currently supported are the json-file we described earlier, as well as syslog, fluentd, journald, gelf, awslogs, splunk, etwlogs, gcplogs, and logentries, which are used for sending logs to various popular logging frameworks and services.

That’s a big list of plug-ins we just threw out there. The supported option that currently is the simplest for running Docker at scale is the option to send your container logs to syslog directly from Docker. You can specify this on the Docker command line with the --log-driver=syslog option or set it as the default in the daemon.json file for all containers.

There are also a number of third-party plug-ins available. We’ve seen mixed results from third-party plug-ins, primarily because they complicate installing and maintaining Docker. However, you may find that there is a third-party implementation that’s perfect for your system, and it might be worth the installation and maintenance hassle.

Traditionally, most Linux systems have some kind of syslog receiver, whether it be syslog, rsyslog, or any of the many other options. This protocol in its various forms has been around for a long time and is fairly well supported by most deployments. When migrating to Docker from a traditional Linux or Unix environment, many companies already have syslog infrastructure in place, which means this is often the easiest migration path as well.

While we think that syslog is the easiest solution, it has its problems. The Docker syslog driver supports TLS, TCP, and UDP connection options, which sounds great, but you should be cautious about streaming logs from Docker to a remote log server over TCP or TLS. The problem with this is that they are both run on top of connection-oriented TCP sessions, and Docker tries to connect to the remote logging server at the time of container startup. If it fails to make the connection, it will block trying to start the container. If you are running this as your default logging mechanism, this can strike at any time on any deployment.

We don’t find that to be a particularly usable state for production systems and thus encourage you to use the UDP option for syslog logging if you intend to use the syslog driver. This does mean your logs are not encrypted and not guaranteed delivery. There are various philosophies around logging, and you’ll need to balance your need for logs against the reliability of your system. We err on the side of reliability, but if you run in a secure audit environment you may have different priorities. In a tighter security environment, you could run a syslog relay on the Docker host to receive over UDP and relay the logs off box over TLS.

One final caveat to be aware of regarding most of the logging plug-ins: they are blocking by default, which means that logging back-pressure can cause issues with your application. You can change this behavior by setting --log-opt mode=non-blocking and then setting the maximum buffer size for logs to something like --log-opt max-buffer-size=4m. Once these are set, the application will no longer block when that buffer fills up. Instead, the oldest loglines in memory will be dropped. Again, reliability needs to be weighed here against your business need to receive all logs.

Non-Plug-In Community Options

Outside of the Docker and third-party plug-ins, there are community contributions with many alternate ways of shipping your logs at scale. The most common non-Docker solution is to use a method to send your logs directly to a syslog-compatible server, bypassing Docker. There are several mechanisms in use:

• Log directly from your application.

• Have a process manager in your container relay the logs (e.g., systemd, upstart, supervisor, or runit).

• Run a logging relay in the container that wraps stdout/stderr from the container.

• Relay the Docker JSON logs themselves to a remote logging framework from the server or another container.

Many of these non-plug-in options share the same drawbacks as changing the logging driver in Docker itself: they hide logs from docker logs so that you cannot access them as easily during debugging without relying on an external application. Let’s see how these options stack up.

Logging directly from your application to syslog might make sense, but if you’re running any third-party applications, this approach probably won’t work and is inflexible, since you must redeploy all your application whenever you want to make changes to how logging is handled. And unless you also emit all your logs on stdout and stderr inside the container, they will not be visible in docker logs.

Spotify has a simple, statically linked Go relay to handle logging your stderr and stdout to syslog for one process inside the container. Generally you run this from the CMD line in the Dockerfile. Because it’s statically compiled, it has no dependencies, which makes it very flexible. By default, this relay swallows the loglines, so they are not visible in docker logs unless you start it with the -tee=true flag.

The svlogd daemon from the runit init system can collect logs from your process’s stdout and ship them to remote hosts over UDP. It’s simple to set up and available in many Linux distributions via native packages, which makes it easy to install.

If you want to have one system to support all your containers, a popular option is Logspout, which runs in a separate container, talks to the Docker daemon, and logs all of the system’s container logs to syslog (UDP, TCP, TLS). The advantage of this approach is that it does not preclude docker logs, but it does require that you set up log rotation. It also will not block starting up any containers when the remote syslog server is unavailable!

Finally, while you really should be capturing your logs somewhere, there are rare situations where you simply don’t want any logging. You can turn them off completely using the --log-driver=none switch.

Container Stats

Let’s start with the CLI tools that ship with Docker itself. The docker CLI has an endpoint for viewing stats of running containers. The command-line tool can stream from this endpoint and every few seconds report back on one or more listed containers, giving basic statistics information about what’s happening. docker stats, like the Linux top command, takes over the current terminal and updates the same lines on the screen with the current information. It’s hard to show that in print so we’ll just give an example, but this updates every few seconds by default.

$ docker stats b668353c3af5
CONTAINER     CPU %    MEM USAGE/LIMIT    MEM %  NET I/O    BLK I/O    PIDS
b668353c3af5  1.50%    60.39MiB/200MiB  30.19% 335MB/9.1GB  45.4MB/0B  17

$ docker run -d ubuntu:latest sleep 1000
91c86ec7b33f37da9917d2f67177ebfaa3a95a78796e33139e1b7561dc4f244a

$ curl --unix-socket /var/run/docker.sock \
    http://v1/containers/91c86ec7b33f/stats

$ curl --unix-socket /var/run/docker.sock \
    http://v1/containers/91c86ec7b33f/stats | head -1

$ curl --unix-socket /var/run/docker.sock \
    http://v1/containers/91c86ec7b33f/stats \
    | head -1 | python -m json.tool

{
    "blkio_stats": {
        "io_merged_recursive": [],
        "io_queue_recursive": [],
        "io_service_bytes_recursive": [
            {
                "major": 8,
                "minor": 0,
                "op": "Read",
                "value": 6098944
            },
...
        ],
        "io_service_time_recursive": [],
        "io_serviced_recursive": [
            {
                "major": 8,
                "minor": 0,
                "op": "Read",
                "value": 213
            },
...
        ],
        "io_time_recursive": [],
        "io_wait_time_recursive": [],
        "sectors_recursive": []
    },
    "cpu_stats": {
        "cpu_usage": {
...
        },
        "system_cpu_usage": 1884140000000,
        "throttling_data": {
            "periods": 0,
            "throttled_periods": 0,
            "throttled_time": 0
        }
    },
    "memory_stats": {
        "failcnt": 0,
        "limit": 1035853824,
        "max_usage": 7577600,
        "stats": {
...
            "total_active_anon": 1368064,
            "total_active_file": 221184,
            "total_cache": 6148096,
            "total_inactive_anon": 24576,
            "total_inactive_file": 5890048,
            "total_mapped_file": 2215936,
            "total_pgfault": 2601,
            "total_pgmajfault": 46,
            "total_pgpgin": 2222,
            "total_pgpgout": 390,
            "total_rss": 1355776,
            "total_unevictable": 0,
            "unevictable": 0
        },
        "usage": 7577600
    },
    "network": {
        "rx_bytes": 936,
        "rx_dropped": 0,
        "rx_errors": 0,
        "rx_packets": 12,
        "tx_bytes": 468,
        "tx_dropped": 0,
        "tx_errors": 0,
        "tx_packets": 6
    },
    "read": "2018-02-11T15:20:22.930379289-08:00"
}

Container Health Checks

As with any other application, when you launch a container it is possible that it will start and run, but never actually enter a healthy state where it could receive traffic. Production systems also fail and your application may become unhealthy at some point during its life, so you need to be able to deal with that.

Many production environments have standardized ways to health-check applications. Unfortunately, there’s no clear standard for how to do that across organizations and so it’s unlikely that many companies do it in the same way. For this reason, monitoring systems have been built to handle that complexity so that they can work in a lot of different production systems. It’s a clear place where a standard would be a big win.

To help remove this complexity and standardize on a universal interface, Docker has added a health-check mechanism. Following the shipping container metaphor, Docker containers should really look the same to the outside world no matter what is inside the container, so Docker’s health-check mechanism not only standardizes health checking for containers, but also maintains the isolation between what is inside the container and what it looks like on the outside. This means that containers from Docker Hub or other shared repositories can implement a standardized health-checking mechanism and it will work in any other Docker environment designed to run production containers.

Health checks are a build-time configuration item and are created with a check definition in the Dockerfile. This directive tells the Docker daemon what command it can run inside the container to ensure the container is in a healthy state. As long as the command exits with a code of zero (0), Docker will consider the container to be healthy. Any other exit code will indicate to Docker that the container is not in a healthy state, at which point appropriate action can be taken by a scheduler or monitoring system.

We will be using the following project to explore Docker Compose in a few chapters. But, for the moment, it includes a useful example of Docker health checks. Go ahead and pull down a copy of the code and then navigate into the rocketchat-hubot-demo/mongodb/docker/ directory:

$ git clone https://github.com/spkane/rocketchat-hubot-demo.git \
    --config core.autocrlf=input
$ cd rocketchat-hubot-demo/mongodb/docker

In this directory, you will see a Dockerfile and a script called docker-healthcheck. If you view the Dockerfile, this is all that you will see:

<code class="k">FROM</code> <code class="s2">mongo:3.2</code>

<code class="k">COPY</code> <code class="s2">docker-healthcheck /usr/local/bin/</code>

<code class="k">HEALTHCHECK CMD</code> <code class="s2">["docker-healthcheck"]</code>

It is very short because we are basing this on the upstream Mongo image, and our image inherits a lot of things from that including the entry point, default command, and port to expose:

ENTRYPOINT ["docker-entrypoint.sh"]
EXPOSE 27017
CMD ["mongod"]

So, in our Dockerfile we are only adding a single script that can health-check our container, and defining a health-check command that runs that script.

You can build the container like this:

$ docker build -t mongo-with-check:3.2 .
Sending build context to Docker daemon  3.072kB
Step 1/3 : FROM mongo:3.2
 ---> 56d7fa068c3d
Step 2/3 : COPY docker-healthcheck /usr/local/bin/
 ---> 217c13baf475
Step 3/3 : HEALTHCHECK CMD ["docker-healthcheck"]
 ---> Running in d18658520abf
Removing intermediate container d18658520abf
 ---> f69ff65ac29c
Successfully built f69ff65ac29c
Successfully tagged mongo-with-check:3.2

And then run the container and looking at the docker ps output:

$ docker run -d --name mongo-hc mongo-with-check:3.2
1ad213256ab2f24fd65d91e5ce47c8a6c3c47a74cf0251e3afa6fdfc2fbadf0e

$ docker ps
... IMAGE                ... STATUS                         PORTS     ...
... mongo-with-check:3.2 ... Up 1 second (health: starting) 27017/tcp ...

You should notice that the STATUS column now has a health section in parentheses. Initially this will display health: starting as the container is starting up. You can change the amount of time that Docker waits for the container to initialize using the --health-start-period argument to docker run. The status will change to healthy once the container is up and the health check is successful.

$ docker ps
... IMAGE                ... STATUS                      PORTS     ...
... mongo-with-check:3.2 ... Up About a minute (healthy) 27017/tcp ...

You can query this status directly, using the docker inspect command.

$ docker inspect --format='{{.State.Health.Status}}' mongo-hc
healthy

$ docker inspect --format='{{json .State.Health}}' mongo-hc | jq

{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    ...
  ]
}

If your container began failing its health check, the status would change to unhealthy and you could then determine how to handle the situation.

$ docker ps
... IMAGE                ... STATUS                   PORTS     ...
... mongo-with-check:3.2 ... Up 9 minutes (unhealthy) 27017/tcp ...

This feature is very useful, and you should strongly consider using it in all of your containers. This will help you improve both the reliability of your environment and the visibility you have into how things are running in it. It is also supported by most production schedulers and many monitoring systems, so it should be easy to implement.

Docker Events

The dockerd daemon internally generates an events stream around the container lifecycle. This is how various parts of the system find out what is going on in other parts. You can also tap into this stream to see what lifecycle events are happening for containers on your Docker server. This, as you probably expect by now, is implemented in the docker CLI tool as another command-line argument. When you run this command, it will block and continually stream messages to you. Behind the scenes, this is a long-lived HTTP request to the Docker API that returns messages in JSON blobs as they occur. The docker CLI tool decodes them and prints some data to the terminal.

This events stream is useful in monitoring scenarios or in triggering additional actions, like wanting to be alerted when a job completes. For debugging purposes, it allows you to see when a container died even if Docker restarts it later. Down the road, this is a place where you might also find yourself directly implementing some tooling against the API. Here’s how we use it on the command line:

$ docker events
2018-02-18T14:00:39-08:00 1b3295bf300f: (from 0415448f2cc2) die

2018-02-18T14:00:39-08:00 1b3295bf300f: (from 0415448f2cc2) stop

2018-02-18T14:00:42-08:00 1b3295bf300f: (from 0415448f2cc2) start

In this example, we initiated a stop signal with docker stop, and the events stream logs this as a “die” message. The “die” message actually marks the beginning of the shutdown of the container. It doesn’t stop instantaneously. So, following the “die” message is a “stop” message, which is what Docker says when a container has actually stopped execution. Docker also helpfully tells us the ID of the image that the container is running on. This can be useful for tying deployments to events, for example, because a deployment usually involves a new image.

Once the container was completely down, we initiated a docker start to tell it to run again. Unlike the “die/stop” operations, this is a single command that marks the point at which the container is actually running. We don’t get a message telling us that someone explicitly started it. So what happens when we try to start a container that fails?

2015-02-18T14:03:31-08:00 e64a279663aa: (from e426f6ef897e) die

Note that here the container was actually asked to start, but it failed. Rather than seeing a “start” and a “die,” all we see is a “die.”

If you have a server where containers are not staying up, the docker events stream is pretty helpful in seeing what’s going on and when. But if you’re not watching it at the time, Docker very helpfully caches some of the events and you can still get at them for some time afterward. You can ask it to display events after a time with the --since option, or before with the --until option. You can also use both to limit the window to a narrow scope of time when an issue you are investigating may have occurred. Both options take ISO time formats like those in the previous example (e.g., 2018-02-18T14:03:31-08:00).

cAdvisor

docker stats and docker events are useful but don’t yet get us graphs to look at. And graphs are pretty helpful when we’re trying to see trends. Of course, other people have filled some of this gap. When you begin to explore the options for monitoring Docker, you will find that many of the major monitoring tools now provide some functionality to help you improve the visibility into your containers’ performance and ongoing state.

In addition to the commercial tooling provided by companies like DataDog, GroundWork, and New Relic, there are plenty of options for free, open source tools like Prometheus or even Nagios. We’ll talk about Prometheus in the next section, but first we’ll start with a nice offering from Google. A few years ago they released their own internal container advisor as a well-maintained open source project on GitHub, called cAdvisor. Although cAdvisor can be run outside of Docker, by now you’re probably not surprised to hear that the easiest implementation of cAdvisor is to simply run it as a Docker container.

To install cAdvisor on most Linux systems, all you need to do is run this code:

$ docker run \
    --volume=/:/rootfs:ro \
    --volume=/var/run:/var/run:rw \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:ro \
    --publish=8080:8080 \
    --detach=true \
    --name=cadvisor \
    google/cadvisor:latest

Unable to find image 'google/cadvisor:latest' locally
Pulling repository google/cadvisor
f0643dafd7f5: Download complete
...
ba9b663a8908: Download complete
Status: Downloaded newer image for google/cadvisor:latest
f54e6bc0469f60fd74ddf30770039f1a7aa36a5eda6ef5100cddd9ad5fda350b

Once you have done this, you will be able to navigate to your Docker host on port 8080 to see the cAdvisor web interface (i.e., http://172.17.42.10:8080/) and the various detailed charts it has for the host and individual containers (see Figure 6-1).

Figure 6-1. cAdvisor CPU graphs

cAdvisor provides a REST API endpoint, which can easily be queried for detailed information by your monitoring systems:

$ curl http://172.17.42.10:8080/api/v1.3/containers/

{
  "name": "/",
  "subcontainers": [
    {
      "name": "/docker"
    }
  ],
  "spec": {
    "creation_time": "2015-04-05T00:05:40.249999996Z",
    "has_cpu": true,
    "cpu": {
      "limit": 1024,
      "max_limit": 0,
      "mask": "0-7"
    },
    "has_memory": true,
    "memory": {
      "limit": 2095771648,
      "swap_limit": 1073737728
    },
    "has_network": true,
    "has_filesystem": true,
    "has_diskio": true
  },
  "stats": [
    {
      "timestamp": "2015-04-05T00:26:50.679218419Z",
      "cpu": {
        "usage": {
          "total": 123375166639,
          "per_cpu_usage": [
            41967365270,
...
            9229283688
          ],
          "user": 22990000000,
          "system": 43890000000
        },
        "load_average": 0
      },
      "diskio": {},
      "memory": {
        "usage": 1910095872,
        "working_set": 1025523712,
        "container_data": {
...
        },
        "hierarchical_data": {
...
        }
      },
      "network": {
...
      },
      "filesystem": [
        {
          "device": "/dev/sda1",
          "capacity": 19507089408,
          "usage": 2070806528,
...
        }
      ],
      "task_stats": {
        "nr_sleeping": 0,
        "nr_running": 0,
        "nr_stopped": 0,
        "nr_uninterruptible": 0,
        "nr_io_wait": 0
      }
    },
    ...
    }
  ]
}

As you can see from the preformatted output, the amount of detail provided here should be sufficient for many of your graphing and monitoring needs.