Promise media types, not objects

Over time, object models are bound to change—and these models are likely to change often for new services. Trying to get all your service consumers to learn and track all your object model changes is not a good idea. And, even if you wanted all API consumers to keep up with your team’s model changes, that means your feature velocity will be limited to that of the slowest API consumer in your ecosystem. This can be especially problematic when your API consumers are customers and not fellow colleagues within the same company.

Instead of exposing object models in your APIs, promise standard message formats (e.g., HTML, Atom, HAL, Cj, Siren, etc.). These formats don’t require consumers to understand your service’s internal object model. That means you’re free to modify your internal model without breaking your promise to API consumers. This also means providers will need to handle the task of translating internal domain data into external message formats, but we covered that already in Chapter 3, The Representor Pattern.

Well-designed formats should allow API designers to safely introduce semantic changes (e.g., data carried within the message model), and well-implemented API consumers will be able to parse/render these content changes without the need for code updates. These same formats might support structural changes to messages (e.g., format extensions) in order to safely introduce changes that can be ignored by clients that do not understand them.

Document link identifiers, not URLs

Your API should not bake static URLs into the design. URLs are likely to change over time, especially in cases where your initial service is running in a test bed or small online community to start. Tricking API consumers into baking explicit URLs into their source code increases the likelihood that their code will become obsolete, and that forces consumers into unwanted recode-test-deploy cycles if and when your URLs change.

Instead, your API design should promise to support a named operation (shoppingCartCheckOut, computeTax, findCustomer) instead of promising exact addresses for those operations (e.g., http://api.example.org/findCustomer). Documenting (and promising operations by name) is a much more stable and maintainable design feature.

If you want new operations to be ignored by existing clients, make it part of the structure of the message (e.g., <findCustomer … />). However, when you want the operation to be automatically parsed and/or rendered, favor formats that allow you to include the operation identifiers as part of the message’s semantic content (e.g., <operation name="findCustomer" … />). Good candidates for semantic identifers are properties such as id, name, class, and rel.

Publish vocabularies, not models

The notion of canonical models has been around a long time—especially in large enterprise IT shops. The hope is that, with enough hard work, a single grand model of the company’s business domain will be completely defined and properly described so that everyone (from the business analyst on through to the front-line developer) will have a complete picture of the entire company’s domain data. But this never works out.

The two things conspiring against canonical models are (1) scope and (2) time. As the scope of the problem grows (e.g., the company expands, product offerings increase, etc.), the model becomes unwieldy. And as time goes on, even simple models experience modifications that complicate a single-model view of the world. The good news is there is another way to solve this problem: vocabularies.

Once you move to promising formats instead of object models (see “Promise media types, not objects”), the work of providing shared understanding of your API’s domain data and actions needs to be kept somewhere else. A great place for this is in a shared vocabulary. Eric Evans refers to this using the name ubiquitous language—a common rigorous language shared between domain experts and system implementors. By focusing on a shared vocabulary designers can constantly probe domain experts for clarification, and developers can implement features and share data with a high degree of confidence.

Another important reason to rely on vocabularies is that you can define consistent binding rules between the vocabulary terms and the output formats used in your API. For example, you might document that data element names in the vocabulary will always appear in the class property of HTML responses and the name property of Collection+JSON responses, and so forth. This also helps API providers and consumers write general-use code that will work even when new vocabulary terms are added over time.

So, when designing APIs you should:

• Promise media types, not objects

• Document link identifiers, not URLs

• Publish vocabularies, not models

Don’t take things away

One of the most important aspects of maintaining a backward-compatible service implementation is don’t take things away such as operations or data elements. In API projects I work on, I make this an explicit promise. Once API consumers know that you will not take things away, the value of your API will go up. That’s because API consumers will be assured that, even when you add new data elements and actions to the API, the existing API consumer code will still be valid.

One big reason to make this promise is that it allows API consumer teams to move at their own pace. They don’t need to stop current sprints or feature work in order to deal with potentially breaking changes from some service API they are using in their application. That also means the services don’t need to wait for the slowest API consumer team before they introduce new elements and actions in the API. This loose-coupling in the API update process can result in overall faster development processes since it reduces potential blocking scenarios.

So, what does this backward-compatibility promise look like? Here’s an example I learned from Jason Rudolph at GitHub a few years ago. This is an example of what they call evolutionary design for APIs. He says:

When people are building on top of our API, we’re really asking them to trust us with the time they’re investing in building their applications. And to earn that trust, we can’t make changes [to the API] that would cause their code to break.

Here’s an example of evolutionary design in action. They supported an API response that returned the current status of an account’s request rate limit. It looked like this:

*** REQUEST ***
GET rate_limit
Accept: application/vnd.github+json
...

*** RESPONSE ***
200 OK HTTP/1.1
Content-Type: application/vnd.github+json
...

{
  "rate" {
    "limit" : 5000,
    "remaining : 4992,
    "reset" : 1379363338
  }
}

Over time, the GitHub team learned that this response was more coarse-grained than was needed. It turns out they wanted to separate search-related rate limits from typical core API calls. So the new design would look like this:

*** REQUEST ***
GET rate_limit
Accept: application/vnd.github+json
...

*** RESPONSE ***
200 OK HTTP/1.1
Content-Type: application/vnd.github+json
...

{
  "resources" : {
    "core" : {
      "limit" : 5000,
      "remaining : 4992,
      "reset" : 1379363338
    },
    "search" : {
      "limit" : 20,
      "remaining : 19,
      "reset" : 1379361617
    }
  }
}

Now, they had a dilemma on their hands. How could they make this important change to the interface without breaking existing implementations? Their solution was, I think, quite smart. Rather than changing the response body, they extended it. The new response for the rate_limit request now looks like this:

*** REQUEST ***
GET rate_limit
Accept: application/vnd.github+json
...

*** RESPONSE ***
200 OK HTTP/1.1
Content-Type: application/vnd.github+json
...

{
  "rate" : { 
    "limit" : 5000,
    "remaining : 4992,
    "reset" : 1379363338
  },
  "resources" : { 
    "core" : {
      "limit" : 5000,
      "remaining : 4992,
      "reset" : 1379363338
    },
    "search" : {
      "limit" : 20,
      "remaining : 19,
      "reset" : 1379361617
    }
  }
}

Notice that GitHub applied a structural change to the response. The original structure () and the new structure () both appear in the response message. This results in a change that can be safely ignored by clients that don’t understand the new, detailed structural element ("resources"). This is just one example of implementing backward-compatibility by not taking things away. The same general approach can be made for links and forms in a response, too.

Don’t change the meaning of things

Another important backward-compatibility principle for service providers is don’t change the meaning of things. That means that, once you publish a link or form with an identifier that tells API consumers what is returned (e.g., <a href="…" rel="users" /> returns a list of users), you should not later use that same identifier to return something completely different (e.g., <a href="…" rel="users" /> later only returns a list of inactive users). Consistency of what the link identifier and/or data element represents is very important for maintaining backward compatibility over time.

In cases where you want to represent some new functionality to the API, it is much better to make a semantic change by adding the new feature. And you should do this without removing any existing functionality. To use the preceding example, if you want to add the ability to return a list of inactive users, it is better to introduce an additional link (and identifier) while maintaining the existing one:

*** REQUEST ***
GET /user-actions HTTP/1.1
Accept: application/vnd.hal+json
...

**** RESPONSE ***
200 OK HTTP/1.1
Content-Type: application/vnd.hal+json
...

{
  "_links" :  {
    "users" : {"href" : "/user-list"},
    "inactive" : {"href" : "/user-list?status=inactive"}
  }
}
...

In cases where the preceding response is used to create a human-driven UI, both the links will appear on the screen and the person running the app can decide which link to select. In the case of a service-only interface (e.g., some middleware that is tasked with collecting a list of users and processing it in some unique way), the added semantic information (e.g., the inactive link) will not be “known” to existing apps and will be safely ignored. In both cases, this maintains backward compatibility and does not break existing implementations.

All new things are optional

Another important change over time principle for service implementors is to make sure all new things are optional. This especially applies to new arguments (e.g., filters or update values)—they cannot be treated as required elements. Also, any new functionality or workflow steps (e.g., you introduce a new workflow step between login and checkout) cannot be required in order to complete the process.

One example of this is similar to the GitHub case just described (see “Don’t take things away”). It is possible that, over time, you’ll find that some new filters are needed when making requests for large lists of data. You might even want to introduce a default page-size to limit the load time of a resource and speed up responsiveness in your API. Here’s how a filter form looks before the introduction of the page-size argument:

*** REQUEST ***
GET /search-form HTTP/1.1
Accept: application/vnd.collection+json
...

*** RESPONSE ***
200 OK HTTP/1.1
Content-Type: application/vnd.collection+json
...

{
  "collection" : {
    "queries" : [
      {
        "rel" : "search"
        "href" : "/search-results",
        "prompt" : Search Form",
        "data" : [
          {
            "name" : "filter",
            "value" : "",
            "prompt" : "Filter",
            "required" : true
          }
        ]
      }
    ]
  }
}

And here is the same response after introducing the page-size argument:

*** REQUEST ***
GET /search-form HTTP/1.1
Accept: application/vnd.collection+json
...

*** RESPONSE ***
200 OK HTTP/1.1
Content-Type: application/vnd.collection+json
...

{
  "collection" : {
    "queries" : [
      {
        "rel" : "search"
        "href" : "/search-results",
        "prompt" : Search Form",
        "data" : [
          {
            "name" : "filter",
            "value" : "",
            "prompt" : "Filter",
            "required" : true
          },
          {
            "name" : page-size",
            "value" : "all",
            "prompt" : "Page Size",
            "required" : false
          }
        ]
      }
    ]
  }
}

In the updated rendition, you can see that the new argument (page-size) was explicitly marked optional ("required" : false). You can also see that the a default value was provided ("value" : "all"). This may seem a bit counterintuitive. The update was introduced in order to limit the number of records sent in responses. So why set the default value to "all"? It is set to "all" because that was the initial promise in the first rendition of the API. We can’t change the results of this request now to only include some of the records. This also follows the don’t change the meaning of things principle.

So, as service implementors, you can go a long way toward maintaining backward compatibility by supporting these three principles:

• Don’t take things away

• Don’t change the meaning of things

• Make new things optional

Code defensively

The first thing API consumers can do is adopt a coding strategy that protects the app from cases where expected data elements and/or actions are missing in a response. This can be accomplished when you code defensively. You can think of this as honoring Postel’s Law (see “TCP/IP’s Robustness Principle”) by being “liberal in what you accept from others.” There are a couple of very simple ways to do this.

For example, when I write client code to process a response, I almost always include code that first checks for the existence of an element before attempting to parse it. Here’s some client code that you’ll likely find in the examples associated with this book:

// handle title
function title() {
  var elm;

  if(hasTitle(global.cj.collection)===true) { 
    elm = domHelper.find("title");
    elm.innerText = global.cj.collection.title;
    elm = domHelper.tags("title");
    elm[0].innerText = global.cj.collection.title;
  }
}

You can see that I first check to see if the collection object has a title property (). If so, I can continue processing it.

Here’s another example where I supply local default values for cases where the service response is missing expected elements (, , , , ) and check for the existence of a property ():

function input(args) {
  var p, lbl, inp;

  p = domHelper.node("p");
  p.className = "inline field";
  lbl = domHelper.node("label");
  inp = domHelper.node("input");
  lbl.className = "data";
  lbl.innerHTML = args.prompt||""; 
  inp.name = args.name||""; 
  inp.className = "value "+ args.className;
  inp.value = args.value.toString()||""; 
  inp.required = (args.required||false); 
  inp.readOnly = (args.readOnly||false); 
  if(args.pattern) { 
    inp.pattern = args.pattern;
  }
  domHelper.push(lbl,p);
  domHelper.push(inp,p);

  return p;
}

There are other examples of coding defensively that I won’t include here. The main idea is to make sure that client applications can continue functioning even when any given response is missing expected elements. When you do this, even most unexpected changes will not cause your API consumer to crash.

Code to the media type

Another important principle for building resilient API consumer apps is to code to the media type. Essentially, this is using the same approach that was discussed in Chapter 3, The Representor Pattern, except that this time, instead of focusing on creating a pattern for converting internal domain data into a standard message format (via a Message Translator), the opposite is the goal for API consumers: convert a standardized message format into a useful internal domain model. By doing this, you can go a long way toward protecting your client application from both semantic and structural changes in the service responses.

For all the client examples I implement in this book, the media type messages (HTML, HAL, Cj, and Siren) are converted into the same internal domain model: the HTML Document Object Model (DOM). The DOM is a consistent model, and writing client-side JavaScript for it is the way most browser-based API clients work.

Here is a short code snippet that shows how I convert Siren entities into HTML DOM objects for rendering in the browser:

  // entities
  function entities() {
    var elm, coll;
    var ul, li, dl, dt, dd, a, p;

    elm = domHelper.find("entities");
    domHelper.clear(elm);

    if(global.siren.entities) {
      coll = global.siren.entities;
      for(var item of coll) {
        segment = domHelper.node("div"); 
        segment.className = "ui segment";

        a = domHelper.anchor({ 
          href:item.href,
          rel:item.rel.join(" "),
          className:item.class.join(" "),
          text:item.title||item.href});
        a.onclick = httpGet;
        domHelper.push(a, segment);

        table = domHelper.node("table"); 
        table.className = "ui very basic collapsing celled table";
        for(var prop in item) {
          if(prop!=="href" &&
            prop!=="class" &&
            prop!=="type" &&
            prop!=="rel") {
            tr = domHelper.data_row({ 
              className:"item "+item.class.join(" "),
              text:prop+" ",
              value:item[prop]+" "
            });
            domHelper.push(tr,table);
          }
        }
        domHelper.push(table, segment, elm); 
      }
    }
  }

It might be a bit tough to see how the HTML DOM is utilized in this example since I use a helper class (the domHelper object) to access most of the DOM functions. But you can see that, for each Siren entity I create an HTML div tag (). I then create an HTML anchor tag () for each item. I set up an HTML <table> to hold the Siren entity’s properties () and add a new table row (<tr>) for each one (). Finally, after completing all the rows in the table, I add the results to the HTML page for visible display ().

This works because all the implementation examples in this book are intended for common HTML browsers. For cases where the target clients are mobile devices or native desktop applications, I need to work out another strategy. One way to handle this is to create reverse representors for each platform. In other words, create a custom Format-to-Domain handler for iOS, Android, and Windows Mobile, etc. Then the same for Linux, Mac, and Windows desktops, and so forth. This can get tedious, though. That’s why using the browser DOM is still appealing and why some mobile apps rely on tools like Apache Cordova, Mono, Appcelerator, and other cross-platform development environments.

Leverage the API vocabulary

Once you start building clients that code to the media type, you’ll find that you still need to know domain-specific details that appear in responses. Things like:

• Does this response contain the list of users I asked for?

• How do I find all the inactive customers?

• Which of these invoice records are overdue?

• Is there a way for me to find all the products that are no longer in stock in the warehouse?

All these questions are domain-specific and are not tied to any single response format like HAL, Cj, or Siren. One of the reasons the HTML browser has been so powerful is that the browser source code doesn’t need to know anything about accounting in order to host an accounting application. That’s because the user driving the browser knows that stuff. The browser is just the agent of the human user. For many API client cases, there is a human user available to interpret and act upon the domain-specific information in API responses. However, there are cases where the API client is not acting as a direct user agent. Instead, it is just a middleware component or utility app tasked with some job by itself (e.g., find all the overdue invoices). In these cases, the client app needs to have enough domain information to complete its job. And that’s where API vocabularies come in.

There are a handful of projects focused on documenting and sharing domain-specific vocabularies over the WWW. One of the best-known examples of this is the Schema.org project (pronounced schema dot org). Schema.org contains lists of common terms for all sorts of domains. Large web companies like Google and Microsoft use Schema.org vocabularies to drive parts of their system.

So what does this all look like? How can you use vocabularies to enable API clients to act on their own safely? Basically, you need to “teach” the API consumer to perform tasks based on some baked-in domain knowledge. For example, I might want to create an API consumer that uses one service to find overdue invoices and then pass that information off to another service for further processing. This means the API consumer needs to “know” about invoices and what it means to be “overdue.” If the API I am using has published a vocabulary, I can look there for the data and action element identifiers I need to perform my work.

As just one example, here’s what that published vocabulary might look like as expressed in a simplified ALPS XML document:

<alps>
  <doc>Invoice Management Vocabluary</doc>
  <link rel="invoice-mgmt" href="api.example.org/profile/invoice-mgmt" />

  <!-- data elements -->
  <descriptor id="invoice-href" />
  <descriptor id="invoice-number" />
  <descriptor id="invoice-status">
    <doc>Valid values are: "active", "closed", "overdue"</doc>
  </descriptor>

  <!-- actions -->
  <descriptor id="invoice-list" type="safe" />
  <descriptor id="invoice-detail" type="safe" />
  <descriptor id="invoice-search" type="safe">
    <descriptor href="#invoice-status" />
  </descriptor>
  <descriptor id="write-invoice" type="unsafe">
  <descriptor href="#invoice-href" />
  <descriptor href="#invoice-number" />
  <descriptor href="#invoice-status">
  </dscriptor>
</alps>

Now when I build my client application, I know that I can “teach” that app to understand how to deal with an invoice record (invoice-number and invoice-status) and know how to search for overdue invoices (use search-invoice with the invoice-status value set to "overdue"). All I need is the starting address for the service and the ability to recognize and execute the search for overdue invoices. The pseudo-code for that example might look like this:

:: DECLARE ::
search-link = "invoice-search" 
search-status = "overdue"
write-invoice = "write-invoice"
invoice-mgmt = "api.example.org/profile/invoice-mgmt"

search-href = "http://api.example.org/invoice-mgmt"
search-accept = "application/vnd.siren+json"

write-href = "http://third-party.example.org/write-invoices"
write-accept = "application/vnd.hal+json"

:: EXECUTE ::
response = REQUEST(search-href AS search-accept)
IF(response.vocabulary IS invoivce-mgmt) THEN 
  FOR-EACH(link IN response)
    IF(link IS search-link) THEN
      invoices = REQUEST(search-link AS search-accept WITH search-status) 
      FOR-EACH(link IN invoices)
        REQUEST(write-href AS write-accept
          FOR write-invoice WITH EACH invoice) 
      END-FOR
    END-IF
  END-FOR
END-IF

:: EXIT ::

Although this is only pseudo-code, you can see the app has been loaded with domain-specific information (). Then, after the initial request is made, the response is checked to see if it promises to use the invoice-mgmt vocabluary (). If that check passes, the app searches all the links in the response to find the search-link and, if found, executes a search for all invoices with the status of overdue (). Finally, if any invoices are returned in that search, they are sent to a new service using the write-invoice action ().

Something to note here is that the defensive coding is on display (the if statements) and the code has only initial URLs memorized—the remaining URLs come from the responses themselves.

Leveraging vocabularies for your API means you can focus on the important aspects (the data elements and actions) and not worry about plumbing details such as URL matching, memorizing the exact location of a data element within a document, etc.

React to link relations for workflow

The last client implementation principle I’ll cover here is to react to link relations for workflow. This means that when working to solve a multistep problem, focus on selected link relation values instead of writing client apps that memorize a fixed set of steps. This is important because memorizing a fixed set of steps is a kind of tight-binding of the client to a fixed sequence of events that may not actually happen at runtime due to transient context issues (e.g., part of the service is down for maintenance, the logged-in user no longer has rights to one of the steps, etc.). Or over time, new steps might be introduced or the order of events might change within the service. These are all reasons not to bake multistep details into your client app.

Instead, since the service you are using has also followed the API principles of document link identifiers, publish vocabularies, and don’t take things away, you can implement a client that is trained to look for the proper identifiers and use vocabulary information to know which data elements need to be passed for each operation. Now, even if the links are moved within a response (or even moved to a different response) your client will still be able to accomplish your goal well into the future.

One way to approach this react to links principle is to isolate all the important actions the client will need to take and simply implement them as standalone, stateless operations. Once that is done, you can write a single routine that (1) makes a request, and (2) inspects that request for one of the known actions, and when found, executes the recognized action.

Following is an example of a Twitter-like quote-bot I created for my 2011 book Building Hypermedia APIs:

/* these are the things this bot can do */
function processResponse(ajax) {
  var doc = ajax.responseXML;

  if(ajax.status===200) {
    switch(context.status) {
      case 'start':
        findUsersAllLink(doc);
        break;
      case 'get-users-all':
        findMyUserName(doc);
        break;
      case 'get-register-link':
        findRegisterLink(doc);
        break;
      case 'get-register-form':
        findRegisterForm(doc);
        break;
      case 'post-user':
        postUser(doc);
        break;
      case 'get-message-post-link':
        findMessagePostForm(doc);
        break;
      case 'post-message':
        postMessage(doc);
        break;
      case 'completed':
        handleCompleted(doc);
        break;
      default:
        alert('unknown status: ['+g.status+']');
        return;
    }
  }
  else {
    alert(ajax.status);
  }
}

In the preceding example, this routine constantly monitors the app’s current internal context.status and, as it changes from one state to another, the app knows just what to be looking for within the current response and/or what action to take in order to advance to the next step in the effort to reach the final goal. For this bot, the goal is to post inspirational quotes to the available social media feed. This bot also knows that it might need to authenticate in order to access the feed, or possibly even create a new user account. Notice the use of the JavaScript switch…case structure here. There is no notion of execution order written into the code—just a set of possible states and related operations to attempt to execute.

Writing clients in this way allows you to create middleware components that can accomplish a set goal without forcing that client to memorize a particular order of events. That means even when the order of things changes over time—as long as the changes are made in a backward-compatible way—this client will still be able to complete its assigned tasks.

So, some valuable principles for implementing clients that support change over time include:

• Code defensively

• Code to the media type

• Leverage the API vocabulary

• React to link relations for workflow