JSONPath query syntax

JSONPath query syntax is based on XPath (which is used to search XML documents). Table 6-1 lists some JSONPath queries based on our cities example.

These example queries use JSONPath keywords and symbols:

• $ represents the document root-level object.

• .. returns all elements and subelements that have a particular name.

• [] with an index is an Array query, and the index is based on the JavaScript slice() function. The Mozilla Developer Network (MDN) provides a full description. Here’s a brief overview of JSONPath slice():

  • It provides the ability to select a portion of an Array.

  • The begin parameter (as with JS slice()) is the beginning index, is zero-based, and defaults to zero if omitted.

  • The end parameter (as with JS slice()) is the end index (noninclusive), and defaults to the end of the Array if omitted.

  • The step parameter (added by JSONPath slice()) represents the step, and defaults to 1. A step value of 1 returns all Array elements specified by the begin and end parameters; a value of 2 returns every other (or second) element, and so on.

• @ represents the current element.

• [?(…)] enables a conditional search. The code inside the parentheses can be any valid JS expression, including conditionals (e.g., == or >) and Regular Expressions.

JSONPath online tester

A couple of online JSONPath testers enable you to practice JSONPath queries before writing a single line of code. I like the tester provided by Kazuki Hamasaki. Just paste in the data/cities-weather.json document (from the Chapter 6 code examples) in the left text box, and enter a JSONPath query. The results appear in the text box on the righthand side of the page as shown in Figure 6-2.

Figure 6-2. JSONPath Online Evaluator with OpenWeather API data

You’ll notice that only the data values are returned in the JSONPath results text box, and that the keys are not returned.

JSONPath Unit Test

The Unit Test in Example 6-2 exercises several of the example JSONPath queries that were shown earlier. This code leverages the jsonpath Node.js module to search against the JSON data returned by the Cities API that runs on your local machine. See https://github.com/dchester/jsonpath for a detailed description of the jsonpath module.

'use strict';

/* Attribution: Cities Weather data provided by OpenWeatherMap API
   ([http://openweathermap.org]) under Creative Commons Share A Like
   License (https://creativecommons.org/licenses/by-sa/4.0).
   Changes were made to the data to work with json-server.
   This does not imply an endorsement by the licensor.

   This code is distributed under Creative Commons Share A Like License.
*/

var expect = require('chai').expect;
var jp = require('jsonpath');
var unirest = require('unirest');

describe('cities-jsonpath', function() {
  var req;

  beforeEach(function() {
    req = unirest.get('http://localhost:5000/cities')
      .header('Accept', 'application/json');
  });

  it('should return a 200 response', function(done) {
    req.end(function(res) {
      expect(res.statusCode).to.eql(200);
      expect(res.headers['content-type']).to.eql(
        'application/json; charset=utf-8');
      done();
    });
  });

  it('should return all cities', function(done) {
    req.end(function(res) {
      var cities = res.body;

      expect(cities.length).to.eql(110);
      done();
    });
  });

  it('should return every other city', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var citiesEveryOther = jp.query(cities, '$[0::2]');

      expect(citiesEveryOther[1].name).to.eql('Rosarito');
      expect(citiesEveryOther.length).to.eql(55);
      done();
    });
  });

  it('should return the last city', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var lastCity = jp.query(cities, '$[(@.length-1)]');

      expect(lastCity[0].name).to.eql('Moreno Valley');
      done();
    });
  });

  it('should return the 1st 3 cities', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var citiesFirstThree = jp.query(cities, '$[:3]');
      var citiesFirstThreeNames = jp.query(cities, '$[:3].name');

      expect(citiesFirstThree.length).to.eql(3);
      expect(citiesFirstThreeNames.length).to.eql(3);
      expect(citiesFirstThreeNames).to.eql(['Rancho Palos Verdes',
        'San Pedro', 'Rosarito'
      ]);

      done();
    });
  });

  it('should return cities within a temperature range', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var citiesTempRange = jp.query(cities,
        '$[?(@.main.temp >= 84 && @.main.temp <= 85.5)]'
      );

      for (var i = 0; i < citiesTempRange.length; i++) {
        expect(citiesTempRange[i].main.temp).to.be.at.least(84);
        expect(citiesTempRange[i].main.temp).to.be.at.most(85.5);
      }

      done();
    });
  });

  it('should return cities with cloudy weather', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var citiesWeatherCloudy = jp.query(cities,
        '$[?(@.weather[0].main == "Clouds")]'
      );

      checkCitiesWeather(citiesWeatherCloudy);
      done();
    });
  });

  it('should return cities with cloudy weather using regex', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var citiesWeatherCloudyRegex = jp.query(cities,
        '$[?(@.weather[0].main.match(/Clo/))]'
      );

      checkCitiesWeather(citiesWeatherCloudyRegex);
      done();
    });
  });

  function checkCitiesWeather(cities) {
    for (var i = 0; i < cities.length; i++) {
      expect(cities[i].weather[0].main).to.eql('Clouds');
    }
  }
});

Note the following in this example:

• The test sets up the URI and Accept for unirest using Mocha’s beforeEach() method, so that setup occurs in only one place in the code. Mocha executes beforeEach() before running each test (i.e., it) within the context of the describe.

• Each test exercises one or more example JSONPath queries and uses expect-style assertions.

• The calls to the jsonpath module work as follows:

  • jp.query() takes a JavaScript Object and a String-based JSONPath query as parameters, and synchronously returns the result set as a JavaScript Object.

• Each JSONPath query omits the leading .cities because json-server takes the name of the cities Array (from the cities-weather.json file) and adds cities to the URI:

  • The URI address is http://localhost:5000/cities.

  • Use $[:3] to get the first three cities, rather than $.cities[:3].

To run this test from the command line (in a second terminal session), do the following:

cd cities-weather-test

npm test

You should see the following results:

json-at-work => npm test

...

> mocha test

...

cities-jsonpath
  ✓ should return a 200 response
  ✓ should return all cities
  ✓ should return every other city
  ✓ should return the last city
  ✓ should return 1st 3 cities
  ✓ should return cities within a temperature range
  ✓ should return cities with cloudy weather
  ✓ should return cities with cloudy weather using regex

...

If you call console.log() with the cities variable in any of the preceding tests, you’ll see that the jsonpath module returns a valid JSON document with key/value pairs.

JSONPath on other platforms

JSONPath is not limited to JavaScript and Node.js. Most major platforms have excellent support for JSONPath, including these:

• Ruby on Rails

• Python

• Java

There are other good JSONPath libraries are available, but please verify that they follow the syntax mentioned in Stefan Goessner’s article. Otherwise, it’s not really JSONPath. To borrow a phrase from The Princess Bride, “You keep using that word, but I do not think it means what you think it means.”

JSONPath scorecard

Table 6-2 provides a scorecard for JSONPath based on the evaluation criteria from the beginning of this chapter.

JSONPath provides a rich set of set of search features and works across most major platforms. The only downsides are that JSONPath is not a standard and lacks a CLI implementation, but don’t let that stop you from using it. JSONPath enjoys wide community usage and acceptance, and has an excellent online tester. JSONPath reduces the amount of code needed to search and access a JSON document, and gets the subset of data that you need.

JSON Pointer query syntax

For example, consider the following document:

{
  "cities": [
    {
      "id": 5386035,
      "name": "Rancho Palos Verdes"
    },
    {
      "id": 5392528,
      "name": "San Pedro"
    },
    {
      "id": 5358705,
      "name": "Huntington Beach"
    }
  ]
}

Table 6-3 describes the preceding document’s common JSON Pointer query syntax:

JSON Pointer query syntax is quite simple, and works as follows:

• / is a path separator.

• Array indexing is zero-based.

You’ll notice that in the JSON Pointer specification, only the data values are returned when making a query, and that the keys are not returned.

JSON Pointer Unit Test

The Unit Test in Example 6-3 exercises some of the example JSON Pointer queries that were shown earlier. This code leverages the json-pointer Node.js module to search against the cities API. See https://github.com/manuelstofer/json-pointer for a detailed description of the json-pointer module.

'use strict';

/* Attribution: Cities Weather data provided by OpenWeatherMap API
   ([http://openweathermap.org]) under Creative Commons Share A Like
   License (https://creativecommons.org/licenses/by-sa/4.0).
   Changes were made to the data to work with json-server.
   This does not imply an endorsement by the licensor.

   This code is distributed under Creative Commons Share A Like License.
*/

var expect = require('chai').expect;
var pointer = require('json-pointer');
var unirest = require('unirest');

describe('cities-json-pointer', function() {
  var req;

  beforeEach(function() {
    req = unirest.get('http://localhost:5000/cities')
                  .header('Accept', 'application/json');
  });

  it('should return a 200 response', function(done) {
    req.end(function(res) {
      expect(res.statusCode).to.eql(200);
      expect(res.headers['content-type']).to.eql(
                  'application/json; charset=utf-8');
      done();
    });
  });

  it('should return the 1st city', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var firstCity = null;

      firstCity = pointer.get(cities, '/0');
      expect(firstCity.name).to.eql('Rancho Palos Verdes');
      expect(firstCity.weather[0].main).to.eql('Clear');
      done();
    });
  });

  it('should return the name of the 2nd city', function(done) {
    req.end(function(res) {
      var cities = res.body;
      var secondCityName = null;

      secondCityName = pointer.get(cities, '/1/name');
      expect(secondCityName).to.eql("San Pedro");
      done();
    });
  });
});

Note the following in this example:

• Each test runs an example JSON Pointer query and leverages expect-style assertions.

• The calls to the json-pointer module work as follows:

  • pointer.get() takes a JavaScript Object and a String-based JSON Pointer query as parameters, and synchronously returns the result set as a JavaScript Object.

• Each JSON Pointer query omits the leading .cities because json-server takes the name of the cities Array (from the cities-weather.json file) and adds cities to the URI:

  • The URI address is http://localhost:5000/cities.

  • Use /0 to get the first city, rather than /cities/0.

To run this test from the command line, do the following:

cd cities-weather-test

npm test

You should see the following results:

json-at-work => npm test

...

> mocha test

...

cities-json-pointer
  ✓ should return a 200 response
  ✓ should return the 1st city
  ✓ should return the name of the 2nd city

...

If you invoke console.log() on the firstCity variable in the should return the 1st city test above, you’ll see that the json-pointer module returns a valid JSON document with key/value pairs.

JSON Pointer on other platforms

In addition to Node.js, most major platforms have a JSON Pointer library:

• Ruby on Rails

• Python

• Java—Jackson currently supports JSON Pointer, but JavaEE 8 will provide JSON Pointer support as part of JSR 374, Java API for JSON Processing 1.1.

Several tools claim to implement JSON Pointer, but they really don’t follow the JSON Pointer specification. When evaluating a JSON Pointer library or tool, be sure it follows RFC 6901. Again, if it doesn’t expressly mention RFC 6901, it’s not JSON Pointer.

JSON Pointer scorecard

Table 6-4 shows a scorecard for JSON Pointer based on our criteria.

JSON Pointer provides a limited set of search capabilities. Each query returns only a single field from a JSON document. JSON Pointer’s main purpose is to support JSON Schema’s $ref syntax.

Integration with cURL

Many people in the UNIX community use cURL to make HTTP calls to Web APIs from the command line. cURL provides the ability to communicate over multiple protocols in addition to HTTP. To install cURL, please see “Install cURL” in Appendix A.

We’ll start by using cURL to make a GET request from the command against the Cities API as follows:

curl  -X GET 'http://localhost:5000/cities'

Now that we’re able to get a JSON response from the Cities API, let’s pipe the content to jq to filter the Cities API data from the command line. Here’s a simple example:

curl  -X GET 'http://localhost:5000/cities' | jq .[0]

Run this command, and you should see the following:

Note the following in this example:

• cURL makes an HTTP GET call to the OpenWeatherMap API and pipes the JSON response to Standard Output.

• jq reads the JSON from Standard Input, selects the first city from the API, and outputs the JSON to Standard Output.

cURL is a valuable and powerful part of an API developer’s toolkit. cURL also provides the ability to test an API with all the main HTTP verbs (GET, POST, PUT, and DELETE). We’ve just scratched the surface with cURL; visit the main site to learn more.

jq query syntax

Table 6-5 shows some basic jq queries.

Here’s how to execute a jq query to get the last city at the command line:

cd chapter-6/data

jq '.cities[-1]' cities-weather.json

You should see the following:

Let’s go deeper with a more concrete example.

jq online tester—jqPlay

jqPlay is a web-based tester for jq, and provides the ability to iteratively test jq queries against JSON data. To test jqPlay, do the following to get a new Array of Objects that contain the id and name of the first three cities:

1. Visit https://jqplay.org and paste the contents of the chapter-6/data/cities-weather.json file into the JSON text area on the left.

2. Paste the following jq query into the Filter text box: [[].cities[0:3] | .[] | { id, name }]

You should see the screen in Figure 6-3.

Figure 6-3. Search OpenWeather API data with jqPlay

Here’s a breakdown of the [.cities[0:3] | .[] | { id, name }] query:

• The | enables you to chain your filters.

• .cities[0:3] selects the first three elements from the cities Array as a subarray.

• .[] returns all elements from the subarray.

• { id, name } selects only the id and name fields:

  • The curly braces ({ and }) tell jq to create a new Object.

  • The id and name tell jq to include only these fields in the new Object.

• The surrounding Array braces ([ and ]) convert the result set to an Array.

Scroll to the bottom of the jqplay page, and you’ll see that it has a cheat sheet with links to more examples and documentation, as shown in Figure 6-4.

Figure 6-4. jq cheat sheet on jqPlay

jq-tutorial

In addition to an online tester, the Node.js community has contributed a nice jq tutorial, which is available on the npm repository. Install this tutorial as follows:

npm install -g jq-tutorial

Run jq-tutorial from the command line, and you should see this:

This shows all the available jq tutorials. Then, choose one of the tutorials like this:

jq-tutorial objects

This tutorial will show how to use objects with jq. Follow each learning path, and increase your overall jq skill level.

jq Unit Test

The Unit Test in Example 6-4 exercises several of the example jq queries that were shown earlier. This code leverages the node-jq Node.js module to search against the JSON data returned by the Cities API that runs on your local machine. See the node-jq documentation on GitHub for a detailed description.

'use strict';

/* Attribution: Cities Weather data provided by OpenWeatherMap API
   ([http://openweathermap.org]) under Creative Commons Share A Like
   License (https://creativecommons.org/licenses/by-sa/4.0).
   Changes were made to the data to work with json-server.
   This does not imply an endorsement by the licensor.

   This code is distributed under Creative Commons Share A Like License.
*/

var expect = require('chai').expect;
var jq = require('node-jq');
var unirest = require('unirest');
var _ = require('underscore');


describe('cities-jq', function() {
  var req;

  beforeEach(function() {
    req = unirest.get('http://localhost:5000/cities')
      .header('Accept', 'application/json');
  });

  it('should return a 200 response', function(done) {
    req.end(function(res) {
      expect(res.statusCode).to.eql(200);
      expect(res.headers['content-type']).to.eql(
        'application/json; charset=utf-8');
      done();
    });
  });

  it('should return all cities', function(done) {
    req.end(function(res) {
      var cities = res.body;

      expect(cities.length).to.eql(110);
      done();
    });
  });

  it('should return the last city', function(done) {
    req.end(function(res) {
      var cities = res.body;

      jq.run('.[-1]', cities, {
          input: 'json'
        })
        .then(function(lastCityJson) { // Returns JSON String.
          var lastCity = JSON.parse(lastCityJson);
          expect(lastCity.name).to.eql('Moreno Valley');
          done();
        })
        .catch(function(error) {
          console.error(error);
          done(error);
        });
    });
  });

  it('should return the 1st 3 cities', function(done) {
    req.end(function(res) {
      var cities = res.body;

      jq.run('.[:3]', cities, {
          input: 'json'
        })
        .then(function(citiesFirstThreeJson) { // Returns JSON String.
          var citiesFirstThree = JSON.parse(citiesFirstThreeJson);
          var citiesFirstThreeNames = getCityNamesOnly(
            citiesFirstThree);

          expect(citiesFirstThree.length).to.eql(3);
          expect(citiesFirstThreeNames.length).to.eql(3);
          expect(citiesFirstThreeNames).to.eql([
            'Rancho Palos Verdes',
            'San Pedro', 'Rosarito'
          ]);

          done();
        })
        .catch(function(error) {
          console.error(error);
          done(error);
        });
    });
  });

  function getCityNamesOnly(cities) {
    return _.map(cities,
      function(city) {
        return city.name;
      });
  }

  it('should return cities within a temperature range', function(done) {
    req.end(function(res) {
      var cities = res.body;

      jq.run(
          '[.[] | select (.main.temp >= 84 and .main.temp <= 85.5)]',
          cities, {
            input: 'json'
          })
        .then(function(citiesTempRangeJson) { // Returns JSON String.
          var citiesTempRange = JSON.parse(citiesTempRangeJson);

          for (var i = 0; i < citiesTempRange.length; i++) {
            expect(citiesTempRange[i].main.temp).to.be.at.least(
              84);
            expect(citiesTempRange[i].main.temp).to.be.at.most(
              85.5);
          }

          done();
        })
        .catch(function(error) {
          console.error(error);
          done(error);
        });
    });
  });

  it('should return cities with cloudy weather', function(done) {
    req.end(function(res) {
      var cities = res.body;

      jq.run(
          '[.[] | select(.weather[0].main == "Clouds")]',
          cities, {
            input: 'json'
          })
        .then(function(citiesWeatherCloudyJson) { // Returns JSON String.
          var citiesWeatherCloudy = JSON.parse(
            citiesWeatherCloudyJson);

          checkCitiesWeather(citiesWeatherCloudy);

          done();
        })
        .catch(function(error) {
          console.error(error);
          done(error);
        });
    });
  });

  it('should return cities with cloudy weather using regex', function(done) {
    req.end(function(res) {
      var cities = res.body;

      jq.run(
          '[.[] | select(.weather[0].main | test("^Clo"; "i"))]',
          cities, {
            input: 'json'
          })
        .then(function(citiesWeatherCloudyJson) { // Returns JSON String.
          var citiesWeatherCloudy = JSON.parse(
            citiesWeatherCloudyJson);

          checkCitiesWeather(citiesWeatherCloudy);

          done();
        })
        .catch(function(error) {
          console.error(error);
          done(error);
        });
    });
  });

  function checkCitiesWeather(cities) {
    for (var i = 0; i < cities.length; i++) {
      expect(cities[i].weather[0].main).to.eql('Clouds');
    }
  }

});

Note the following in this example:

• The test sets up the URI and Accept for unirest using Mocha’s beforeEach() method, so that setup occurs in only one place in the code. Mocha executes beforeEach() before running each test (i.e., it) within the context of the describe.

• Each test exercises one or more example jq queries and uses expect-style assertions.

• The calls to the node-jq module work as follows. jq.run() does the following:

  • Takes a String-based jq query as the first parameter.

  • Uses an optional second parameter (an Object) that specifies the type of input:

    • { input: 'json' } is a JavaScript Object. The Unit Tests use this option because unirest returns Objects from the HTTP call to the Stub API provided by json-server.

    • { input: 'file' } is a JSON file. This is the default if the caller doesn’t specify an input option.

    • { input: 'string' } is a JSON String.

  • Uses an ES6 JavaScript Promise to asynchronously return the result set as a JSON String. In this case, the Unit Tests all need to do the following:

    • Wrap their code within the then and catch constructs of the Promise.

    • Use JSON.parse() to parse the result into a corresponding JavaScript object structure.

  • Visit the MDN site to learn more about the new Promise syntax.

• Each jq query omits the leading .cities because json-server takes the name of the cities Array (from the cities-weather.json file) and adds cities to the URI:

  • The URI address is http://localhost:5000/cities.

  • Use $[:3] to get the first three cities, rather than $.cities[:3].

To run this test from the command line (in a second terminal session), do the following:

cd cities-weather-test

npm test

You should see the following results:

json-at-work => npm test

...

> mocha test

...

  cities-jq
    ✓ should return a 200 response
    ✓ should return all cities
    ✓ should return the last city
    ✓ should return the 1st 3 cities
    ✓ should return cities within a temperature range
    ✓ should return cities with cloudy weather
    ✓ should return cities with cloudy weather using regex

...

If you call console.log() with the cities variable in any of these tests, you’ll see that the node-jq module returns a valid JSON document with key/value pairs.

jq on other platforms

In addition to Node.js, other major platforms have a jq library:

Ruby

The ruby-jq gem is available at RubyGems.org, and you can also find it on GitHub.

Java

jackson-jq plugs into the Java Jackson library (from Chapter 4).

jq scorecard

Table 6-6 shows how jq stacks up against our evaluation criteria.

jq is excellent because it

• Enjoys solid support in most languages.

• Has excellent documentation.

• Provides a rich set of search and filtering capabilities.

• Can pipe query results to standard UNIX CLI tools (for example, sort, grep, and uniq).

• Works great on the command line with the widely used cURL HTTP client.

• Has a fantastic online tester. jqPlay enables you to test jq queries from a simple web interface. This rapid feedback enables you to iterate to a solution before writing any code.

• Has a useful interactive tutorial (see the “jq-tutorial” section).

The only downside to jq is the initial learning curve. The sheer number of options along with the query syntax can seem overwhelming at first, but the time you spend to learn jq is well worth it.

We’ve covered the basics of jq in this chapter. jq has excellent documentation, and you can find more detailed information at the following websites:

• jq Manual

• jq Tutorial

• jq Cookbook

• HyperPolyGlot JSON Tools: Jq

• Ubuntu jq man pages