A Brief Introduction to APIs

Although countless books, talks, and guides have been written about the intricacies of REST, GraphQL, JSON, and XML APIs, at their core they are based on a simple concept. An API defines a standardized syntax that allows one piece of software to communicate with another piece of software, even though they might be written in different languages or otherwise structured differently.

This section focuses on web APIs (in particular, APIs that allow a web server to communicate to a browser) and uses the term API to refer specifically to that type. But you may want to keep in mind that, in other contexts, API is also a generic term that can be used to, say, allow a Java program to communicate with a Python program running on the same machine. An API does not always have to be “across the internet” and does not necessarily involve any web technologies.

Web APIs are most often used by developers who are using a well-advertised and documented public service. For example, ESPN provides APIs for athlete information, game scores, and more. Google has dozens of APIs in its Developers section for language translations, analytics, and geolocation.

The documentation for these APIs typically describes routes or endpoints, as URLs that you can request, with variable parameters, either in the path of the URL or as GET parameters.

For example, the following provides pathparam as a parameter in the route path:

http://example.com/the-api-route/pathparam

And this provides pathparam as the value for the parameter param1:

http://example.com/the-api-route?param1=pathparam

Both methods of passing variable data to the API are frequently used, although, like many topics in computer science, philosophic debate has raged on when and where variables should be passed through the path or through the parameters.

The response from the API is usually returned in a JSON or XML format. JSON is far more popular in modern times than XML, but you may still see some XML responses. Many APIs allow you to change the response type, usually with the use of another parameter to define which type of response you would like.

Here’s an example of a JSON-formatted API response:

{"user":{"id": 123, "name": "Ryan Mitchell", "city": "Boston"}}

Here’s an example of an XML-formatted API response:

<user><id>123</id><name>Ryan Mitchell</name><city>Boston</city></user>

ip-api.com provides an easy-to-use and simple API that translates IP addresses to actual physical addresses. You can try a simple API request by entering the following in your browser:¹

http://ip-api.com/json/50.78.253.58

This should produce a response like the following:

{"ip":"50.78.253.58","country_code":"US","country_name":"United States",
  "region_code":"MA","region_name":"Massachusetts","city":"Boston",
  "zip_code":"02116","time_zone":"America/New_York","latitude":42.3496,
  "longitude":-71.0746,"metro_code":506}

Notice that the request contains the parameter json in the path. You can request an XML or CSV response by changing this parameter accordingly:

http://ip-api.com/xml/50.78.253.58
http://ip-api.com/csv/50.78.253.58

http://example.com/comments?post=123

{"title": "Great post about APIs!", "body": "Very informative. Really helped me 
out with a tricky technical challenge I was facing. Thanks for taking the time 
to write such a detailed blog post about PUT requests!", "author": {"name": "Ryan 
Mitchell", "website": "http://pythonscraping.com", "company": "O'Reilly Media"}}

<user><firstname>Ryan</firstname><lastname>Mitchell</lastname><username>Kludgist
</username></user>

{"user":{"firstname":"Ryan","lastname":"Mitchell","username":"Kludgist"}}

<user firstname="ryan" lastname="mitchell" username="Kludgist"></user>

{"success": true}

{"error": {"message": "Something super bad happened"}}

Parsing JSON

In this chapter, you’ve looked at various types of APIs and how they function, and you’ve looked at sample JSON responses from these APIs. Now let’s look at how to parse and use this information.

At the beginning of the chapter you saw the example of the ip-api.com API, which resolves IP addresses to physical addresses:

http://ip-api.com/json/50.78.253.58

You can take the output of this request and use Python’s JSON-parsing functions to decode it:

import json
from urllib.request import urlopen

def getCountry(ipAddress):
    response = urlopen('http://ip-api.com/json/'+ipAddress).read()
    ​    .decode('utf-8')
    responseJson = json.loads(response)
    return responseJson.get('country_code')

print(getCountry('50.78.253.58'))

This prints the country code for the IP address 50.78.253.58.

The JSON parsing library used is part of Python’s core library. Just type in import json at the top, and you’re all set! Unlike many languages that might parse JSON into a special JSON object or JSON node, Python uses a more flexible approach and turns JSON objects into dictionaries, JSON arrays into lists, JSON strings into strings, and so forth. In this way, it is extremely easy to access and manipulate values stored in JSON.

The following gives a quick demonstration of how Python’s JSON library handles the values that might be encountered in a JSON string:

import json

jsonString = '{"arrayOfNums":[{"number":0},{"number":1},{"number":2}],
               "arrayOfFruits":[{"fruit":"apple"},{"fruit":"banana"},
                               {"fruit":"pear"}]}'
jsonObj = json.loads(jsonString)

print(jsonObj.get('arrayOfNums'))
print(jsonObj.get('arrayOfNums')[1])
print(jsonObj.get('arrayOfNums')[1].get('number') +
      jsonObj.get('arrayOfNums')[2].get('number'))
print(jsonObj.get('arrayOfFruits')[2].get('fruit'))

Here is the output:

[{'number': 0}, {'number': 1}, {'number': 2}]
{'number': 1}
3
pear

Line 1 is a list of dictionary objects, line 2 is a dictionary object, line 3 is an integer (the sum of the integers accessed in the dictionaries), and line 4 is a string.

Undocumented APIs

So far in this chapter, we’ve discussed only APIs that are documented. Their developers intend them to be used by the public, publish information about them, and assume that the APIs will be used by other developers. But the vast majority of APIs don’t have any published documentation at all.

But why would you create an API without any public documentation? As mentioned in the beginning of this chapter, it all has to do with JavaScript.

Traditionally, the web servers for dynamic websites had several tasks whenever a user requested a page:

• Handle GET requests from users requesting a page of a website

• Retrieve the data from the database that appears on that page

• Format the data into the HTML template for the page

• Send that formatted HTML to the user

As JavaScript frameworks became more ubiquitous, many of the HTML creation tasks handled by the server moved into the browser. The server might send a hardcoded HTML template to the user’s browser, but separate Ajax requests would be made to load the content and place it in the correct slots in that HTML template. All this would happen on the browser/client side.

This was initially a problem for web scrapers. They were used to making a request for an HTML page and getting back exactly that—an HTML page with all of the content already in place. Instead, they now got an HTML template without any content.

Selenium was used to solve this problem. Now the programmer’s web scraper could become the browser, request the HTML template, execute any JavaScript, allow all the data to load in its place, and only then scrape the page for data. Because the HTML was all loaded, it was essentially reduced to a previously solved problem—the problem of parsing and formatting existing HTML.

However, because the entire content management system (that used to reside only in the web server) had essentially moved to the browser client, even the simplest websites could balloon into several megabytes of content and a dozen HTTP requests.

In addition, when Selenium is used, all of the “extras” that the user doesn’t necessarily care about are loaded. Calls to tracking programs, loading sidebar ads, calls to tracking programs for the sidebar ads. Images, CSS, third-party font data—all of it needs to be loaded. This may seem great when you’re using a browser to browse the web, but if you’re writing a web scraper that needs to move fast, collect specific data, and place as little of a load on the web server as possible, you can be loading a hundred times more data than you need.

But there’s a silver lining to all of this JavaScript, Ajax, and web modernization: because servers are no longer formatting the data into HTML, they often act as thin wrappers around the database itself. This thin wrapper simply extracts data from the database, and returns it to the page via an API.

Of course, these APIs aren’t meant to be used by anyone or anything besides the webpage itself, and so developers leave them undocumented and assume (or hope) that no one will notice them. But they do exist.

The New York Times website, for example, loads all of its search results via JSON. If you visit the link

https://query.nytimes.com/search/sitesearch/#/python

this will reveal recent news articles for the search term “python.” If you scrape this page using urllib or the Requests library, you won’t find any search results. These are loaded separately via an API call:

https://query.nytimes.com/svc/add/v1/sitesearch.json
?q=python&spotlight=true&facet=true

If you were to load this page with Selenium, you would be making about 100 requests and transferring 600–700 kB of data with each search. Using the API directly, you make only one request and transfer approximately only the 60 kb of nicely formatted data that you need.

Finding Undocumented APIs

You’ve used the Chrome inspector in previous chapters to examine the contents of an HTML page, but now you’ll use it for a slightly different purpose: to examine the requests and responses of the calls that are used to construct that page.

To do this, open the Chrome inspector window and click the Network tab, shown in Figure 12-1.

Figure 12-1. The Chrome network inspector tool provides a view into all calls your browser is making and receiving

Note that you need to open this window before the page loads. It does not track network calls while it’s closed.

While the page is loading, you’ll see a line appear in real time whenever your browser makes a call back to the web server for additional information to render the page. This may include an API call.

Finding undocumented APIs can take a little detective work (to take the detective work out of this, see “Finding and Documenting APIs Automatically”), especially with larger sites with lots of network calls. Generally, though, you’ll know it when you see it.

API calls tend to have several features that are useful for locating them in the list of network calls:

• They often have JSON or XML in them. You can filter the list of requests by using the search/filter field.

• With GET requests, the URL will contain the parameter values passed to them. This will be useful if, for example, you’re looking for an API call that returns the results of a search or is loading data for a specific page. Simply filter the results with the search term you used, page ID, or other identifying information.

• They will usually be of the type XHR.

APIs may not always be obvious, especially in large sites with lots of features that may make hundreds of calls while loading a single page. However, spotting the metaphorical needle in the haystack becomes much easier with a little practice.

$ python consoleservice.py -h

usage: consoleservice.py [-h] [-u [U]] [-d [D]] [-s [S]] [-c [C]] [-i [I]]
                         [--p]

optional arguments:

  -h, --help  show this help message and exit

  -u [U]      Target URL. If not provided, target directory will be scanned
              for har files.

  -d [D]      Target directory (default is "hars"). If URL is provided,
              directory will store har files. If URL is not provided,
              directory will be scanned.

  -s [S]      Search term

  -c [C]      File containing JSON formatted cookies to set in driver (with
              target URL only)

  -i [I]      Count of pages to crawl (with target URL only)

  --p         Flag, remove unnecessary parameters (may dramatically increase
              runtime)

$ python consoleservice.py -u https://www.target.com/p/rogue-one-a-star-wars-\
story-blu-ray-dvd-digital-3-disc/-/A-52030319 -s "Rogue One: A Star Wars Story"

URL: https://redsky.target.com/v2/pdp/tcin/52030319
METHOD: GET
AVG RESPONSE SIZE: 34834
SEARCH TERM CONTEXT: c":"786936852318","product_description":{"title":
"Rogue One: A Star Wars Story (Blu-ray + DVD + Digital) 3 Disc",
"long_description":...

Combining APIs with Other Data Sources

Although the raison d'être of many modern web applications is to take existing data and format it in a more appealing way, I would argue that this isn’t an interesting thing to do in most instances. If you’re using an API as your only data source, the best you can do is merely copy someone else’s database that already exists, and which is, essentially, already published. What can be far more interesting is to take two or more data sources and combine them in a novel way, or use an API as a tool to look at scraped data from a new perspective.

Let’s look at one example of how data from APIs can be used in conjunction with web scraping to see which parts of the world contribute the most to Wikipedia.

If you’ve spent much time on Wikipedia, you’ve likely come across an article’s revision history page, which displays a list of recent edits. If users are logged into Wikipedia when they make the edit, their username is displayed. If they are not logged in, their IP address is recorded, as shown in Figure 12-2.

Figure 12-2. The IP address of an anonymous editor on the revision history page for Wikipedia’s Python entry

The IP address provided on the history page is 121.97.110.145. By using the ip-api.com API, that IP address is from Quezon, Philippines as of this writing (IP addresses can occasionally shift geographically).

This information isn’t all that interesting on its own, but what if you could gather many points of geographic data about Wikipedia edits and where they occur? A few years ago, I did just that and used Google’s GeoChart library to create an interesting chart that shows where edits on the English-language Wikipedia, along with the Wikipedias written in other languages, originate from (Figure 12-3).

Figure 12-3. Visualization of Wikipedia edits created using Google’s GeoChart library

Creating a basic script that crawls Wikipedia, looks for revision history pages, and then looks for IP addresses on those revision history pages isn’t difficult. Using modified code from Chapter 3, the following script does just that:

from urllib.request import urlopen
from bs4 import BeautifulSoup
import json
import datetime
import random
import re

random.seed(datetime.datetime.now())
def getLinks(articleUrl):
    html = urlopen('http://en.wikipedia.org{}'.format(articleUrl))
    bs = BeautifulSoup(html, 'html.parser')
    return bs.find('div', {'id':'bodyContent'}).find_all('a', 
        href=re.compile('^(/wiki/)((?!:).)*$'))

def getHistoryIPs(pageUrl):
    #Format of revision history pages is: 
    #http://en.wikipedia.org/w/index.php?title=Title_in_URL&action=history
    pageUrl = pageUrl.replace('/wiki/', '')
    historyUrl = 'http://en.wikipedia.org/w/index.php?title={}&action=history'
        .format(pageUrl)
    print('history url is: {}'.format(historyUrl))
    html = urlopen(historyUrl)
    bs = BeautifulSoup(html, 'html.parser')
    #finds only the links with class "mw-anonuserlink" which has IP addresses 
    #instead of usernames
    ipAddresses = bs.find_all('a', {'class':'mw-anonuserlink'})
    addressList = set()
    for ipAddress in ipAddresses:
        addressList.add(ipAddress.get_text())
    return addressList

links = getLinks('/wiki/Python_(programming_language)')

while(len(links) > 0):
    for link in links:
        print('-'*20) 
        historyIPs = getHistoryIPs(link.attrs['href'])
        for historyIP in historyIPs:
            print(historyIP)

    newLink = links[random.randint(0, len(links)-1)].attrs['href']
    links = getLinks(newLink)

This program uses two main functions: getLinks (which was also used in Chapter 3), and the  new getHistoryIPs, which searches for the contents of all links with the class mw-anonuserlink (indicating an anonymous user with an IP address, rather than a username) and returns it as a set.

This code also uses a somewhat arbitrary (yet effective for the purposes of this example) search pattern to look for articles from which to retrieve revision histories. It starts by retrieving the histories of all Wikipedia articles linked to by the starting page (in this case, the article on the Python programming language). Afterward, it randomly selects a new starting page, and retrieves all revision history pages of articles linked to by that page. It will continue until it hits a page with no links.

Now that you have code that retrieves IP addresses as a string, you can combine this with the getCountry function from the previous section in order to resolve these IP addresses to countries. You’ll modify getCountry slightly, in order to account for invalid or malformed IP addresses that will result in a 404 Not Found error:

def getCountry(ipAddress):
    try:
        response = urlopen(
            'http://ip-api.com/json/{}'.format(ipAddress)).read().decode('utf-8')
    except HTTPError:
        return None
    responseJson = json.loads(response)
    return responseJson.get('country_code')
    
links = getLinks('/wiki/Python_(programming_language)')

while(len(links) > 0):
    for link in links:
        print('-'*20) 
        historyIPs = getHistoryIPs(link.attrs["href"])
        for historyIP in historyIPs:
            country = getCountry(historyIP)
            if country is not None:
                print('{} is from {}'.format(historyIP, country))

    newLink = links[random.randint(0, len(links)-1)].attrs['href']
    links = getLinks(newLink)

Here’s the sample output:

-------------------
history url is: http://en.wikipedia.org/w/index.php?title=Programming_
paradigm&action=history
68.183.108.13 is from US
86.155.0.186 is from GB
188.55.200.254 is from SA
108.221.18.208 is from US
141.117.232.168 is from CA
76.105.209.39 is from US
182.184.123.106 is from PK
212.219.47.52 is from GB
72.27.184.57 is from JM
49.147.183.43 is from PH
209.197.41.132 is from US
174.66.150.151 is from US

More About APIs

This chapter has shown a few ways that modern APIs are commonly used to access data on the web, and how those APIs can be used to build faster and more powerful web scrapers. If you’re looking to build APIs instead of just using them, or if you want to learn more about the theory of their construction and syntax, I recommend RESTful Web APIs by Leonard Richardson, Mike Amundsen, and Sam Ruby (O’Reilly). This book provides a strong overview of the theory and practice of using APIs on the web. In addition, Mike Amundsen has a fascinating video series, Designing APIs for the Web (O’Reilly), that teaches you how to create your own APIs—a useful thing to know if you decide to make your scraped data available to the public in a convenient format.

While some might bemoan the ubiquity of JavaScript and dynamic websites, making traditional “grab and parse the HTML page” practices outdated, I, for one, welcome our new robot overlords. As dynamic websites rely less on HTML pages for human consumption and more on strictly formatted JSON files for HTML consumption, this provides a boon for everyone trying to get clean, well-formatted data.

The web is no longer a collection of HTML pages with occasional multimedia and CSS adornments. It’s a collection of hundreds of file types and data formats, whizzing hundreds at a time to form the pages that you consume through your browser. The real trick is often to look beyond the page in front of you and grab the data at its source.