API Features

An API can be as simple as a data response to a request, but it’s rare to find APIs with only that functionality. Most APIs have other useful features. These features may include multiple API request methods (REST or streaming), data timestamps, rate limits, data tiers, and API access objects (keys and tokens). Let’s take a look at these in the context of the Twitter API.

A Simple Data Pull from Twitter’s REST API

With a set of keys, we can now start to access data from Twitter’s API. In this section, we will put together a simple script to pull data from the API by passing a search query. The script in this section is based on a snippet of Python code provided by Twitter as an example. This code uses Python OAuth2, which is a protocol for identifying and connecting securely when using APIs.

To start, we need to install Python OAuth2:

pip install oauth2

Open a new file and start by importing oauth2 and assigning your key variables:

import oauth2

API_KEY = '5Hqg6JTZ0cC89hUThySd5yZcL'
API_SECRET = 'Ncp1oi5tUPbZF19Vdp8Jp8pNHBBfPdXGFtXqoKd6Cqn87xRj0c'
TOKEN_KEY = '3272304896-ZTGUZZ6QsYKtZqXAVMLaJzR8qjrPW22iiu9ko4w'
TOKEN_SECRET = 'nsNY13aPGWdm2QcgOl0qwqs5bwLBZ1iUVS2OE34QsuR4C'

Then add the function to create the OAuth connection:

def oauth_req(url, key, secret, http_method="GET", post_body="",
              http_headers=None):
    consumer = oauth2.Consumer(key=API_KEY, secret=API_SECRET)              
    token = oauth2.Token(key=key, secret=secret)                            
    client = oauth2.Client(consumer, token)                                 
    resp, content = client.request(url, method=http_method,                 
                                   body=post_body, headers=http_headers)
    return content                                                          

Establishes the consumer of the oauth2 object. The consumer is the owner of the keys. This line provides the consumer with the keys so it can properly identify via the API.

Assigns the token to the oauth2 object.

Creates the client, which consists of the consumer and token.

Using the url, which is a function argument, executes the request using the OAuth2 client.

Returns the content received from the connection.

Now we have a function that allows us to connect to the Twitter API. However, we need to define our URL and call the function. The Search API documentation tells us more about what requests we want to use. Using the web interface, we can see that if we search for #childlabor, we end up with the following URL: https://twitter.com/search?q=%23childlabor. The documentation instructs us to reformat the URL so we end up with the following: https://api.twitter.com/1.1/search/tweets.json?q=%23childlabor.

Then, we can add that URL as a variable and call the function using our previously defined variables:

url = 'https://api.twitter.com/1.1/search/tweets.json?q=%23childlabor'
data = oauth_req(url, TOKEN_KEY, TOKEN_SECRET)

print(data)                                                          

Add a print statement at the end, so you can see the output.

When you run the script, you should see the data printed as a long JSON object. You may remember a JSON object looks like a Python dictionary, but if you were to rerun the script with print(type(data)), you would find out that the content is a string. At this point we could do one of two things: we could convert the data into a dictionary and start parsing it, or we could save the string to a file to parse later. To continue parsing the data in the script, add import json at the top of the script. Then, at the bottom, load the string using json and output it:

data = json.loads(data)
print(type(data))

The data variable will now return a Python dictionary. If you want to write the data to a file and parse it later, add the following code instead:

with open('tweet_data.json', 'wb') as data_file:
    data_file.write(data)

Your final script should look like the following:

import oauth2

API_KEY = '5Hqg6JTZ0cC89hUThySd5yZcL'
API_SECRET = 'Ncp1oi5tUPbZF19Vdp8Jp8pNHBBfPdXGFtXqoKd6Cqn87xRj0c'
TOKEN_KEY = '3272304896-ZTGUZZ6QsYKtZqXAVMLaJzR8qjrPW22iiu9ko4w'
TOKEN_SECRET = 'nsNY13aPGWdm2QcgOl0qwqs5bwLBZ1iUVS2OE34QsuR4C'


def oauth_req(url, key, secret, http_method="GET", post_body="",
              http_headers=None):
    consumer = oauth2.Consumer(key=API_KEY, secret=API_SECRET)
    token = oauth2.Token(key=key, secret=secret)
    client = oauth2.Client(consumer, token)
    resp, content = client.request(url, method=http_method,
                                   body=post_body, headers=http_headers)
    return content


url = 'https://api.twitter.com/1.1/search/tweets.json?q=%23popeindc'
data = oauth_req(url, TOKEN_KEY, TOKEN_SECRET)

with open("data/hashchildlabor.json", "w") as data_file:
    data_file.write(data)

From here you can refer back to the section “JSON Data” in Chapter 3 to parse the data.

Advanced Data Collection from Twitter’s REST API

Pulling a single data file from Twitter is not terribly useful, because it only returns about 15 tweets. We are looking to execute multiple queries in a row, so we can collect as many tweets as possible related to our topic. We are going to use another library to do some of the heavy lifting for us—Tweepy. Tweepy can help us manage a series of requests as well as OAuth using Twitter. Start by installing tweepy:

pip install tweepy

At the top of your script, import tweepy and set your keys again:

import tweepy

API_KEY = '5Hqg6JTZ0cC89hUThySd5yZcL'
API_SECRET = 'Ncp1oi5tUPbZF19Vdp8Jp8pNHBBfPdXGFtXqoKd6Cqn87xRj0c'
TOKEN_KEY = '3272304896-ZTGUZZ6QsYKtZqXAVMLaJzR8qjrPW22iiu9ko4w'
TOKEN_SECRET = 'nsNY13aPGWdm2QcgOl0qwqs5bwLBZ1iUVS2OE34QsuR4C'

Then pass your API key and API secret to tweepy’s OAuthHandler object, which will manage the same OAuth protocol covered in the last example. Then set your access token:

auth = tweepy.OAuthHandler(API_KEY, API_SECRET)                         
auth.set_access_token(TOKEN_KEY, TOKEN_SECRET)                          

Creates an object to manage the API authentication via tweepy

Sets token access

Next, pass the authorization object you just created to tweepy.API:

api = tweepy.API(auth)

The tweepy.API object can take a variety of arguments to give you customized control over how tweepy behaves when requesting data. You can directly add retries and delays between requests using parameters like retry_count=3, retry_delay=5. Another useful option is wait_on_rate_limit, which will wait until the rate limit has been lifted to make the next request. Details on all of these niceties and more are included in the tweepy documentation.

We want to create a connection to the Twitter API using tweepy.Cursor. We can then pass the cursor the API method to use, which is api.search, and the parameters associated with that method:

query = '#childlabor'                                                   
cursor = tweepy.Cursor(api.search, q=query, lang="en")                  

Creates the query variable

Establishes the cursor with the query, and limits it to just the English language

According to tweepy’s documentation, cursor can return an iterator on a per-item or per-page level. You can also define limits to determine how many pages or items the cursor grabs. If you look at print(dir(cursor)), you’ll see there are three methods: ['items', 'iterator', 'pages']. A page returns a bunch of items, which are individual tweets from your query. For our needs, we are going to use pages.

Let’s iterate through the pages and save the data. Before we do that, we need to do two things:

1. Add import json to the top of the script.

2. Create a directory called data in the same directory as the script. To do this, run mkdir data on the command line.

Once you’ve done those two things, run the following code to iterate through and save the tweets:

for page in cursor.pages():                                             
    tweets = []                                                         
    for item in page:                                                   
        tweets.append(item._json)                                       

with open('data/hashchildlabor.json', 'wb') as outfile:             
    json.dump(tweets, outfile)

For each page returned in cursor.pages()…

Creates an empty list to store tweets.

For each item (or tweet) in a page…

Extracts the JSON tweet data and saves it to the tweets list.

Opens a file called hashchildlabor.json and saves the tweets.

You will notice not many tweets are being saved to the file. There are only 15 tweets per page, so we’ll need to figure out a way to get more data. Options include:

• Open a file and never close it, or open a file and append the information at the end. This will create one massive file.

• Save each page in its own file (you can use timestamps to ensure you have different filenames for each file).

• Create a new table in your database to save the tweets.

Creating one file is dangerous, because at any moment the process could fail and corrupt the data. Unless you have a small data pull (e.g., 1000 tweets) or are doing development testing, you should use one of the other options.

There are a couple of ways to save the data in a new file every time, the most common ones being creating a filename by using a date and timestamp, or just by incrementing a number and appending it to the end of the filename.

We’ll go ahead and add our tweets to our simple database. To do so, we’ll use this function:

def store_tweet(item):
    db = dataset.connect('sqlite:///data_wrangling.db')
    table = db['tweets']                                    
    item_json = item._json.copy()
    for k, v in item_json.items():
        if isinstance(v, dict):                             
            item_json[k] = str(v)
    table.insert(item_json)                                 

Creates or accesses a new table called tweets

Tests if there are any dictionaries in our tweet item values. Since SQLite doesn’t support saving Python dictionaries, we need to convert dictionaries into strings.

Inserts the cleaned JSON item.

We will also need to add dataset into our import. We will then need to add the use of this function where we were previously storing the pages. We’ll also want to make sure we iterate over every tweet. Your final script should look like the following:

import json
import tweepy
import dataset

API_KEY = '5Hqg6JTZ0cC89hUThySd5yZcL'
API_SECRET = 'Ncp1oi5tUPbZF19Vdp8Jp8pNHBBfPdXGFtXqoKd6Cqn87xRj0c'
TOKEN_KEY = '3272304896-ZTGUZZ6QsYKtZqXAVMLaJzR8qjrPW22iiu9ko4w'
TOKEN_SECRET = 'nsNY13aPGWdm2QcgOl0qwqs5bwLBZ1iUVS2OE34QsuR4C'

def store_tweet(item):
    db = dataset.connect('sqlite:///data_wrangling.db')
    table = db['tweets']
    item_json = item._json.copy()
    for k, v in item_json.items():
        if isinstance(v, dict):
            item_json[k] = str(v)
    table.insert(item_json)

auth = tweepy.OAuthHandler(API_KEY, API_SECRET)
auth.set_access_token(TOKEN_KEY, TOKEN_SECRET)

api = tweepy.API(auth)

query = '#childlabor'
cursor = tweepy.Cursor(api.search, q=query, lang="en")

for page in cursor.pages():
    for item in page:
        store_tweet(item)

Advanced Data Collection from Twitter’s Streaming API

Early in this chapter, we mentioned there are two types of Twitter APIs available: REST and Streaming.

How does the Streaming API differ from the REST API? Here’s a brief rundown:

• The data is live, while the REST API returns only data that has already been tweeted.

• Streaming APIs are less common, but will become more available in the future as more live data is generated and exposed.

• Because up-to-date data is interesting, many people are interested in the data, which means you can find lots of resources and help online.

Let’s create a script to collect from the Streaming API. Such a script builds on all the concepts we’ve covered in this chapter. We’ll first add the basics—imports and keys:

from tweepy.streaming import StreamListener                             
from tweepy import OAuthHandler, Stream                                 

API_KEY = '5Hqg6JTZ0cC89hUThySd5yZcL'
API_SECRET = 'Ncp1oi5tUPbZF19Vdp8Jp8pNHBBfPdXGFtXqoKd6Cqn87xRj0c'
TOKEN_KEY = '3272304896-ZTGUZZ6QsYKtZqXAVMLaJzR8qjrPW22iiu9ko4w'
TOKEN_SECRET = 'nsNY13aPGWdm2QcgOl0qwqs5bwLBZ1iUVS2OE34QsuR4C'

Imports StreamListener, which creates a streaming session and listens for messages

Imports OAuthHandler, which we used before, and Stream, which actually handles the Twitter stream

In this script, we are doing our import statements slightly differently than we did in the last script. Both of these are valid approaches and a matter of preference. Here’s a quick comparison of the two approaches:

Approach 1

import tweepy
...
auth = tweepy.OAuthHandler(API_KEY, API_SECRET)

Approach 2

from tweepy import OAuthHandler
...
auth = OAuthHandler(API_KEY, API_SECRET)

Usually the first approach is used when the library is not used much in the script. It is also good when you have a longer piece of code and want to be explicit. However, when the library is used a lot it gets tiresome to type this out: also, if the library is the cornerstone of the script, it should be fairly obvious to people what modules or classes are imported from the library.

Now we are going to subclass (a concept you learned about in Chapter 12) the StreamListener class we imported because we want to override the on_data method. To do this, we redefine it in our new class, which we call Listener. When there is data, we want to see it in our terminal, so we are going to add a print statement:

class Listener(StreamListener):                                         

    def on_data(self, data):                                            
        print data                                                      
        return True                                                     

Subclasses StreamListener.

Defines the on_data method.

Outputs tweets.

Returns True. StreamListener has an on_data method, which also returns True. As we’re subclassing and redefining it, we must repeat the return value in the subclassed method.

Next, add your authentication handlers:

auth = OAuthHandler(API_KEY, API_SECRET)
auth.set_access_token(TOKEN_KEY, TOKEN_SECRET)

Finally, pass the Listener and auth to the Stream and start filtering with a search term. In this case, we are going to look at child labor because it has more traffic than #childlabor:

stream = Stream(auth, Listener())                                       
stream.filter(track=['child labor'])                                    

Sets up the stream by passing auth and Listener as arguments

Filters the stream and returns only items with the terms child and labor

Your final script should look like this:

from tweepy.streaming import StreamListener
from tweepy import OAuthHandler, Stream


API_KEY = '5Hqg6JTZ0cC89hUThySd5yZcL'
API_SECRET = 'Ncp1oi5tUPbZF19Vdp8Jp8pNHBBfPdXGFtXqoKd6Cqn87xRj0c'
TOKEN_KEY = '3272304896-ZTGUZZ6QsYKtZqXAVMLaJzR8qjrPW22iiu9ko4w'
TOKEN_SECRET = 'nsNY13aPGWdm2QcgOl0qwqs5bwLBZ1iUVS2OE34QsuR4C'


class Listener(StreamListener):

    def on_data(self, data):
        print data
        return True

auth = OAuthHandler(API_KEY, API_SECRET)
auth.set_access_token(TOKEN_KEY, TOKEN_SECRET)

stream = Stream(auth, Listener())
stream.filter(track=['child labor'])

From here, you would add a way to save tweets to your database, file, or other storage using your on_data method as we did earlier in the chapter.

Summary

Being able to interact with application programming interfaces is an important part of data wrangling. In this chapter, we covered some of the API basics (see Table 13-2 for a summary) and processed data from the Twitter API.

We reused many Python concepts we already knew and learned a few new Python concepts in this chapter. The first was the usage of tweepy, a library to handle interactions with the Twitter API. You also learned about authentication and OAuth protocols.

As an extension of interacting with an API, Chapter 14 will help you learn about techniques enabling you to run your API scripts while you are away.