JSON parsing

Javascript Object Notation (JSON) is exactly what it says—the notation used to define objects in JavaScript. The Requests library has built a JSON parser into its Response object.

The json library can parse JSON from strings or files into a Python dictionary (or list, as appropriate). It can also convert Python dictionaries or lists into JSON strings. For example, the following string contains JSON data:

json_string = '{"first_name": "Guido", "last_name":"van Rossum"}'

It can be parsed like this:

import json
parsed_json = json.loads(json_string)

and can now be used as a normal dictionary:

print(parsed_json['first_name'])
"Guido"

You can also convert the following to JSON:

d = {
    'first_name': 'Guido',
    'last_name': 'van Rossum',
    'titles': ['BDFL', 'Developer'],
}

print(json.dumps(d))
'{"first_name": "Guido", "last_name": "van Rossum",
  "titles": ["BDFL", "Developer"]}'

XML parsing

There is an XML parser in the Standard Library (xml.etree.ElementTree’s parse() and fromstring() methods), but this uses the Expat library and creates an ElementTree object that preserves the structure of the XML, meaning we have to iterate down it and look into its children to get content. When all you want is to get the data, try either untangle or xmltodict. You can get both using pip:

$ pip install untangle
$ pip install xmltodict

untangle

untangle takes an XML document and returns a Python object whose structure mirrors the nodes and attributes. For example, an XML file like this:

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <child name="child1" />
</root>

can be loaded like this:

import untangle
obj = untangle.parse('path/to/file.xml')

and then you can get the child element’s name like this:

obj.root.child['name']  # is 'child1'

xmltodict

xmltodict converts the XML to a dictionary. For example, an XML file like this:

<mydocument has="an attribute">
  <and>
    <many>elements</many>
    <many>more elements</many>
  </and>
  <plus a="complex">
    element as well
  </plus>
</mydocument>

can be loaded into an OrderedDict instance (from the collections module in Python’s Standard Library) like this:

import xmltodict

with open('path/to/file.xml') as fd:
    doc = xmltodict.parse(fd.read())

and then you can access elements, attributes, and values like this:

doc['mydocument']['@has']  # is u'an attribute'
doc['mydocument']['and']['many']  # is [u'elements', u'more elements']
doc['mydocument']['plus']['@a']  # is u'complex'
doc['mydocument']['plus']['#text']  # is u'element as well'

With xmltodict, you can also roundtrip the dictionary back to XML with the unparse() function. It has a streaming mode suitable for handling files that don’t fit in memory, and it supports namespaces.

Web scraping

Websites don’t always provide their data in comfortable formats such as CSV or JSON, but HTML is also structured data—this is where web scraping comes in.

Web scraping is the practice of using a computer program to sift through a web page and gather the data that you need in a format most useful to you while at the same time preserving the structure of the data.

lxml

lxml is a pretty extensive library written for parsing XML and HTML documents very quickly, even handling some amount of incorrectly formatted markup in the process. Get it using pip:

$ pip install lxml

Use requests.get to retrieve the web page with our data, parse it using the html module, and save the results in tree:

from lxml import html
import requests

page = requests.get('http://econpy.pythonanywhere.com/ex/001.html')  
tree = html.fromstring(page.content)  

This is a real web page, and the data we show are real—you can visit the page in your browser.

We use page.content rather than page.text because html.fromstring() implicitly expects bytes as input.

Now, tree contains the whole HTML file in a nice tree structure that we can go over in two different ways: XPath or CSSSelect. They are both standard ways to specify a path through an HTML tree, defined and maintained by the World Wide Web Consortium (W3C), and implemented as modules in lxml. In this example, we will use XPath. A good introduction is W3Schools XPath tutorial.

There are also various tools for obtaining the XPath of elements from inside your web browser, such as Firebug for Firefox or the Chrome Inspector. If you’re using Chrome, you can right-click an element, choose “Inspect element”, highlight the code, right-click again and choose “Copy XPath”.

After a quick analysis, we see that in our page the data is contained in two elements—one is a div with title buyer-name, and the other is a span with the class item-price:

<div title="buyer-name">Carson Busses</div>
<span class="item-price">$29.95</span>

Knowing this, we can create the correct XPath query and use lxml’s xpath function like this:

# This will create a list of buyers:
buyers = tree.xpath('//div[@title="buyer-name"]/text()')
# This will create a list of prices
prices = tree.xpath('//span[@class="item-price"]/text()')

Let’s see what we got exactly:

>>> print('Buyers: ', buyers)
Buyers:  ['Carson Busses', 'Earl E. Byrd', 'Patty Cakes',
'Derri Anne Connecticut', 'Moe Dess', 'Leda Doggslife', 'Dan Druff',
'Al Fresco', 'Ido Hoe', 'Howie Kisses', 'Len Lease', 'Phil Meup',
'Ira Pent', 'Ben D. Rules', 'Ave Sectomy', 'Gary Shattire',
'Bobbi Soks', 'Sheila Takya', 'Rose Tattoo', 'Moe Tell']
>>>
>>> print('Prices: ', prices)
Prices:  ['$29.95', '$8.37', '$15.26', '$19.25', '$19.25',
'$13.99', '$31.57', '$8.49', '$14.47', '$15.86', '$11.11',
'$15.98', '$16.27', '$7.50', '$50.85', '$14.26', '$5.68',
'$15.00', '$114.07', '$10.09']

Performance networking tools in Python’s Standard Library

asyncio was introduced in Python 3.4 and includes ideas learned from the developer communities, like those maintaining Twisted and gevent. It’s a concurrency tool, and a frequent application of concurrency is for network servers. Python’s own documentation for asyncore (a predecessor to asyncio), states:

There are only two ways to have a program on a single processor do “more than one thing at a time.” Multi-threaded programming is the simplest and most popular way to do it, but there is another very different technique, that lets you have nearly all the advantages of multi-threading, without actually using multiple threads. It’s really only practical if your program is largely I/O bound. If your program is processor bound, then pre-emptive scheduled threads are probably what you really need. Network servers are rarely processor bound, however.

asyncio is still only in the Python Standard Library on a provisional basis—the API may change in backward-incompatible ways—so don’t get too attached.

Not all of it is new—asyncore (deprecated in Python 3.4) has an event loop, asynchronous sockets² and asynchronous file I/O, and asynchat (also deprecated in Python 3.4) had asynchronous queues.³ The big thing asyncio adds is a formalized implementation of coroutines. In Python, this is formally defined as both a coroutine function—a function definition beginning with async def rather than just def (or uses the older syntax, and is decorated with @asyncio.coroutine)—and also the object obtained by calling a coroutine function (which is usually some sort of computation or I/O operation). The coroutine can yield the processor and thus be able to participate in an asynchronous event loop, taking turns with other coroutines.

The documentation has pages and pages of detailed examples to help the community, as it’s a new concept for the language. It’s clear, thorough, and very much worth checking out. In this interactive session, we just want to show the functions for the event loop and some of the classes available:

>>> import asyncio
>>>
>>> [l for l in asyncio.__all__ if 'loop' in l]
['get_event_loop_policy', 'set_event_loop_policy',
'get_event_loop', 'set_event_loop', 'new_event_loop']
>>>
>>> [t for t in asyncio.__all__ if t.endswith('Transport')]
['BaseTransport', 'ReadTransport', 'WriteTransport', 'Transport',
'DatagramTransport', 'SubprocessTransport']
>>>
>>> [p for p in asyncio.__all__ if p.endswith('Protocol')]
['BaseProtocol', 'Protocol', 'DatagramProtocol',
'SubprocessProtocol', 'StreamReaderProtocol']
>>>
>>> [q for q in asyncio.__all__ if 'Queue' in q]
['Queue', 'PriorityQueue', 'LifoQueue', 'JoinableQueue',
'QueueFull', 'QueueEmpty']

gevent

gevent is a coroutine-based Python networking library that uses greenlets to provide a high-level synchronous API on top of the C library libev event loop. Greenlets are based on the greenlet library—miniature green threads (or user-level threads, as opposed to threads controlled by the kernel) that the developer has the freedom to explicitly suspend, jumping between greenlets. For a great deep dive into gevent, check out Kavya Joshi’s seminar “A Tale of Concurrency Through Creativity in Python.”

People use gevent because it is lightweight and tightly coupled to its underlying C library, libev, for high performance. If you like the idea of integrating asynchronous I/O and greenlets, this is the library to use. Get it using pip:

$ pip install gevent

Here’s an example from the greenlet documentation:

>>> import gevent
>>>
>>> from gevent import socket
>>> urls = ['www.google.com', 'www.example.com', 'www.python.org']
>>> jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]
>>> gevent.joinall(jobs, timeout=2)
>>> [job.value for job in jobs]
['74.125.79.106', '208.77.188.166', '82.94.164.162']

The documentation offers many more examples.

Twisted

Twisted is an event-driven networking engine. It can be used to build applications around many different networking protocols, including HTTP servers and clients, applications using SMTP, POP3, IMAP or SSH protocols, instant messaging and much more. Install it using pip:

$ pip install twisted

Twisted has been around since 2002 and has a loyal community. It’s like the Emacs of coroutine libraries—with everything built in—because all of these things have to be asynchronous to work together. Probably the most useful tools are an asynchronous wrapper for database connections (in twisted.enterprise.adbapi), a DNS server (in twisted.names), direct access to packets (in twisted.pair), and additional protocols like AMP, GPS, and SOCKSv4 (in twisted.protocols). Most of Twisted now works with Python 3—when you pip install in a Python 3 environment, you’ll get get everything that’s currently been ported. If you find something you wanted in the API that’s not in your Twisted, you should still use Python 2.7.

For more information, consult Jessica McKellar and Abe Fettig’s Twisted (O’Reilly). In addition, this webpage shows over 42 Twisted examples, and this one shows their latest speed performance.

PyZMQ

PyZMQ is the Python binding for ZeroMQ. You can get it using pip:

$ pip install pyzmq

ØMQ (also spelled ZeroMQ, 0MQ, or ZMQ) describes itself as a messaging library designed to have a familiar socket-style API, and aimed at use in scalable distributed or concurrent applications. Basically, it implements asynchronous sockets with queues attached and provides a custom list of socket “types” that determine how the I/O on each socket behaves. Here’s an example:

import zmq
context = zmq.Context()
server = context.socket(zmq.REP)  
server.bind('tcp://127.0.0.1:5000')  

while True:
    message = server.recv().decode('utf-8')
    print('Client said: {}'.format(message))
    server.send(bytes('I don't know.', 'utf-8'))

# ~~~~~ and in another file ~~~~~

import zmq
context = zmq.Context()
client = context.socket(zmq.REQ)  
client.connect('tcp://127.0.0.1:5000')  

client.send(bytes("What's for lunch?", 'utf-8'))
response = client.recv().decode('utf-8')
print('Server replied: {}'.format(response))

The socket type zmq.REP corresponds to their “request-response” paradigm.

Like with normal sockets, you bind the server to an IP and port.

The client type is zmq.REQ—that’s all, ZMQ defines a number of these as constants: zmq.REQ, zmq.REP, zmq.PUB, zmq.SUB, zmq.PUSH, zmq.PULL, zmq.PAIR. They determine how the socket’s sending and receiving behaves.

As usual, the client connects to the server’s bound IP and port.

So, these look and quack like sockets, enhanced with queues and various I/O patterns. The point of the patterns is to provide the building blocks for a distributed network. The basic patterns for the socket types are:

request-reply

zmq.REQ and zmq.REP connect a set of clients to a set of services. This can be for a remote procedure call pattern or a task distribution pattern.

publish-subscribe

zmq.PUB and zmq.SUB connect a set of publishers to a set of subscribers. This is a data distribution pattern—one node is distributing data to other nodes, or this can be chained to fan out into a distribution tree.

push-pull (or pipeline)

zmq.PUSH and zmq.PULL connect nodes in a fan-out/fan-in pattern that can have multiple steps, and loops. This is a parallel task distribution and collection pattern.

One great advantage of ZeroMQ over message-oriented middleware is that it can be used for message queuing without a dedicated message broker. PyZMQ’s documentation notes some enhancements they added, like tunneling via SSH. The rest of the documentation for the ZeroMQ API is better on the main ZeroMQ guide.

RabbitMQ

RabbitMQ is an open source message broker software that implements the Advanced Message Queuing Protocol (AMQP). A message broker is an intermediary program that receives messages from senders and sends them to receivers according to a protocol. Any client that also implements AMQP can communicate with RabbitMQ. To get RabbitMQ, go to the RabbitMQ download page, and follow the instructions for your operating system.

Client libraries that interface with the broker are available for all major programming languages. The top two for Python are pika and Celery—either can be installed with pip:

$ pip install pika
$ pip install celery

pika

pika is a lightweight, pure-Python AMQP 0-9-1 client, preferred by RabbitMQ. RabbitMQ’s introductory tutorials for Python use pika. There’s also an entire page of examples to learn from. We recommend playing with pika when you first set up RabbitMQ, regardless of your final library choice, because it is straightforward without the extra features and so crystallizes the concepts.

Celery

Celery is a much more featureful AMQP client—it can use either RabbitMQ or Redis (a distributed in-memory data store) as a message broker, can track the tasks and results (and optionally store them in a user-selected backend), and has a web administration tool/task monitor, Flower. It is popular in the web development community, and there are integration packages for Django, Pyramid, Pylons, web2py, and Tornado (Flask doesn’t need one). Start with the Celery tutorial.