Read HowDoI’s documentation

HowDoI’s documentation is in the README.rst file in the HowDoI repository on GitHub: it’s a small command-line application that allows users to search the Internet for answers to programming questions.

From the command line in a terminal shell, we can type howdoi --help for the usage statement:

(venv)$ howdoi --help
usage: howdoi [-h] [-p POS] [-a] [-l] [-c] [-n NUM_ANSWERS] [-C] [-v]
              [QUERY [QUERY ...]]

instant coding answers via the command line

positional arguments:
  QUERY                 the question to answer

optional arguments:
  -h, --help            show this help message and exit
  -p POS, --pos POS     select answer in specified position (default: 1)
  -a, --all             display the full text of the answer
  -l, --link            display only the answer link
  -c, --color           enable colorized output
  -n NUM_ANSWERS, --num-answers NUM_ANSWERS
                        number of answers to return
  -C, --clear-cache     clear the cache
  -v, --version         displays the current version of howdoi

That’s it—from the documentation we know that HowDoI gets answers to coding questions from the Internet, and from the usage statement we know we can choose the answer in a specific position, can colorize the output, get multiple answers, and that it keeps a cache that can be cleared.

Use HowDoI

We can confirm we understand what HowDoI does by actually using it. Here’s an example:

(venv)$ howdoi --num-answers 2 python lambda function list comprehension
--- Answer 1 ---
[(lambda x: x*x)(x) for x in range(10)]

--- Answer 2 ---
[x() for x in [lambda m=m: m for m in [1,2,3]]]
# [1, 2, 3]

We’ve installed HowDoI, read its documentation, and can use it. On to reading actual code!

Read HowDoI’s code

If you look inside the howdoi/ directory, you’ll see it contains two files: an __init__.py, which contains a single line that defines the version number, and howdoi.py, which we’ll open and read.

Skimming howdoi.py, we see each new function definition is used in the next function, making it is easy to follow. And each function does just one thing—the thing its name says. The main function, command_line_runner(), is near the bottom of howdoi.py.

Rather than reprint HowDoI’s source here, we can illustrate its call structure using the call graph in Figure 5-1. It was created by Python Call Graph, which provides a visualization of the functions called when running a Python script. This works well with command-line applications thanks to a single start point and the relatively few paths through their code. (Note that we manually deleted functions not in the HowDoI project from the rendered image to legibly fit it on the page, and slightly recolored and reformatted it.)

Figure 5-1. Clean paths and clear function names in this howdoi call graph

The code could have been all one large, incomprehensible spaghetti function. Instead, intentional choices structure the code into compartmentalized functions with straightforward names. Here’s a brief description of the execution depicted in Figure 5-1: command_line_runner() parses the input and passes the user flags and the query to howdoi(). Then, howdoi() wraps _get_instructions() in a try/except statement so that it can catch connection errors and print a reasonable error message (because application code should not terminate on exceptions).

The primary functionality is in _get_instructions(): it calls _get_links() to do a Google search of Stack Overflow for links that match the query, then calls _get_answer() once for each resulting link (up to the number of links that the user specified on the command line—the default is just one link).

The _get_answer() function follows a link to Stack Overflow, extracts code from the answer, colorizes it, and returns it to _get_instructions(), which will combine all of the answers into one string, and return it. Both _get_links() and _get_answer() call _get_result() to actually do the HTTP request: _get_links() for the Google query, and _get_answer() for the resulting links from the Google query.

All _get_result() does is wrap requests.get() with a try/except statement so that it can catch SSL errors, print an error message, and re-raise the exception so that the top-level try/except can catch it and exit. Catching all exceptions before exiting is best practice for application programs.

Let each function do just one thing

We can’t reiterate enough how beneficial it is for readers to separate out HowDoI’s internal functions to each do just one thing. Also, there are functions whose sole purpose is to wrap other functions with a try/except statement. (The only function with a try/except that doesn’t follow this practice is _format_output(), which leverages try/except clauses to identify the correct coding language for syntax highlighting, not for exception handling.)

Leverage data available from the system

HowDoI checks and uses relevant system values, such as urllib.request.getproxies(), to handle the use of proxy servers (this can be the case in organizations like schools that have an intermediary server filtering the connection to the Internet), or in this snippet:

XDG_CACHE_DIR = os.environ.get(
    'XDG_CACHE_HOME',
    os.path.join(os.path.expanduser('~'), '.cache')
)

How do you know that these variables exist? The need for urllib.request.getproxies() is evident from the optional arguments in requests.get()—so part of this information comes from understanding the API of libraries you call. Environment variables are often utility-specific, so if a library is intended for use with a particular database or other sister application, those applications’ documentation list relevant environment variables. For plain POSIX systems, a good place to start is Ubuntu’s list of default environment variables, or else the base list of environment variables in the POSIX specification, which links to various relevant other lists.

Underscore-prefixed function names (we are all responsible users)

Almost every function in HowDoI is prefixed with an underscore. This identifies them as for internal use only. For most of them, this is because if called, there is the possibility of an uncaught exception—anything that calls _get_result() risks this—until the howdoi() function, which handles the possible exceptions.

The rest of the internal functions (_format_output(), _is_question(), _enable_cache(), and _clear_cache()) are identified as such because they’re simply not intended for use outside of the package. The testing script, howdoi/test_howdoi.py, only calls the nonprefixed functions, checking that the formatter works by feeding a command-line argument for colorization to the top-level howdoi.howdoi() function, rather than by feeding code to howdoi._format_output().

Handle compatibility in just one place (readability counts)

Differences between versions of possible dependencies are handled before the main code body so the reader knows there won’t be dependency issues, and version checking doesn’t litter the code elsewhere. This is nice because HowDoI is shipped as a command-line tool, and the extra effort means users won’t be forced to change their Python environment just to accommodate the tool. Here is the snippet with the workarounds:

try:
    from urllib.parse import quote as url_quote
except ImportError:
    from urllib import quote as url_quote

try:
    from urllib import getproxies
except ImportError:
    from urllib.request import getproxies

And the following snippet resolves the difference between Python 2 and Python 3’s Unicode handling in seven lines, by creating the function u(x) to either do nothing or emulate Python 3. Plus it follows Stack Overflow’s new citation guideline, by citing the original source:

# Handle Unicode between Python 2 and 3
# http://stackoverflow.com/a/6633040/305414
if sys.version < '3':
    import codecs
    def u(x):
        return codecs.unicode_escape_decode(x)[0]
else:
    def u(x):
        return x

Pythonic choices (beautiful is better than ugly)

The following snippet from howdoi.py shows thoughtful, Pythonic choices. The function get_link_at_pos() returns False if there are no results, or else identifies the links that are to Stack Overflow questions, and returns the one at the desired position (or the last one if there aren’t enough links):

def _is_question(link):  
    return re.search('questions/\d+/', link)

# [ ... skip a function ... ]

def get_link_at_pos(links, position):
    links = [link for link in links if _is_question(link)]  
    if not links:
        return False  

    if len(links) >= position:
        link = links[position-1]  
    else:
        link = links[-1]  
    return link  

The first function, _is_question(), is defined as a separate one liner, giving clear meaning to an otherwise opaque regular expression search.

The list comprehension reads like a sentence, thanks to the separate definition of _is_question() and meaningful variable names.

The early return statement flattens the code.

The additional step of assigning to the variable link here…

…and here, rather than two separate return statements with no named variable at all, reinforces the purpose of get_link_at_pos() with clear variable names. The code is self-documenting.

The single return statement at the highest indentation level explicitly shows that all paths through the code exit either right away—because there are no links—or at the end of the function, returning a link. Our quick rule of thumb works: we can read the first and last line of this function and understand what it does. (Given multiple links and a position, get_link_at_pos() returns one single link: the one at the given position.)

Read Diamond’s documentation

First, we can get a sense of what the project is and what it does by scanning the online documentation. Diamond’s goal is to make it easy to gather system metrics on clusters of machines. Originally open sourced by BrightCove, Inc., in 2011, it now has over 200 contributors.

After describing its history and purpose, the documentation tells you how to install it, and then says how to run it: just modify the example configuration file (in our download it’s in conf/diamond.conf.example), put it in the default location (/etc/diamond/diamond.conf) or a path you’ll specify on the command line, and you’re set. There’s also a helpful section on configuration in the Diamond wiki page.

From the command line, we can get the usage statement via diamond --help:

(venv)$ diamond --help
Usage: diamond [options]

Options:
  -h, --help            show this help message and exit
  -c CONFIGFILE, --configfile=CONFIGFILE
                        config file
  -f, --foreground      run in foreground
  -l, --log-stdout      log to stdout
  -p PIDFILE, --pidfile=PIDFILE
                        pid file
  -r COLLECTOR, --run=COLLECTOR
                        run a given collector once and exit
  -v, --version         display the version and exit
  --skip-pidfile        Skip creating PID file
  -u USER, --user=USER  Change to specified unprivileged user
  -g GROUP, --group=GROUP
                        Change to specified unprivileged group
  --skip-change-user    Skip changing to an unprivileged user
  --skip-fork           Skip forking (damonizing) process

From this, we know it uses a configuration file; by default, it runs in the background; it has logging; you can specifiy a PID (process ID) file; you can test collectors; you can change the process’s user and group; and it by default will daemonize (fork) the process.⁴

Use Diamond

To understand it even better, we can run Diamond. We need a modified configuration file, which we can put in a directory we make called Diamond/tmp. From inside the Diamond directory, type:

(venv)$ mkdir tmp
(venv)$ cp conf/diamond.conf.example tmp/diamond.conf

Then edit tmp/diamond.conf to look like this:

### Options for the server
[server]
# Handlers for published metrics.  
handlers = diamond.handler.archive.ArchiveHandler
user =  
group =
# Directory to load collector modules from  
collectors_path = src/collectors/

### Options for handlers  
[handlers]
[[default]]

[[ArchiveHandler]]
log_file = /dev/stdout

### Options for collectors
[collectors]
[[default]]
# Default Poll Interval (seconds)
interval = 20

### Default enabled collectors
[[CPUCollector]]
enabled = True

[[MemoryCollector]]
enabled = True

We can tell from the example configuration file that:

There are multiple handlers, which we can select by class name.

We have control over the user and group that the daemon runs as (empty means to use the current user and group).

We can specify a path to look for collector modules. This is how Diamond will know where the custom Collector subclasses are: we directly state it in the configuration file.

We can also store configure handlers individually.

Next, run Diamond with options that set logging to /dev/stdout (with default formatting configurations), that keep the application in the foreground, that skip writing the PID file, and that use our new configuration file:

(venv)$ diamond -l -f --skip-pidfile --configfile=tmp/diamond.conf

To end the process, type Ctrl+C until the command prompt reappears. The log output demonstrates what collectors and handlers do: collectors collect different metrics (such as the MemoryCollector’s total, available, free, and swap memory sizes), which the handlers format and send to various destinations, such as Graphite, MySQL, or in our test case, as log messages to /dev/stdout.

Reading Diamond’s code

IDEs can be useful when reading larger projects—they can quickly locate the original definitions of functions and classes in the source code. Or, given a definition, they can find all places in the project where it is used. For this functionality, set the IDE’s Python interpreter to the one in your virtual environment.⁵

Instead of following each function as we did with HowDoI, Figure 5-2 follows the import statements; the diagram just shows which modules in Diamond import which other modules. Drawing sketches like these helps by providing a very high-level look for larger projects: you hide the trees so you can see the forest. We can start with the diamond executable file on the top left and follow the imports through the Diamond project. Aside from the diamond executable, every square outline denotes a file (module) or directory (package) in the src/diamond directory.

Figure 5-2. The module import structure of Diamond

Diamond’s well-organized and appropriately named modules make it possible to get an idea of what the code is doing solely from our diagram: diamond gets the version from util, then sets up logging using utils.log and starts a Server instance using server. The Server imports from almost all of the modules in the utils package, using utils.classes to acess both the Handlers in handler and the collectors, config to read the configuration file and obtain settings for the collectors (and the extra paths to the user-defined collectors), and scheduler and signals to set the polling interval for the collectors to calculate their metrics, and to set up and start the handlers processing the queue of metrics to send them to their various destinations.

The diagram doesn’t include the helper modules convertor.py and gmetric.py, which are used by specific collectors, or the over 20 handler implementations defined in the handler subpackage, or the over 100 collector implementations defined in the project’s Diamond/src/collectors/ directory (which is installed elsewhere when not installed the way we did for reading—that is, using PyPI or Linux package distributions, instead of source). These are imported using diamond.classes.load_dynamic_class(), which then calls the function diamond.util.load_class_from_name() to load the classes from the string names given in the configuration file, so the import statements do not explicitly name them.

To understand why there is both a utils package and a util module, you have to dig into the actual code: the util module provides functions related more to Diamond’s packaging than to its operation—a function to get the version number from version.__VERSION__, and two functions that parse strings that identify either modules or classes, and import them.

Separate different functionality into namespaces (they are one honking great idea)

The diagram in Figure 5-2 shows the server module interacting with three other modules in the project: diamond.handler, diamond.collector, and diamond.utils. The utils subpackage could realistically have contained all of its classes and functions in a single, large util.py module, but there was an opportunity to use namespaces to separate code into related groups, and the development team took it. Honking great!

All of the implementations of Handlers are contained in diamond/handler (which makes sense), but the structure for the Collectors is different. There’s not a directory, only a module diamond/collector.py that defines the Collector and ProcessCollector base classes. All implementations of the Collectors are defined instead in Diamond/src/collectors/ and would be installed in the virtual environment under venv/share/diamond/collectors when installing from PyPI (as recommended) rather than from GitHub (like we did to read it). This helps the user to create new implementations of Collectors: placing all of the collectors in the same location makes it easier for the application to find them and easier for library users to follow their example.

Finally, each Collector implementation in Diamond/src/collectors is in its own directory (rather than in a single file), which makes it possible to keep each Collector implementation’s tests separate. Also honking great.

User-extensible custom classes (complex is better than complicated)

It’s easy to add new Collector implementations: just subclass the diamond.collector.Collector abstract base class,⁶ implement a Collector.collect() method, and place the implementation in its own directory in venv/src/collectors/.

Underneath, the implementation is complex, but the user doesn’t see it. This section shows both the simple user-facing part of Diamond’s Collector API and the complex code that makes this user interface possible.

# coding=utf-8
import diamond.collector
import psutil

class CPUCollector(diamond.collector.Collector):

    def collect(self):
        #  In Collector, this just contains raise(NotImplementedError)
        metric_name = "cpu.percent"
        metric_value = psutil.cpu_percent()
        self.publish(metric_name, metric_value)

[[CPUCollector]]
enabled = True
hostname_method = smart

def load_collectors(paths=None, filter=None):
    """Scan for collectors to load from path"""
    # Initialize return value
    collectors = {}
    log = logging.getLogger('diamond')

    if paths is None:
        return

    if isinstance(paths, basestring):  
        paths = paths.split(',')
        paths = map(str.strip, paths)

    load_include_path(paths)  

    for path in paths:
        ##~~ Skip lines that confirm 'path' exists.

        for f in os.listdir(path):

            # Are we a directory? If so, process down the tree
            fpath = os.path.join(path, f)
            if os.path.isdir(fpath):
                subcollectors = load_collectors([fpath])  
                for key in subcollectors:  
                    collectors[key] = subcollectors[key]

            # Ignore anything that isn't a .py file
            elif (os.path.isfile(fpath)
                  ##~~ ... Skip tests confirming fpath is a Python module ...
                  ):

                ##~~ ... Skip the part that ignores filtered paths ...
                modname = f[:-3]

                try:
                    # Import the module
                    mod = __import__(modname, globals(), locals(), ['*'])  
                except (KeyboardInterrupt, SystemExit), err:
                    ##~~ ... Log the exception and quit ...
                except:
                    ##~~ ... Log the exception and continue ...

                # Find all classes defined in the module
                for attrname in dir(mod):
                    attr = getattr(mod, attrname)  
                    # Only attempt to load classes that are subclasses
                    # of Collectors but are not the base Collector class
                    if (inspect.isclass(attr)
                            and issubclass(attr, Collector)
                            and attr != Collector):
                        if attrname.startswith('parent_'):
                            continue
                        # Get class name
                        fqcn = '.'.join([modname, attrname])
                        try:
                            # Load Collector class
                            cls = load_dynamic_class(fqcn, Collector)  
                            # Add Collector class
                            collectors[cls.__name__] = cls  
                        except Exception:
                            ##~~ log the exception and continue ...

    # Return Collector classes
    return collectors

Example use of a closure (when the gotcha isn’t a gotcha)

A closure is a function that makes use of variables available in the local scope that would otherwise not be available when the function is called. They can be difficult to implement and understand in other languages, but are not hard to implement in Python, because Python treats functions just like any other object.⁹ For example, functions can be passed around as arguments, or returned from other functions. Here’s an example excerpt from the diamond executable that shows how to implement a closure in Python:

##~~ ... Skip the import statements ...  

def main():
    try:
        ##~~  ... Skip code that creates the command-line parser ...

        # Parse command-line Args
        (options, args) = parser.parse_args()

        ##~~  ... Skip code that parses the configuration file ...
        ##~~  ... Skip code that sets up the logger ...

    # Pass the exit upstream rather then handle it as an general exception
    except SystemExit, e:
        raise SystemExit

    ##~~  ... Skip code that handles other exceptions related to setup ...

    try:
        # PID MANAGEMENT  
        if not options.skip_pidfile:
            # Initialize PID file
            if not options.pidfile:
                options.pidfile = str(config['server']['pid_file'])

            ##~~ ... Skip code to open and read the PID file if it exists, ...
            ##~~ ... and then delete the file if there is no such PID ...
            ##~~ ... or exits if there is already a running process. ...


        ##~~  ... Skip the code that sets the group and user ID ...
        ##~~  ... and the code that changes the PID file permissions. ...

        ##~~  ... Skip the code that checks whether to run as a daemon, ...
        ##~~  ... and if so detaches the process. ...


        # PID MANAGEMENT  
        if not options.skip_pidfile:
            # Finish initializing PID file
            if not options.foreground and not options.collector:
                # Write PID file
                pid = str(os.getpid())
                try:
                    pf = file(options.pidfile, 'w+')
                except IOError, e:
                    log.error("Failed to write child PID file: %s" % (e))
                    sys.exit(1)
                pf.write("%s\n" % pid)
                pf.close()
                # Log
                log.debug("Wrote child PID file: %s" % (options.pidfile))

        # Initialize server
        server = Server(configfile=options.configfile)

        def sigint_handler(signum, frame):  
            log.info("Signal Received: %d" % (signum))
            # Delete PID file
            if not options.skip_pidfile and os.path.exists(options.pidfile):  
                os.remove(options.pidfile)
                # Log
                log.debug("Removed PID file: %s" % (options.pidfile))
            sys.exit(0)

        # Set the signal handlers
        signal.signal(signal.SIGINT, sigint_handler)  
        signal.signal(signal.SIGTERM, sigint_handler)

        server.run()

    # Pass the exit upstream rather then handle it as a general exception
    except SystemExit, e:
        raise SystemExit

    ##~~  ... Skip code that handles any other exceptions ...
    ##~~  ... and all of the rest of the script.

When we skip code, the missing parts will be summarized by a comment preceded by two tildes (##~~ like this).

The reason for the PID¹⁰ file is to make sure the daemon is unique (i.e., not accidentally started twice), to communicate the associated process ID quickly to other scripts, and to make it evident that an abnormal termination has occurred (because in this script, the PID file is deleted upon normal termination).

All of this code is just to provide context leading up to the closure. At this point, either the process is running as a daemon (and now has a different process ID than before) or it will skip this part because it’s already written its correct PID to the PID file.

This (sigint_handler()) is the closure. It is defined inside of main(), rather than at the top level, outside of any functions, because it needs to know whether to look for a PID file, and if so where to look.

It gets this information from the command-line options, which it can’t obtain until after the call to main(). That means all of the options related to the PID file are local variables in main’s namespace.

The closure (the function sigint_handler()) is sent to the signal handler and will be used to handle SIGINT and SIGTERM.

Read Tablib’s documentation

Tablib’s documentation starts off immediately with a use case, and then goes into describing its capabilities in more detail: it provides a Dataset object that has rows, headers, and columns. You can do I/O from various formats to the Dataset object. And the advanced usage section says you can add tags to rows, and create derived columns that are functions of other columns.

Use Tablib

Tablib is a library, not an executable like HowDoI or Diamond, so you can open a Python interactive session and have the expectation that you can use use the help() function to explore the API. Here’s our example of the tablib.Dataset class, the different data formats, and how I/O works:

>>> import tablib
>>> data = tablib.Dataset()
>>> names = ('Black Knight', 'Killer Rabbit')
>>>
>>> for name in names:
...     fname, lname = name.split()
...     data.append((fname, lname))
...
>>> data.dict
[['Black', 'Knight'], ['Killer', 'Rabbit']]
>>>
>>> print(data.csv)
Black,Knight
Killer,Rabbit

>>> data.headers=('First name', 'Last name')
>>> print(data.yaml)
- {First name: Black, Last name: Knight}
- {First name: Killer, Last name: Rabbit}

>>> with open('tmp.csv', 'w') as outfile:
...     outfile.write(data.csv)
...
64
>>> newdata = tablib.Dataset()
>>> newdata.csv = open('tmp.csv').read()
>>> print(newdata.yaml)
- {First name: Black, Last name: Knight}
- {First name: Killer, Last name: Rabbit}

Read Tablib’s code

The file structure under tablib/ looks like this:

tablib
|--- __init__.py
|--- compat.py
|--- core.py
|--- formats/
|--- packages/

The two directories, tablib/formats/ and tablib/packages/, will be discussed in a few sections.

Python supports module-level docstrings as well as the docstrings we’ve already described—a string literal that is the first statement in a function, class, or class method. Stack Overflow has good advice on how to document a module. For us, this means another way to explore source code is by typing head *.py in a terminal shell while in the directory at the top level of the package—to show all of the module docstrings at once. Here’s what we get:

(venv)$ cd tablib
(venv)$ head *.py
==> __init__.py <==  
""" Tablib. """

from tablib.core import (
    Databook, Dataset, detect, import_set, import_book,
    InvalidDatasetType, InvalidDimensions, UnsupportedFormat,
    __version__
)



==> compat.py <==  
# -*- coding: utf-8 -*-

"""
tablib.compat
~~~~~~~~~~~~~

Tablib compatiblity module.

"""


==> core.py <==  
# -*- coding: utf-8 -*-
"""
    tablib.core
    ~~~~~~~~~~~

    This module implements the central Tablib objects.

    :copyright: (c) 2014 by Kenneth Reitz.
    :license: MIT, see LICENSE for more details.
"""

We learn that:

The top-level API (the contents of __init__.py are accessible from tablib after an import tablib statement) has just nine entry points: the Databook and Dataset classes are mentioned in the documentation, detect could be for identifying formatting, import_set and import_book must import data, and the last three classes—InvalidDatasetType, InvalidDimensions, and UnsupportedFormat—look like exceptions. (When code follows PEP 8, we can tell which objects are custom classes from their capitalization.)

tablib/compat.py is a compatibility module. A quick look inside will show that it handles Python 2/Python 3 compatibility issues in a similar way to HowDoI, by resolving different locations and names to the same symbol for use in tablib/core.py.

tablib/core.py, like it says, implements the central Tablib objects like Dataset and Databook.

No needless object-oriented code in formats (use namespaces for grouping functions)

The formats directory contains all of the defined file formats for I/O. The module names, _csv.py, _tsv.py, _json.py, _yaml.py, _xls.py, _xlsx.py, _ods.py, and _xls.py are prefixed with an underscore—this indicates to the library user that they are not intended for direct use. We can change directories into formats, and search for classes and functions. Using grep ^class formats/*.py reveals there are no class definitions, and using grep ^def formats/*.py shows that each module contains some or all of the following functions:

• detect(stream) infers the file format based on the stream content.

• dset_sheet(dataset, ws) formats the Excel spreadsheet cells.

• export_set(dataset) exports the Dataset to the given format, returning a formatted string with the new format. (Or, for Excel, returning a bytes object—or a binary-formatted string in Python 2.)

• import_set(dset, in_stream, headers=True) replaces the contents of the dataset with the contents of the input stream.

• export_book(databook) exports the Datasheets in the Databook to the given format, returning a string or bytes object.

• import_book(dbook, in_stream, headers=True) replaces the contents of the databook with the contents of the input stream.

This is an example of using modules as namespaces (after all, they are one honking great idea) to separate functions, rather than using unnecessary classes. We know each function’s purpose from its name: for example, formats._csv.import_set(), formats._tsv.import_set(), and formats._json.import_set() import datasets from CSV, TSV, and JSON-formatted files, respectively. The other functions do data exporting and file format detection, when possible, for each of Tablib’s available formats.

Descriptors and the property decorator (engineer immutability when the API would benefit)

Tablib is our first library that uses Python’s decorator syntax, described in “Decorators”. The syntax uses the @ symbol in front of a function name, placed directly above another function. It modifies (or “decorates”) the function directly below. In the following excerpt, property changes the functions Dataset.height and Dataset.width into descriptors—classes with at least one of the __get__(), __set__(), or __delete__() (“getter”, “setter”, or “delete”) methods defined. For example, the attribute lookup Dataset.height will trigger the getter, setter, or delete function depending on the context in which that attribute is used. This behavior is only possible for new-style classes, discussed momentarily. See this useful Python tutorial on descriptors for more information.

class Dataset(object):
    #
    # ... omit the rest of the class definition for clarity
    #

    @property  
    def height(self):
        """The number of rows currently in the :class:`Dataset`.
           Cannot be directly modified.  
        """
        return len(self._data)


    @property
    def width(self):
        """The number of columns currently in the :class:`Dataset`.
           Cannot be directly modified.
        """
        try:
            return len(self._data[0])
        except IndexError:
            try:
                return len(self.headers)
            except TypeError:
                return 0

This is how to use a decorator. In this case, property modifies Dataset.height to behave as a property rather than as a bound method. It can only operate on class methods.

When property is applied as a decorator, the height attribute will return the height of the Dataset but it is not possible to assign a height to the Dataset by invoking Dataset.height.

Here is what the height and width attributes look like when used:

>>> import tablib
>>> data = tablib.Dataset()
>>> data.header = ("amount", "ingredient")
>>> data.append(("2 cubes", "Arcturan Mega-gin"))
>>> data.width
2
>>> data.height
1
>>>
>>> data.height = 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: can't set attribute

So, data.height can be accessed like an attribute, but it’s not settable—it’s calculated from the data and so is always current. This is ergonomic API design: data.height is easier to type than data.get_height(); it’s clear what the meaning of data.height is; and because it is calculated from the data (and the property is not settable—only the “getter” function is defined), there isn’t a danger that it will be out of sync from the correct number.

The property decorator can only be applied to attributes of classes, and only to classes that derive from the base object object (e.g., class MyClass(object) not class MyClass()—inheritance from object is always the case in Python 3).

This same tool is used to create Tablib’s data import and export API in the various formats: Tablib does not store the string value for each of the CSV, JSON, and YAML outputs. Rather, the Dataset attributes csv, json, and yaml are properties, like Dataset.height and Dataset.width in the preceding example—they call a function that generates the result from the stored data or parses the input format and then replaces the core data. But there’s only one dataset.

When data.csv is on the left of an equals sign, the property’s “setter” function is called, to parse the dataset from the CSV format. And when data.yaml is on the right of an equals sign, or alone, the “getter” is called, to create a string with the given format from the internal dataset. Here is an example:

>>> import tablib
>>> data = tablib.Dataset()
>>>
>>> data.csv = "\n".join((  
...     "amount,ingredient",
...     "1 bottle,Ol' Janx Spirit",
...     "1 measure,Santraginus V seawater",
...     "2 cubes,Arcturan Mega-gin",
...     "4 litres,Fallian marsh gas",
...     "1 measure,Qalactin Hypermint extract",
...     "1 tooth,Algolian Suntiger",
...     "a sprinkle,Zamphuor",
...     "1 whole,olive"))
>>>
>>> data[2:4]
[('2 cubes', 'Arcturan Mega-gin'), ('4 litres', 'Fallian marsh gas')]
>>>
>>> print(data.yaml)  
- {amount: 1 bottle, ingredient: Ol Janx Spirit}
- {amount: 1 measure, ingredient: Santraginus V seawater}
- {amount: 2 cubes, ingredient: Arcturan Mega-gin}
- {amount: 4 litres, ingredient: Fallian marsh gas}
- {amount: 1 measure, ingredient: Qalactin Hypermint extract}
- {amount: 1 tooth, ingredient: Algolian Suntiger}
- {amount: a sprinkle, ingredient: Zamphuor}
- {amount: 1 whole, ingredient: olive}

data.csv on the lefthand side of the equals sign (assignment operator) invokes formats.csv.import_set(), with data as the first argument, and the string of Gargle Blaster ingredients as its second argument.

data.yaml alone invokes formats.yaml.export_set(), with data as its argument, outputting the formatted YAML string for the print() function.

The “getter”, “setter”, and also a “deleter” function can be bound to a single attribute using property. Its signature is property(fget=None, fset=None, fdel=None, doc=None), in which fget identifies the “getter” function (formats.csv.import_set()), fset identifies the “setter” function (formats.csv.export_set()), and fdel identifies the “deleter” function, which is left as None. We will see the code where the formatting properties are set, programmatically, next.

Programmatically registered file formats (don’t repeat yourself)

Tablib places all of the file formatting routines in the formats subpackage. This structure choice makes the main core.py module cleaner and the entire package modular; it’s easy to add new file formats. Although it would have been possible to paste chunks of nearly identical code and import each file format’s import and export behaviors separately, all of the formats are programmatically loaded into the Dataset class to properties named after each format.

We’re printing the entire contents of formats/__init__.py in the following code example because it’s not too large a file, and we want to show where formats.available is defined:

# -*- coding: utf-8 -*-  

""" Tablib - formats
"""

from . import _csv as csv
from . import _json as json
from . import _xls as xls
from . import _yaml as yaml
from . import _tsv as tsv
from . import _html as html
from . import _xlsx as xlsx
from . import _ods as ods

available = (json, xls, yaml, csv, tsv, html, xlsx, ods)  

This line explicitly tells the Python interpreter that the file encoding is UTF-8.¹¹

Here’s the definition of formats.available, right in formats/__init__.py. It’s also available via dir(tablib.formats), but this explicit list is easier to understand.

In core.py, rather than about 20 (ugly, hard to maintain) repeated function definitions for each format option, the code imports each format programmatically by calling self._register_formats() at the end of the Dataset’s __init__() method. We’ve excerpted just Dataset._register_formats() here:

class Dataset(object):
    #
    #  ... skip documentation and some definitions  ...
    #

    @classmethod  
    def _register_formats(cls):
        """Adds format properties."""
        for fmt in formats.available:  
            try:
                try:
                    setattr(cls, fmt.title,
                    property(fmt.export_set, fmt.import_set))  
                except AttributeError:  
                    setattr(cls, fmt.title, property(fmt.export_set))  

            except AttributeError:
                pass  

    #
    # ... skip more definitions ...
    #

    @property  
    def tsv():
        """A TSV representation of the :class:`Dataset` object. The top
        row will contain headers, if they have been set. Otherwise, the
        top row will contain the first row of the dataset.

        A dataset object can also be imported by setting
        the :class:`Dataset.tsv` attribute. ::

            data = tablib.Dataset()
            data.tsv = 'age\tfirst_name\tlast_name\n90\tJohn\tAdams'  

        Import assumes (for now) that headers exist.
        """
        pass

The @classmethod symbol is a decorator, described more extensively in “Decorators”, that modifies the method _register_formats() so that it passes the object’s class (Dataset) rather than the object instance (self) as its first argument.

The formats.available is defined in formats/__init__.py and contains all of the available formatting options.

In this line, setattr assigns a value to the attribute named fmt.title (i.e., Dataset.csv or Dataset.xls). The value it assigns is a special one; property(fmt.export_set, fmt.import_set) turns Dataset.csv into a property.

There will be an AttributeError if fmt.import_set is not defined.

If there is no import function, try to assign just the export behavior.

If there is neither an export nor an import function to assign, just don’t assign anything.

Each of the file formats is defined as a property here, with a descriptive docstring. The docstring will be retained when property() is called at tag or to assign the extra behaviors.

The \t and \n are string escape sequences that represent the Tab character and a newline, respectively. They’re all listed in Python’s string literals documentation.

Vendorized dependencies in packages (an example of how to vendorize)

Tablib’s dependencies are currently vendorized (meaning they are shipped bundled with the code—in this case, in the directory packages) but may be moved to a plug-in system in the future. The packages directory contains third-party packages included inside Tablib to ensure compatibility, rather than the other option, which is to specify versions in the setup.py file that will be downloaded and installed when Tablib is installed. This technique is discussed in “Vendorizing Dependencies”; the choice for Tablib was made both to reduce the number of dependencies the user would have to download, and because sometimes there are different packages for Python 2 and Python 3, which are both included. (The appropriate one is imported, and their functions set to a common name, in tablib/compat.py). That way, Tablib can have one single code base instead of two—one for each version of Python. Because each of these dependencies has its own license, a NOTICE document was added to the top level of the project directory that lists each dependency’s license.

Saving memory with __slots__ (optimize judiciously)

Python prefers readability over speed. Its entire design, its Zen aphorisms, and its early influence from educational languages like ABC are all about placing user-friendliness above performance (although we’ll talk about more optimization options in “Speed”).

The use of __slots__ in tablib is a case where optimization matters. This is a slightly obscure reference, and it’s only available for new-style classes (described in a few pages), but we want to show that it’s possible to optimize Python when necessary. This optimization is only useful when you have tons of very small objects by reducing the footprint of each class instance by the size of one dictionary (large objects would make this small savings irrelevant, and fewer objects make the savings not worth it). Here is an excerpt from the __slots__ documentation:

By default, instances of classes have a dictionary for attribute storage. This wastes space for objects having very few instance variables. The space consumption can become acute when creating large numbers of instances.

The default can be overridden by defining __slots__ in a class definition. The __slots__ declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because __dict__ is not created for each instance.

Normally, this isn’t something to care about—notice that __slots__ doesn’t appear in the Dataset or Databook classes, just the Row class—but because there can be thousands of rows of data, __slots__ is a good idea. The Row class is not exposed in tablib/__init__.py because it is a helper class to Dataset, instantiated once for every row. This is how its definition looks in the beginning part of the definition of the Row class:

class Row(object):
    """Internal Row object. Mainly used for filtering."""

    __slots__ = ['_row', 'tags']

    def __init__(self, row=list(), tags=list()):
        self._row = list(row)
        self.tags = list(tags)

    #
    #  ... etc. ...
    #

The problem now is that there is no longer a __dict__ attribute in the Row instances, but the pickle.dump() function (used for object serialization) by default uses __dict__ to serialize the object unless the method __getstate__() is defined. Likewise, during unpickling (the process that reads the serialized bytes and reconstructs the object in memory), if __setstate__() is not defined, pickle.load() will load to the object’s __dict__ attribute. Here is how to get around that:

class Row(object):
    #
    #  ... skip the other definitions ...
    #

    def __getstate__(self):

        slots = dict()

        for slot in self.__slots__:
            attribute = getattr(self, slot)
            slots[slot] = attribute
        return slots

    def __setstate__(self, state):
        for (k, v) in list(state.items()):
            setattr(self, k, v)

For more information about __getstate__() and __setstate__() and pickling, see the __getstate__ documentation.

Operator overloading (beautiful is better than ugly)

This code section uses Python’s operator overloading to enable operations on either the Dataset’s rows or columns. The following first sample code shows interactive use of the bracket operator ([ ]) for both numerical indices and column names, and the second one shows the code that uses this behavior:

>>> data[-1]  
('1 whole', 'olive')
>>>
>>> data[-1] = ['2 whole', 'olives']  
>>>
>>> data[-1]
('2 whole', 'olives')  
>>>
>>> del data[2:7]  
>>>
>>> print(data.csv)
amount,ingredient  
1 bottle,Ol' Janx Spirit
1 measure,Santraginus V seawater
2 whole,olives

>>> data['ingredient']  
["Ol' Janx Spirit", 'Santraginus V seawater', 'olives']

When using numbers, accessing data via the bracket operator ([]) gives the row at the specified location.

This is assignment using the bracket operator …

… it becomes 2 olives instead of the original one.

This is deletion using a slice—2:7 denotes all of the numbers 2,3,4,5,6 but not 7.

See how the recipe afterward is much smaller.

It is also possible to access columns by name.

The part of the Dataset code that defines the behavior of the bracket operator shows how to handle access both by column name and by row number:

class Dataset(object):
    #
    #  ... skip  the rest of the definitions for brevity ...
    #

    def __getitem__(self, key):
        if isinstance(key, str) or isinstance(key, unicode):  
            if key in self.headers:  
                pos = self.headers.index(key) # get 'key' index from each data
                return [row[pos] for row in self._data]
            else:  
                raise KeyError
        else:
            _results = self._data[key]
            if isinstance(_results, Row):  
                return _results.tuple
            else:
                return [result.tuple for result in _results]  


    def __setitem__(self, key, value): 
        self._validate(value)
        self._data[key] = Row(value)


    def __delitem__(self, key):
        if isinstance(key, str) or isinstance(key, unicode):  
            if key in self.headers:
                pos = self.headers.index(key)
                del self.headers[pos]

                for row in self._data:
                    del row[pos]
            else:
                raise KeyError
        else:
            del self._data[key]

First, check whether we are seeking a column (True if key is a string) or a row (True if the key is an integer or slice).

Here the code checks for the key to be in self.headers, and then…

…explicitly raises a KeyError so that access by column name behaves as one would expect a dictionary to. The whole if/else pair is not necessary for the operation of the function—if it were omitted, a ValueError would still be raised by self.headers.index(key) if key were not in self.headers. The only purpose for this check is to provide a more informative error for the library user.

This is how the code determines whether key was a number or a slice (like 2:7). If a slice, the _results would be a list, not a Row.

Here is where the slice is processed. Because the rows are returned as tuples, the values are an immutable copy of the acual data, and the dataset’s values (actually stored as lists) won’t accidentally be corrupted by an assignment.

The __setitem__() method can change a single row but not a column. This is intentional; there is no way provided to change the content of an entire column; and for data integrity, this is probably not a bad choice. The user can always transform the column and insert it at any position using one of the methods insert_col(), lpush_col(), or rpush_col().

The __delitem__() method can either delete a column or a row, using the same logic as __getitem__().

For more information about additional operator overloading and other special methods, see the Python documentation on Special method names.

Read Requests’s documentation

Requests is a bigger package, so first just scan the section titles from the Requests documentation. Requests extends urrlib and httplib from Python’s standard library to provide methods that perform HTTP requests. The library includes support for international domains and URLs, automatic decompression, automatic content decoding, browser style SSL verification, HTTP(S) proxy support, and other features, which are all defined by the Internet Engineering Task Force (IETF) standards for HTTP in their requests for comment (RFCs) 7230 through 7235.¹²

Requests strives to cover all of the IETF’s HTTP specifications, using only a handful of functions, a bunch of keyword arguments, and a few featureful classes.

Use Requests

Like with Tablib, there is enough information in the docstrings to use Requests without actually reading the online documentation. Here’s a brief interaction:

>>> import requests
>>> help(requests)  # Shows a usage statement and says to see `requests.api`
>>> help(requests.api)  # Shows a detailed API description
>>>
>>> result = requests.get('https://pypi.python.org/pypi/requests/json')
>>> result.status_code
200
>>> result.ok
True
>>> result.text[:42]
'{\n    "info": {\n        "maintainer": null'
>>>
>>> result.json().keys()
dict_keys(['info', 'releases', 'urls'])
>>>
>>> result.json()['info']['summary']
'Python HTTP for Humans.'

Read Requests’s code

Here are the contents of the Requests package:

$ ls
__init__.py      cacert.pem     exceptions.py    sessions.py
adapters.py      certs.py         hooks.py         status_codes.py
api.py           compat.py        models.py        structures.py
auth.py          cookies.py       packages/      utils.py

cacert.pem is a default certificate bundle to use when checking SSL certificates.

Requests has a flat structure, except for a packages directory that vendorizes (contains the external libraries) chardet and urllib3. These dependencies are imported as requests.packages.chardet and requests.packages.urllib3, so programmers can still access chardet and urllib3 from the standard library.

We can mostly figure out what’s happening thanks to well-chosen module names, but if we want a little more imformation, we can again peek at the module docstrings by typing head *.py in the top-level directory. The following lists displays these module docstrings, slightly truncated. (It doesn’t show compat.py. We can tell from its name, especially because it’s named the same as in Reitz’s Tablib library, that it takes care of Python 2 to Python 3 compatibility.)

api.py

Implements the Requests API

hooks.py

Provides the capabilities for the Requests hooks system

models.py

Contains the primary objects that power Requests

sessions.py

Provides a Session object to manage and persist settings across requests (cookies, auth, proxies)

auth.py

Contains the authentication handlers for Requests

status_codes.py

A lookup table mapping status titles to status codes

cookies.py

Compatibility code to be able to use cookielib.CookieJar with requests

adapters.py

Contains the transport adapters Requests uses to define and maintain connections

exceptions.py

All of Requests’ exceptions

structures.py

Data structures that power Requests

certs.py

Returns the preferred default CA certificate bundle listing trusted SSL certificates

utils.py

Provides utility functions that are used within Requests that are also useful for external consumption

Insights from reading all of the headers:

• There is a hook system (hooks.py), implying the user can modify how Requests works. We won’t discuss it in depth because it will take us too far off topic.

• The main module is models.py, as it contains “the primary objects that power Requests.”

• The reason sessions.Session exists is to persist cookies across multiple requests (that might occur during authentication, for example).

• The actual HTTP connection is made by objects from adapters.py.

• The rest is kind of obvious: auth.py is for authentication, status_codes.py has the status codes, cookies.py is for adding and removing cookies, exceptions.py is for exceptions, structures.py contains data structures (e.g., a case-insensitive dictionary), and utils.py contains utility functions.

The idea to put communication separately in adapters.py is innovative (at least to this writer). It means models.Request, models.PreparedRequest, and models.Response don’t actually do anything—they just store data, possibly manipulating it a bit for presentation, pickling, or encoding purposes. Actions are handled by separate classes that exist specifically to perform an action, like authentication or communication. Every class does just one thing, and each module contains classes that do similar things—a Pythonic approach most of us already adhere to with our function definitions.

Top-level API (preferably only one obvious way to do it)

The functions defined in api.py (except request()) are named after HTTP request methods.¹³ Each request method is the same except for its method name and the choice of exposed keyword parameters, so we’re truncating this exerpt from requests/api.py after the get() function:

# -*- coding: utf-8 -*-

"""
requests.api
~~~~~~~~~~~~

This module implements the Requests API.

:copyright: (c) 2012 by Kenneth Reitz.
:license: Apache2, see LICENSE for more details.

"""

from . import sessions


def request(method, url, **kwargs):  
    """Constructs and sends a :class:`Request <Request>`.

    :param method: method for the new :class:`Request` object.
    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string
                   for the :class:`Request`.

    ... skip the documentation for the remaining keyword arguments ...  

    :return: :class:`Response <Response>` object
    :rtype: requests.Response

    Usage::

      >>> import requests
      >>> req = requests.request('GET', 'http://httpbin.org/get')
      <Response [200]>
    """

    # By using the 'with' statement, we are sure the session is closed, thus we
    # avoid leaving sockets open which can trigger a ResourceWarning in some
    # cases, and look like a memory leak in others.
    with sessions.Session() as session:  
        return session.request(method=method, url=url, **kwargs)


def get(url, params=None, **kwargs):  
    """Sends a GET request.

    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string
                   for the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response
    """

    kwargs.setdefault('allow_redirects', True) 
    return request('get', url, params=params, **kwargs) 

The request() function contains a **kwargs in its signature. This means extraneous keyword arguments will not cause an exception, and it hides options from the user.

The documentation omitted here for brevity describes every keyword argument that has an associated action. If you use **kwargs in your function signature, this is the only way the user can tell what the contents of **kwargs should be, short of looking at the code themselves.

The with statement is how Python supports a runtime context. It can be used with any object that has an __enter__() and an __exit__() method defined. __enter()__ will be called upon entering the with statement, and __exit__() will be called upon exit, regardless of whether that exit is normal or due to an exception.

The get() function specifically pulls out the params=None keyword, applying a default value of None. The params keyword argument relevant for get because it’s for terms to be used in an HTTP query string. Exposing selected keyword arguments gives flexibility to the advanced user (via the remaining **kwargs) while making usage obvious for the 99% of people who don’t need advanced options.

The default for the request() function is to not allow redirects, so this step sets it to True unless the user set it already.

The get() function then simply calls request() with its first parameter set to "get". Making get a function has two advantages over just using a string argument like request("get", ...). First, it becomes obvious, even without documentation, which HTTP methods are available with this API. Second, if the user makes a typographical error in the method name, a NameError will be raised earlier, and probably with a less confusing traceback, than would happen with error checking deeper in the code.

No new functionality is added in requests/api.py; it exists to present a simple API for the user. Plus, putting the string HTTP methods directly into the API as function names means any typographical error with the method name will be caught and identified early, for example:

>>> requests.foo('http://www.python.org')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'foo'
>>>
>>> requests.request('foo', 'http://www.python.org')
<Response [403]>

The Request and PreparedRequest objects (we’re all responsible users)

__init__.py exposes Request, PreparedRequest, and Response from models.py as part of the main API. Why does models.Request even exist? There’s already a urllib.requests.Request in the standard library, and in cookies.py there is specifically a MockRequest object that wraps models.Request so that it works like urllib.requests.Request for http.cookiejar.¹⁴ That means whatever methods are needed for the Request object to interface with the cookies library are intentionally excluded from requests.Request. What is the point of all this extra work?

The extra methods in MockRequest (which exists to emulate urllib.request.Request for the cookie library) are used by the cookie library to manage cookies. Except for the get_type() function (which usually returns “http” or “https” when using Requests) and the unverifiable property (True for our case), they’re all related to the URL or the request headers:

Related to the header

add_unredirected_header()

Add a new key, value pair to the header.

get_header()

Get a specific name in the header dictionary.

get_new_headers()

Get the dictionary containing new headers (added by cookielib).

has_header()

Check whether a name exists in the header dictionary.

Related to the URL

get_full_url()

Does just what it says.

host and origin_req_host

Properties that are set by calling the methods get_host() and get_origin_req_host(), respectively.

get_host()

Extract the host from the URL (e.g., www.python.org from https://www.python.org/dev/peps/pep-0008/).

get_origin_req_host()

Call get_host().¹⁵

They’re all access functions, except for MockRequest.add_unredirected_header(). The MockRequest docstring notes that “the original request object is read-only.”

In requests.Request, data attributes are instead directly exposed. This makes all of the accessor functions unnecessary: to get or set the headers, access request-instance.headers. It’s just a dictionary. Likewise, the user can just get or change the string URL: request-instance.url.

The PreparedRequest object is initialized empty, and is populated with a call to prepared-request-instance.prepare(), filled with the relevant data (usually from the calling the Request object). It’s at this point that things like correct capitalization and encoding are applied. The object’s contents will, once prepared, be ready to send to the server, but every attribute is still directly exposed. Even PreparedRequest._cookies is exposed, although its prepended underscore is a gentle reminder that the attribute is not intended for use outside of the class, without forbidding such access (we are all responsible users).

This choice exposes the objects to user modification, but they are much more readable, and a little bit of extra work inside of PreparedRequest corrects capitalization issues and allows use of a dictionary in place of a CookieJar (look for the if isinstance()/else statement):

#
#  ... from models.py ...
#

class PreparedRequest():
    #
    #  ... skip everything else ...
    #

    def prepare_cookies(self, cookies):
        """Prepares the given HTTP cookie data.

        This function eventually generates a ``Cookie`` header from the
        given cookies using cookielib. Due to cookielib's design, the header
        will not be regenerated if it already exists, meaning this function
        can only be called once for the life of the
        :class:`PreparedRequest <PreparedRequest>` object. Any subsequent calls
        to ``prepare_cookies`` will have no actual effect, unless the "Cookie"
        header is removed beforehand."""

        if isinstance(cookies, cookielib.CookieJar):
            self._cookies = cookies
        else:
            self._cookies = cookiejar_from_dict(cookies)

        cookie_header = get_cookie_header(self._cookies, self)
        if cookie_header is not None:
            self.headers['Cookie'] = cookie_header

These things may not seem like a big deal, but it’s small choices like these that make an API intuitive to use.

Sets and set arithmetic (a nice, Pythonic idiom)

We haven’t yet shown an example use of Python sets in action. Python sets behave like sets in math—you can do subtraction, unions (the or operator), and intersections (the and operator):

>>> s1 = set((7,6))
>>> s2 = set((8,7))
>>> s1
{6, 7}
>>> s2
{8, 7}
>>> s1 - s2  # set subtraction
{6}
>>> s1 | s2  # set union
{8, 6, 7}
>>> s1 & s2  # set intersection
{7}

Here’s one, down toward the end of this function from cookies.py (with the label ):

#
# ... from cookies.py ...
#

def create_cookie(name, value, **kwargs):  
    """Make a cookie from underspecified parameters.

    By default, the pair of `name` and `value` will be set for the domain ''
    and sent on every request (this is sometimes called a "supercookie").
    """
    result = dict(
        version=0,
        name=name,
        value=value,
        port=None,
        domain='',
        path='/',
        secure=False,
        expires=None,
        discard=True,
        comment=None,
        comment_url=None,
        rest={'HttpOnly': None},
        rfc2109=False,)

    badargs = set(kwargs) - set(result)  
    if badargs:
        err = 'create_cookie() got unexpected keyword arguments: %s'  
        raise TypeError(err % list(badargs))  

    result.update(kwargs)  
    result['port_specified'] = bool(result['port']) 
    result['domain_specified'] = bool(result['domain'])
    result['domain_initial_dot'] = result['domain'].startswith('.')
    result['path_specified'] = bool(result['path'])

    return cookielib.Cookie(**result) 

The **kwargs specification allows the user to provide any or none of the keyword options for a cookie.

Set arithmetic! Pythonic. Simple. And in the standard library. On a dictionary, set() forms a set of the keys.

This is a great example of spliting a long line into two shorter lines that make much better sense. No harm done from the extra err variable.

The result.update(kwargs) updates the result dictionary with the key/value pairs in the kwargs dictionary, replacing existing pairs or creating ones that didn’t exist.

Here the call to bool() coerces the value to True if the object is truthy (meaning it evaluates to True—in this case, bool(result['port']) evaluates to True if it’s not None and it’s not an empty container).

The signature to initialize cookielib.Cookie is actually 18 positional arguments and one keyword argument (rfc2109 defaults to False). It’s impossible for us average humans to memorize which position has which value, so here Requests takes advantage of being able to assign positional arguments by name as keyword arguments, sending the whole dictionary.

Status codes (readability counts)

The entire status_codes.py exists to create an object that can look up status codes by attribute. We’re showing the definition of the lookup dictionary in status_codes.py first, and then an excerpt of code that uses it from sessions.py:

#
#  ... excerpted from requests/status_codes.py ...
#

_codes = {

    # Informational.
    100: ('continue',),
    101: ('switching_protocols',),
    102: ('processing',),
    103: ('checkpoint',),
    122: ('uri_too_long', 'request_uri_too_long'),
    200: ('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\\o/', '✔'),  
    201: ('created',),
    202: ('accepted',),
    #
    #  ... skipping ...
    #

    # Redirection.
    300: ('multiple_choices',),
    301: ('moved_permanently', 'moved', '\\o-'),
    302: ('found',),
    303: ('see_other', 'other'),
    304: ('not_modified',),
    305: ('use_proxy',),
    306: ('switch_proxy',),
    307: ('temporary_redirect', 'temporary_moved', 'temporary'),
    308: ('permanent_redirect',
          'resume_incomplete', 'resume',), # These 2 to be removed in 3.0  

    #
    #  ... skipping the rest ...
    #
}

codes = LookupDict(name='status_codes')  

for code, titles in _codes.items():
    for title in titles:
        setattr(codes, title, code)  
        if not title.startswith('\\'):
            setattr(codes, title.upper(), code)  

All of these options for an OK status will become keys in the lookup dictionary. Except for the happy person (\\o/) and the check mark (✔).

The deprecated values are on a separate line so that the future delete will be clean and obvious in version control.

The LookupDict allows dot-access of its elements like in the next line.

codes.ok == 200 and codes.okay == 200.

And also codes.OK == 200 and codes.OKAY == 200.

All of this work for the status codes was to make the lookup dictionary codes. Why? Instead of very typo-prone hardcoded integers all over the code, this is easy to read, with all of the code numbers localized in a single file. Because it starts as a dictionary keyed on the status codes, each status code integer only exists once. The possibility of typos is much, much lower than if this were just a bunch of global variables manually embedded into a namespace.

And converting the keys into attributes instead of using them as strings in a dictionary again reduces the risk of typographical errors. Here’s the example in sessions.py that’s so much easier to read with words than numbers:

#
#  ... from sessions.py ...
#      Truncated to only show relevant content.
#
from .status_codes import codes  


class SessionRedirectMixin(object):  
    def resolve_redirects(self, resp, req, stream=False, timeout=None,
                          verify=True, cert=None, proxies=None,
                          **adapter_kwargs):
        """Receives a Response. Returns a generator of Responses."""

        i = 0
        hist = [] # keep track of history

        while resp.is_redirect:  
            prepared_request = req.copy()

            if i > 0:
                # Update history and keep track of redirects.
                hist.append(resp)
                new_hist = list(hist)
                resp.history = new_hist

            try:
                resp.content  # Consume socket so it can be released
            except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
                resp.raw.read(decode_content=False)

            if i >= self.max_redirects:
                raise TooManyRedirects(
                        'Exceeded %s redirects.' % self.max_redirects
                )

            # Release the connection back into the pool.
            resp.close()

            #
            #  ... skipping content ...
            #

            # http://tools.ietf.org/html/rfc7231#section-6.4.4
            if (resp.status_code == codes.see_other and  
                    method != 'HEAD'):
                method = 'GET'

            # Do what the browsers do, despite standards...
            # First, turn 302s into GETs.
            if resp.status_code == codes.found and method != 'HEAD':  
                method = 'GET'

            # Second, if a POST is responded to with a 301, turn it into a GET.
            # This bizarre behavior is explained in Issue 1704.
            if resp.status_code == codes.moved and method == 'POST':  
                method = 'GET'

            #
            #  ... etc. ...
            #

Here’s where the status code lookup codes is imported.

We describe mixin classes later in “Mixins (also one honking great idea)”. This mixin provides redirect methods for the main Session class, which is defined in this same file but not shown in our excerpt.

We’re entering a loop that’s following the redirects for us to get at the content we want. The entire loop logic is deleted from this excerpt for brevity.

Status codes as text are so much more readable than unmemorizable integers: codes.see_other would otherwise be 303 here.

And codes.found would be 302, and codes.moved would be 301. So the code is self-documenting; we can tell meaning from the variable names; and we’ve avoided the possibility of littering the code with typographical errors by using dot-notation instead of a dictionary to look up strings (e.g., codes.found instead of codes["found"]).

Read Werkzeug’s documentation

Werkzeug’s documentation lists the main things it provides—an implementation of the WSGI 1.0 (PEP 333) specification, a URL routing system, the capability to parse and dump HTTP headers, objects that represent HTTP requests and HTTP responses, session and cookie support, file uploads, and other utilities and community add-ons. Plus, a full-featured debugger.

The tutorials are good, but we’re using the API documentation instead to see more of the library’s components. The next section takes from Werkzeug’s wrappers and routing documentation.

Use Werkzeug

Werkzeug provides utilities for WSGI applications, so to learn what Werkzeug provides, we can start with a WSGI application, and then use a few of Werkzeug’s utilities. This first application is a slightly changed version of what’s in PEP 333, and doesn’t use Werkzeug yet. The second one does the same thing as the first, but using Werkzeug:

def wsgi_app(environ, start_response):
    headers = [('Content-type', 'text/plain'), ('charset', 'utf-8')]
    start_response('200 OK', headers)
    yield 'Hello world.'

# This app does the same thing as the one above:
response_app = werkzeug.Response('Hello world!')

Werkzeug implements a werkzeug.Client class to stand in for a real web sever when doing one-off testing like this. The client response will have the type of the response_wrapper argument. In this code, we create clients and use them to call the WSGI applications we made earlier. First, the plain WSGI app (but with the response parsed into a werkzeug.Response):

>>> import werkzeug
>>> client = werkzeug.Client(wsgi_app, response_wrapper=werkzeug.Response)
>>> resp=client.get("?answer=42")
>>> type(resp)
<class 'werkzeug.wrappers.Response'>
>>> resp.status
'200 OK'
>>> resp.content_type
'text/plain'
>>> print(resp.data.decode())
Hello world.

Next, using the werkzeug.Response WSGI app:

>>> client = werkzeug.Client(response_app, response_wrapper=werkzeug.Response)
>>> resp=client.get("?answer=42")
>>> print(resp.data.decode())
Hello world!

The werkzeug.Request class provides the contents of the environment dictionary (the environ argument above to wsgi_app()) in a form that’s easier to use. It also provides a decorator to convert a function that takes a a werkzeug.Request and returns a werkzeug.Response into a WSGI app:

>>> @werkzeug.Request.application
... def wsgi_app_using_request(request):
...     msg = "A WSGI app with:\n   method: {}\n   path: {}\n   query: {}\n"
...     return werkzeug.Response(
...         msg.format(request.method, request.path, request.query_string))
...

which, when used, gives:

>>> client = werkzeug.Client(
...     wsgi_app_using_request, response_wrapper=werkzeug.Response)
>>> resp=client.get("?answer=42")
>>> print(resp.data.decode())
A WSGI app with:
   method: GET
   path: /
   query: b'answer=42'

So, we know how to use the werkzeug.Request and werkzeug.Response objects. The other thing that was featured in the documentation was the routing. Here’s an excerpt that uses it—callout numbers identify both the pattern and its match:

>>> import werkzeug
>>> from werkzeug.routing import Map, Rule
>>>
>>> url_map = Map([  
...     Rule('/', endpoint='index'),  
...     Rule('/<any("Robin","Galahad","Arthur"):person>', endpoint='ask'),  
...     Rule('/<other>', endpoint='other')  
... ])

>>> env = werkzeug.create_environ(path='/shouldnt/match')  
>>> urls = url_map.bind_to_environ(env)
>>> urls.match()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[...path...]/werkzeug/werkzeug/routing.py", line 1569, in match
    raise NotFound()
werkzeug.exceptions.NotFound: 404: Not Found

The werkzeug.Routing.Map provides the main routing functions. The rule matching is done in order; the first rule to match is the one selected.

When there are no angle-bracket terms in the rule’s placeholder string, it only matches on an exact match, and the second result from urls.match() is an empty dictionary:

>>> env = werkzeug.create_environ(path='/')
>>> urls = url_map.bind_to_environ(env)
>>> urls.match()
('index', {})

Otherwise, the second entry is a dictionary mapping the named terms in the rule to their value—for example, mapping 'person' to the value 'Galahad':

>>> env = werkzeug.create_environ(path='/Galahad?favorite+color')
>>> urls = url_map.bind_to_environ(env)
>>> urls.match()
('ask', {'person': 'Galahad'})

Note that 'Galahad' could have mached the route named 'other', but it did not—but 'Lancelot' did—because the first rule to match the pattern is chosen:

>>> env = werkzeug.create_environ(path='/Lancelot')
>>> urls = url_map.bind_to_environ(env)
>>> urls.match()
('other', {'other': 'Lancelot'})

And an exception is raised if there are no matches at all in the list of rules:

>>> env = werkzeug.test.create_environ(path='/shouldnt/match')
>>> urls = url_map.bind_to_environ(env)
>>> urls.match()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "[...path...]/werkzeug/werkzeug/routing.py", line 1569, in match
raise NotFound()
werkzeug.exceptions.NotFound: 404: Not Found

You’d use the map to route a request to an appropriate endpoint. The following code continues on from the predceding example to do this:

@werkzeug.Request.application
def send_to_endpoint(request):
    urls = url_map.bind_to_environ(request)
    try:
        endpoint, kwargs = urls.match()
        if endpoint == 'index':
            response = werkzeug.Response("You got the index.")
        elif endpoint == 'ask':
            questions = dict(
                Galahad='What is your favorite color?',
                Robin='What is the capital of Assyria?',
                Arthur='What is the air-speed velocity of an unladen swallow?')
            response = werkzeug.Response(questions[kwargs['person']])
        else:
            response = werkzeug.Response("Other: {other}".format(**kwargs))
    except (KeyboardInterrupt, SystemExit):
        raise
    except:
        response = werkzeug.Response(
            'You may not have gone where you intended to go,\n'
            'but I think you have ended up where you needed to be.',
            status=404
        )
    return response

To test it, use the werkzeug.Client again:

>>> client = werkzeug.Client(send_to_endpoint, response_wrapper=werkzeug.Response)
>>> print(client.get("/").data.decode())
You got the index.
>>>
>>> print(client.get("Arthur").data.decode())
What is the air-speed velocity of an unladen swallow?
>>>
>>> print(client.get("42").data.decode())
Other: 42
>>>
>>> print(client.get("time/lunchtime").data.decode())  # no match
You may not have gone where you intended to go,
but I think you have ended up where you needed to be.

Read Werkzeug’s code

When test coverage is good, you can learn what a library does by looking at the unit tests. The caveat is that with unit tests, you’re intentionally looking at the “trees” and not the “forest”—exploring obscure use cases intended to ensure the code doesn’t break, rather than looking for interconnections between modules. This should be OK for a toolkit like Werkzeug, which we expect to have modular, loosely coupled components.

Because we familiarized ourselves with how the routing and the request and response wrappers work, werkzeug/test_routing.py and werkzeug/test_wrappers.py are good choices to read for now.

When we first open werkzeug/test_routing.py, we can quickly look for interconnection between the modules by searching through the entire file for the imported objects. Here are all of the import statements:

import pytest  

import uuid  

from tests import strict_eq  

from werkzeug import routing as r  
from werkzeug.wrappers import Response  
from werkzeug.datastructures import ImmutableDict, MultiDict  
from werkzeug.test import create_environ  

Of course, pytest is used here for testing.

The uuid module is used in just one function, test_uuid_converter(), to confirm that the conversion from string to a uuid.UUID object (the Universal Unique Identifier string uniquely identifying objects on the Internet) works.

The strict_eq() function is used often, and defined in werkzeug/tests/__init__.py. It’s for testing, and is only necessary because in Python 2 there used to be implicit type conversion between Unicode and byte strings, but relying on this breaks things in Python 3.

The werkzeug.routing module is the one that’s being tested.

The Reponse object is used in just one function, test_dispatch(), to confirm that werkzeug.routing.MapAdapter.dispatch() passes the correct information through to the dispatched WSGI application.

These dictionary objects are used only once each, ImmutableDict to confirm that an immutable dictionary in werkzeug.routing.Map is indeed immutable, and MultiDict to provide multiple keyed values to the URL builder and confirm that it still builds the correct URL.

The create_environ() function is for testing—it creates a WSGI environment without having to use an actual HTTP request.

The point of doing the quick searching was to quickly see the interconnection between modules. What we found out was that werkzeug.routing imports some special data structures, and that’s all. The rest of the unit tests show the scope of the routing module. For example, non-ASCII characters can be used:

def test_environ_nonascii_pathinfo():
    environ = create_environ(u'/лошадь')
    m = r.Map([
        r.Rule(u'/', endpoint='index'),
        r.Rule(u'/лошадь', endpoint='horse')
    ])
    a = m.bind_to_environ(environ)
    strict_eq(a.match(u'/'), ('index', {}))
    strict_eq(a.match(u'/лошадь'), ('horse', {}))
    pytest.raises(r.NotFound, a.match, u'/барсук')

There are tests to build and parse URLs, and even utilities to find the closest available match, when there wasn’t an actual match. You can even do all kinds of crazy custom processing when handling the type conversions/parsing from the path and the URL string:

def test_converter_with_tuples():
    '''
    Regression test for https://github.com/pallets/werkzeug/issues/709
    '''
    class TwoValueConverter(r.BaseConverter):

        def __init__(self, *args, **kwargs):
            super(TwoValueConverter, self).__init__(*args, **kwargs)
            self.regex = r'(\w\w+)/(\w\w+)'

        def to_python(self, two_values):
            one, two = two_values.split('/')
            return one, two

        def to_url(self, values):
            return "%s/%s" % (values[0], values[1])

    map = r.Map([
        r.Rule('/<two:foo>/', endpoint='handler')
    ], converters={'two': TwoValueConverter})
    a = map.bind('example.org', '/')
    route, kwargs = a.match('/qwert/yuiop/')
    assert kwargs['foo'] == ('qwert', 'yuiop')

Similarly, werkzeug/test_wrappers.py does not import much. Reading through the tests gives an example of the scope of available functionality for the Request object—cookies, encoding, authentication, security, cache timeouts, and even multilanguage encoding:

def test_modified_url_encoding():
    class ModifiedRequest(wrappers.Request):
        url_charset = 'euc-kr'

    req = ModifiedRequest.from_values(u'/?foo=정상처리'.encode('euc-kr'))
    strict_eq(req.args['foo'], u'정상처리')

In general, reading the tests provides a way to see the details of what the library provides. Once satisfied that we have an idea of what Werkzeug is, we can move on.

Elegant way to guess type (if the implementation is easy to explain, it may be a good idea)

If you’re like most of us, you’ve had to parse text files and convert content to various types. This solution is particularly Pythonic, so we wanted to include it:

_PYTHON_CONSTANTS = {
    'None':     None,
    'True':     True,
    'False':    False
}


def _pythonize(value):
    if value in _PYTHON_CONSTANTS: 
        return _PYTHON_CONSTANTS[value]
    for convert in int, float:  
        try:  
            return convert(value)
        except ValueError:
            pass
    if value[:1] == value[-1:] and value[0] in '"\'':  
        value = value[1:-1]
    return text_type(value)