Chapter 7. User Interaction

Libraries in this chapter help developers write code that interacts with end users. We describe the Jupyter project—it’s unique—then cover the more typical command-line and graphical user interfaces (GUIs), and finish with a discussion of tools for web applications.

Jupyter Notebooks

Jupyter is a web application that allows you to display and execute Python interactively. It’s listed here because it’s a user-to-user interface.

Users view Jupyter’s client interface—written in CSS, HTML, and JavaScript—in a web browser on the client machine. The client communicates with a a kernel written in Python (or a number of other languages) that executes blocks of code and responds with the result. Content is stored on the server machine in “notebook” (*.nb) format—text-only JSON divided into a series of “cells” that can contain HTML, Markdown (a human-readable markup language like what’s used on wiki pages), raw notes, or executable code. The server can be local (on the user’s own laptop), or remote like the sample notebooks at https://try.jupyter.org/.

The Jupyter server requires at least Python 3.3 and has been made compatible with Python 2.7. It comes bundled with the most recent versions of commercial Python redistributions (described in “Commercial Python Redistributions”) such as Canopy and Anaconda, so no further installation is required with these tools, provided you can compile and build C code on your system, as we discussed in Chapter 2. After the proper setup, you can install Jupyter from the command line with pip:

$ pip install jupyter

A recent study using Jupyter in the classroom found it to be an effective and popular way to create interactive lectures for students new to coding.

Command-Line Applications

Command-line (also called console) applications are computer programs designed to be used from a text interface, such as a shell. They can be simple commands—such as pep8 or virtualenv—or interactive, like the python interpreter or ipython. Some have subcommands, such as pip install, pip uninstall, or pip freeze—all of which have their own options in addition to pip’s general options. All of them are likely started in a main() function; our BDFL has shared his opinion about what makes a good one.

We’ll use an example pip call to name the components that can be present when invoking a command-line application:

              
$ pip install --user -r requirements.txt

: The command is the name of the executable being invoked.
: Arguments come after the command and do not begin with a dash. They may also be referred to as parameters or subcommands.
: Options begin with either one dash (for single characters, such as -h) or else with two dashes (for words, such as --help). They may also be referred to as flags or switches.

The libraries in Table 7-1 all provide different options for parsing arguments or provide other useful tools for command-line applications.

Table 7-1. Command-line tools
Library	License	Reasons to use
argparse	PSF license	In the Standard Library. Gives you standard argument and option parsing.
docopt	MIT license	Gives you control over the layout of your help message. Parses the command line according to the utility conventions defined in the POSIX standard.
plac	BSD 3-clause license	Autogenerates the help message from an existing function signature. Parses the command-line arguments behind the scenes, passing them directly to your function.
click	BSD 3-clause license	Provides decorators to make the help message and parser (a lot like plac) Lets you compose multiple subcommands together Interfaces with other Flask plug-ins (Click is independent from Flask, but originally written to help users compose together command-line tools from different Flask plug-ins without breaking things, so it is already used in the Flask ecosystem)
clint	Internet Software Consortium (ISC) license	Provides formatting features like colors, indentation, and columnar displays for your text output Also provides type checking (e.g., check against a regex, for an integer, or a path) in your interactive input Gives you direct access to the argument list, with some simple filtering and grouping tools
cliff	Apache 2.0 license	Provides a structured framework for a large Python project with multiple subcommands Builds an interactive environment to use the subcommands, with no additional coding

In general, you should try and use tools provided in Python’s Standard Library first, and only add other libraries to your project when they offer something you want that the Standard Library doesn’t have.

The following sections provide more details about each of the command-line tools listed in Table 7-1.

argparse

The module argparse (which replaces the now deprecated optparse) exists in Python’s Standard Library to help with parsing command-line options. The command-line interface provided by the HowDoI project uses argparse—you can refer to it when building your own command-line interface.

Here is the code to generate the parser:

import argparse
#
# ... skip a lot of code ...
#

def get_parser():
    parser = argparse.ArgumentParser(description='...truncated for brevity...')
    parser.add_argument('query', metavar='QUERY', type=str, nargs='*',
                        help='the question to answer')
    parser.add_argument('-p','--pos',
                        help='select answer in specified position (default: 1)',
                        default=1, type=int)
    parser.add_argument('-a','--all', help='display the full text of the answer',
                        action='store_true')
    parser.add_argument('-l','--link', help='display only the answer link',
                        action='store_true')
    parser.add_argument('-c', '--color', help='enable colorized output',
                        action='store_true')
    parser.add_argument('-n','--num-answers', help='number of answers to return',
                        default=1, type=int)
    parser.add_argument('-C','--clear-cache', help='clear the cache',
                        action='store_true')
    parser.add_argument('-v','--version',
                        help='displays the current version of howdoi',
                        action='store_true')
    return parser

The parser will parse the command line and create a dictionary that maps each argument to a value. The action='store_true' indicates the option is intended as a flag that, if present on the command line, will be stored as True in the parser’s dictionary.

docopt

docopt’s core philosophy is that documentation should be beautiful and understandable. It provides one main command, docopt.docopt(), plus a few convenience functions and classes for power users. The function docopt.docopt() takes developer-written POSIX-style usage instructions, uses them to interpret the user’s command-line arguments, and returns a dictionary with all of the arguments and options parsed from the command line. It also appropriately handles the --help and --version options.

In the following example, the value of the variable arguments is a dictionary with keys name, --capitalize, and --num_repetitions:

#!/usr/bin env python3
"""Says hello to you.

  Usage:
    hello <name>... [options]
    hello -h | --help | --version

    -c, --capitalize  whether to capitalize the name
    -n REPS, --num_repetitions=REPS number of repetitions [default: 1]
"""

__version__ = "1.0.0"  # Needed for --version

def hello(name, repetitions=1):
    for rep in range(repetitions):
        print('Hello {}'.format(name))

if __name__ == "__main__":
    from docopt import docopt
    arguments = docopt(__doc__, version=__version__)
    name = ' '.join(arguments['<name>'])
    repetitions = arguments['--num_repetitions']
    if arguments['--capitalize']:
        name = name.upper()
    hello(name, repetitions=repetitions)

Since version 0.6.0, docopt can be used to create complex programs with subcommands that behave like the git command or Subversion’s svn command. It can even be used when a subcommand is written in different languages. There is a complete example application that mocks up a reimplementation of the git command that explains how.

Plac

Plac’s philosophy is that all of the information necessary to parse a command invocation is in the target function’s signature. It is a lightweight (around 200 lines) wrapper around the Python Standard Library’s argparse and provides one main command, plac.plac(), which infers the argument parser from the function signature, parses the command line, and then invokes the function.

The library was supposed to be named Command-Line Argument Parser (clap) but that ended up becoming taken a few days after its author chose the name, so it’s Plac—clap in reverse. The usage statements aren’t informative, but look how few lines this example takes:

# hello.py

def hello(name, capitalize=False, repetitions=1):
  """Says hello to you."""
    if capitalize:
        name = name.upper()
    for rep in range(repetitions):
        print('Hello {}'.format(name))


if __name__ == "__main__":
    import plac
    plac.call(hello)

The usage statement looks like this:

$ python hello.py --help
usage: hello.py [-h] name [capitalize] [repetitions]

Says hello to you.

positional arguments:
  name
  capitalize   [False]
  repetitions  [1]

optional arguments:
  -h, --help   show this help message and exit

If you want to typecast (convert to the correct type) any of the arguments before you pass them to the function, use the annotations decorator:

import plac

@plac.annotations(
    name = plac.Annotation("the name to greet", type=str),
    capitalize = plac.Annotation("use allcaps", kind="flag", type=bool),
    repetitions = plac.Annotation("total repetitions", kind="option", type=int)
def hello(name, capitalize=False, repetitions=1):
  """Says hello to you."""
    if capitalize:
        name = name.upper()
    for rep in range(repetitions):
        print('Hello {}'.format(name))

Also, plac.Interpreter provides a lightweight way to make a very quick interactive command-line application. Examples are with plac’s interactive mode documentation.

Click

The main purpose of Click (the “Command Line-Interface Creation Kit”) is to help developers create composable command-line interfaces with as little code as possible. The Click documentation clarifies its relation to docopt:

The aim of Click is to make composable systems, whereas the aim of docopt is to build the most beautiful and hand-crafted command-line interfaces. These two goals conflict with one another in subtle ways. Click actively prevents people from implementing certain patterns in order to achieve unified command-line interfaces. You have very little input on reformatting your help pages for instance.

Its defaults will satisfy most developers’ needs, but it’s highly configurable for power users. Like Plac, it uses decorators to tie the parser definitions to the functions that will use them, keeping command-line argument management out of the functions themselves.

The hello.py application with Click looks like this:

import click

@click.command()
@click.argument('name', type=str)
@click.option('--capitalize', is_flag=True)
@click.option('--repetitions', default=1,
              help="Times to repeat the greeting.")
def hello(name, capitalize, repetitions):
    """Say hello, with capitalization and a name."""
    if capitalize:
        name = name.upper()
    for rep in range(repetitions):
        print('Hello {}'.format(name))

if __name__ == '__main__':
    hello()

Click parses the description from the command’s docstring and creates its help message using a custom parser derived from the Python Standard Library’s now deprecated optparse, which was more compliant with the POSIX standard than argparse.¹ The help message looks like this:

$ python hello.py --help
Usage: hello.py [OPTIONS] NAME

  Say hello, with capitalization and a name.

Options:
  --capitalize
  --repetitions INTEGER  Times to repeat the greeting.
  --help                 Show this message and exit.

But the real value of Click is its modular composability—you can add an outer grouping function, and then any other click-decorated functions in your project will become subcommands to that top-level command:

import click

@click.group()  
@click.option('--verbose', is_flag=True)
@click.pass_context  
def cli(ctx, verbose):
    ctx.obj = dict(verbose = verbose)  
    if ctx.obj['verbose']:
        click.echo("Now I am verbose.")

#  The 'hello' function is the same as before...

if __name__ == '__main__':
    cli()

: The group() decorator makes a top-level command that runs first, before executing the invoked subcommand.
: The pass_context decorator is how to (optionally) pass objects from the grouped command to a subcommand, by making the first argument a click.core.Context object.
: That object has a special attribute ctx.obj that can be passed to subcommands that use an @click.pass_context decorator.
: Now instead of calling the function hello(), call the function that was decorated with @click.group()—in our case, cli().

Clint

The Clint library is, like its name says, a collection of “Command-Line INterface Tools.” It supports features such as CLI colors and indentation, a simple and powerful column printer, iterator-based progress bars, and implicit argument handling. This example shows the colorization and indentation tools:

"""Usage string."""
from clint.arguments import Args
from clint.textui import colored, columns, indent, puts

def hello(name, capitalize, repetitions):
    if capitalize:
        name = name.upper()
    with indent(5, quote=colored.magenta(' ~*~', bold=True)):  
        for i in range(repetitions):
            greeting = 'Hello {}'.format(colored.green(name))  
            puts(greeting)  


if __name__ == '__main__':
    args = Args()  
    # First check and display the help message
    if len(args.not_flags) == 0 or args.any_contain('-h'):
        puts(colored.red(__doc__))
        import sys
        sys.exit(0)

    name = " ".join(args.grouped['_'].all)  
    capitalize = args.any_contain('-c')
    repetitions = int(args.value_after('--reps') or 1)
    hello(name, capitalize=capitalize, repetitions=repetitions)

: Clint’s indent is a context manager—intuitive to use in the with statement. The quote option prefixes each line with a bold magenta ~*~.
: The colored module has eight color functions and an option to turn coloring off.
: The puts() function is like print() but also handles the indentation and quoting.
: Args provides some simple filtering tools for the argument list. It returns another Args object, to make it possible to chain the filters.
: Here’s how to use the args made by Args().

cliff

cliff (the Command-Line Interface Formulation Framework) is a framework for building command-line programs. The framework is meant to be used to create multilevel commands that behave like svn (Subversion) or git, and interactive programs like a Cassandra shell or a SQL shell.

cliff’s functionality is grouped into abstract base classes. You have to implement cliff.command.Command once for every subcommand, and then cliff.commandmanager.CommandManager will delegate to the correct command. Here is a minimal hello.py:

import sys

from argparse import ArgumentParser  
from pkg_resources import get_distribution

from cliff.app import App
from cliff.command import Command
from cliff.commandmanager import CommandManager

__version__ = get_distribution('HelloCliff').version  


class Hello(Command):
    """Say hello to someone."""

    def get_parser(self, prog_name):  
        parser = ArgumentParser(description="Hello command", prog=prog_name)
        parser.add_argument('--num', type=int, default=1, help='repetitions')
        parser.add_argument('--capitalize', action='store_true')
        parser.add_argument('name', help='person\'s name')
        return parser

    def take_action(self, parsed_args):  
        if parsed_args.capitalize:
            name = parsed_args.name.upper()
        else:
            name = parsed_args.name
        for i in range(parsed_args.num):
            self.app.stdout.write("Hello from cliff, {}.\n".format(name))


class MyApp(cliff.app.App):  
    def __init__(self):
        super(MyApp, self).__init__(
            description='Minimal app in Cliff',
            version=__version__,
            command_manager=CommandManager('named_in_setup_py'),  
        )


def main(argv=sys.argv[1:]):
    myapp = MyApp()
    return myapp.run(argv)

: cliff uses argparse.ArgumentParser directly for its command-line interface.
: Get the version from setup.py (the last time pip install was run).
: get_parser() is required by the abstract base class—it should return an argparse.ArgumentParser.
: take_action() is required by the abstract base class—it runs when the Hello command is invoked.
: The main application subclasses cliff.app.App, and is responsible for setting up logging, I/O streams, and anything else that globally applies to all subcommands.
: The CommandManager manages all of the Command classes. It uses the entry_points content from setup.py to find the command names.

GUI Applications

In this section, we first list widget libraries—toolkits and frameworks that provide buttons, scroll bars, progress bars, and other prebuilt components. We also quickly list game libraries at the end.

Widget Libraries

In the context of GUI development, widgets are buttons, sliders, scroll bars, and other commonly used UI control and display elements. With them, you don’t have to deal with low-level coding like identifying which button (if any) was underneath the mouse when it was clicked, or even lower-level tasks, like the different windowing APIs used by each operating system.

When you’re new to GUI development, the first thing you want is something that’s easy to use—so that you can learn how to make GUIs. For that we recommend Tkinter, already in Python’s Standard Library. After that, you probably care about the structure and function of the toolkit underlying the library, so we group libraries by toolkits, listing the more popular ones first.

Table 7-2. GUI widget libraries
Underlying library (language)	Python library	License	Reasons to use
Tk (Tcl)	tkinter	Python Software Foundation license	All of the dependencies are already bundled with Python. It provides standard UI widgets like buttons, scroll bars, text boxes, and a drawing canvas.
SDL2 (C)	Kivy	MIT, or LGPL3 (before 1.7.2)	It can be used to make an Android app. It has multitouch features. It is optimized to C when possible, and uses the GPU.
Qt (C++)	PyQt	GNU General Public License (GPL) or Commercial	It provides a consistent look and feel across platforms. Many applications and libraries already rely on Qt, (e.g., the Eric IDE, Spyder, and/or Matplotlib), so it may already be installed. Qt5 (which can’t be used alongside Qt4) provides utilities to make an Android app.
Qt (C++)	PySide	GNU Lesser General Public License (LGPL)	It’s a drop-in replacement for PyQt with a more permissive license.
GTK (C) (GIMP Toolkit)	PyGObject (PyGi)	GNU Lesser General Public License (LGPL)	It provides Python bindings for GTK+ 3. It should be familiar to those who already develop for the GNOME desktop system.
GTK (C)	PyGTK	GNU Lesser General Public license (LGPL)	Only use this if your project already uses PyGTK; you should be porting old PyGTK code to PyGObject.
wxWindows (C++)	wxPython	wxWindows license (a modified LGPL)	Provides a native look and feel by directly exposing the various windowing libraries for each platform. This means parts of your code will be different for each platform.
Objective C	PyObjC	MIT license	Provides an interface to (and from) Objective C. Will give a native feel for your OS X project. It cannot be used on other platforms.

The following sections provide more detail on the different GUI options for Python, grouped by each one’s underlying toolkit.

Tk

The module Tkinter in Python’s Standard Library is a thin object-oriented layer on top of Tk, the widget library written in the language Tcl. (Both are usually written together as Tcl/Tk.²) Because it’s in the Standard Library, it’s the most convenient and compatible GUI toolkit in this list. Both Tk and Tkinter are available on most Unix platforms, as well as on Windows and OS X.

There’s a good multilanguage Tk tutorial with Python examples at TkDocs, and more information available on the Python wiki.

Also, if you have a standard distribution of Python, you shoud have IDLE, a GUI interactive coding environment written entirely in Python that is part of Python’s Standard Library—you can launch it from the command line by typing idle, or view all of its source code. You can find the path where it’s installed by typing this in a shell:

$ python -c"import idlelib; print(idlelib.__path__[0])"

There are many files in the directory; the main IDLE application is launched from the module PyShell.py.

Likewise, for an example use of the drawing interface, tkinter.Canvas, see the code for the turtle module. You can find it by typing this in a shell:

$ python -c"import turtle; print(turtle.__file__)"

Kivy

Kivy is a Python library for development of multitouch enabled media rich applications. Kivy is actively being developed by a community, has a permissive BSD-like license, and operates on all major platforms (Linux, OS X, Windows, and Android). It’s written in Python and does not use any underlying windowing toolkit—it interfaces directly with SDL2 (Simple DirectMedia Layer), a C library that provides low-level access to user input devices,³ and audio, plus access to 3D rendering using OpenGL (or Direct3D for Windows). It has some widgets (they’re in the module kivy.uix), but not nearly as many as the most popular alternatives, Qt and GTK. If you’re developing a traditional desktop business application, Qt or GTK are probably better.

To install it, go to the Kivy downloads page, find your operating system, download the correct ZIP file for your version of Python, and follow the instructions linked for your operating system. The code comes with a directory of over a dozen examples that demonstrate different parts of the API.

Qt

Qt (pronounced “cute”) is a cross-platform application framework that is widely used for developing software with a GUI, but can also be used for non-GUI applications. Plus, there is a Qt5 version for Android. If you already have Qt installed (because you’re using Spyder, Eric IDE, Matplotlib, or other tools using Qt), you can check your version of Qt from the command line by using:

$ qmake -v

Qt is released under the LGPL license, allowing you to distribute binaries that work with Qt so long as you don’t change Qt. A commercial license will get you add-on tools like data visualization and in-application purchasing. Qt is a framework—it provides some prebuilt scaffolding for different types of applications. Both Python interfaces to Qt, PyQt and PySide, don’t do well with documentation, so your best option is Qt’s actual C++ documentation. Here are brief descriptions of each:

PyQt

Riverbank Computing’s PyQt is more up to date than PySide (which doesn’t yet have a Qt5 version). To install, follow the documentation for PyQt4 installation or PyQt5 installation. PyQt4 only works with Qt4, and PyQt5 only works with Qt5. (We suggest Docker, a user-space isolation tool discussed in “Docker”, if you really have to develop using both—just to not have to deal with changing your library paths.)

Riverbank Computing also publishes pyqtdeploy, a PyQt5-only GUI tool that generates platform-specific C++ code you can use to build binaries for distribution. For more information, check out these PyQt4 tutorials and PyQt5 examples.

PySide

PySide was released while Nokia owned Qt, because they couldn’t get Riverside Computing, the makers of PyQt, to change PyQt’s license from GPL to LGPL. It is intended to be a drop-in replacement for PyQt, but tends to lag PyQt in development. This wiki page describes the differences between PySide and PyQt.

To install PySide, follow the instructions in the Qt documentation; there is also a page to help you write your first PySide application.

GTK+

The GTK+ toolkit (which stands for the GIMP⁴ Toolkit) provides an API to the backbone of the GNOME desktop environment. Programmers might choose GTK+ over Qt either because they prefer C and are more comfortable looking in GTK+’s source code when they have to, or because they have programmed GNOME applications before and are comfortable with the API. The two libraries with Python bindings to GTK+ are:

pyGTK

PyGTK provides Python bindings for GTK+ but only currently supports the GTK+ 2.x API (not GTK+ 3+). It is no longer being developed, and its team recommends PyGTK not be used for new projects and that existing applications be ported from PyGTK to PyGObject.

PyGObject (aka PyGI)

PyGObject provides Python bindings that give access to the entire GNOME software platform. It is also known as PyGI because it makes use of, and provides a Python API for, GObject Introspection, which is an API bridge between other languages and GNOME’s core C libraries, GLib, provided they follow the convention used to define a GObject. It is fully compatible with GTK+ 3. The Python GTK+ 3 Tutorial is a good place to start.

To install, get the binaries from the PyGObject download site, or on OS X, install it with homebrew using brew install pygobject.

wxWidgets

The design philosophy behind wxWidgets is that the best way for an app to get a native look and feel is to use the API native to each operating system. Both Qt and GTK+ now also can use other windowing libraries than X11 under the hood, but Qt abstracts them, and GTK makes them look like you’re programming GNOME. The benefit of using wXWidgets is that you are directly interfacing with each platform, and the license is much more permissive. The problem, though, is that you now have to handle each platform slightly differently.

The Python extension module that wraps wxWidgets for Python users is called wxPython. It at one point was the most popular windowing library in Python, possibly because of its philosophy of using native interface tools, but now the workarounds in Qt and GTK+ seem to have become good enough. Still, to install it, go to http://www.wxpython.org/download.php#stable and download the appropriate package for your OS, and get started with their wxPython tutorial.

Objective-C

Objective-C is the proprietary language used by Apple for the OS X and iOS operating systems, and providing access to the Cocoa framework for application development on OS X. Unlike the other options, Objective-C is not cross-platform; it is only for Apple products.

PyObjC is a bidirectional bridge between the OS X Objective-C languages and Python, meaning it not only allows Python access to the Cocoa framework for application development on OS X, but allows Objective-C programmers to access Python.⁵

Note

The Cocoa framework is only available on OS X, so don’t choose Objective-C (via PyObjC) if you’re writing a cross-platform application.

You will need to have Xcode installed, as described in “Installing Python on Mac OS X”, because PyObjC needs a compiler. Also, PyObjC works only with the standard CPython distribution—not with other distributions like PyPy or Jython—and we recommend using the Python executable provided by OS X, because that Python was modified by Apple and configured specifically to work with OS X.

To make your virtual environment using your system’s Python interpreter, use the whole path when invoking it. If you do not want to install as the super user, install using the --user switch, which will save the library under $HOME/Library/Python/2.7/lib/python/site-packages/:

$ /usr/bin/python -m pip install --upgrade --user virtualenv

Activate the environment, enter it, and install PyObjC:

$ /usr/bin/python -m virtualenv venv
$ source venv/bin/activate
(venv)$ pip install pyobjc

This takes a while. PyObjC comes bundled with py2app (discussed in “py2app”), which is the OS X–specific tool to create distributable standalone application binaries. There are sample applications on the PyObjC examples page.

Game Development

Kivy has become really popular really quickly, but has a much larger footprint than the libraries listed in this section. It was categorized as a toolkit because it provides widgets and buttons but is frequently used to build games. The Pygame community hosts a Python game developer website that welcomes all game developers, whether or not they’re using Pygame. The most popular game development libraries are:

cocos2d: cocos2d is released under the BSD license. It builds on top of pyglet, providing a framework to structure your game as a set of scenes connected via custom workflows, all managed by a director. Use it if you like the scene-director-workflow style they use as described in the documentation, or want pyglet for drawing, plus SDL2 for joystick and audio. You can install cocos2D using pip. For SDL2, check your package manager first, then download from the SDL2 site. The best way to get started is with their example cocos2d applications.
pyglet: pyglet is released under the BSD license. It’s a set of lightweight wrappers around OpenGL, plus tools for presenting and moving sprites around a window. Just install it—pip should be all you need because most every computer has OpenGL—and run some of the example applications, including a complete Asteroids clone in fewer than 800 lines of code.
Pygame: Pygame is released under the Zlib license, plus the GNU LGPLv2.1 for SDL2. It has a large, active community, with tons of Pygame tutorials, but it has been using SDL1, a prior version of the library. It isn’t available on PyPI, so check your package manager first, and if they don’t have it, download Pygame.
Pygame-SDL2: Pygame-SDL2 was recently announced as an effort to reimplement Pygame with an SDL2 backend. It is released under the same licenses as Pygame.
PySDL2: PySDL2 runs on CPython, IronPython, and PyPy, and is a thin Python interface to the SDL2 library. If you want the lightest-weight interface to SDL2 in Python, this is your library. For more information, see the PySDL2 tutorial.

Web Applications

As a powerful scripting language adapted to both fast prototyping and bigger projects, Python is widely used in web application development (YouTube, Pinterest, Dropbox, and The Onion all use it).

Two of the libraries we profiled in Chapter 5—“Werkzeug”, and “Flask” were related to building web applications. With them, we briefly described the Web Server Gateway Interface (WSGI), a Python standard defined in PEP 3333 that specifies how web servers and Python web applications communicate. This section will look at Python web frameworks, their templating systems, the servers they interface with, and the platforms they run on.

Web Frameworks/Microframeworks

Broadly speaking, a web framework consists of a set of libraries and a main handler within which you can build custom code to implement a web application (i.e., an interactive website providing a client interface to code running on a server). Most web frameworks include patterns and utilities to accomplish at least the following:

URL routing: Match an incoming HTTP request to a particular Python function (or callable).
Handling Request and Response objects: Encapsulate the information received from or sent to a user’s browser.
Templating: Inject Python variables into HTML templates or other output, allowing programmers to separate an application’s logic (in Python) from the layout (in the template).
Development web service for debugging: Run a miniature HTTP server on development machines to enable rapid development; often automatically reloading server-side code when files are updated.

You shouldn’t have to code around your framework. It should already provide the things you need—tested and used by thousands of other developers—so if you still don’t find what you need, keep exploring the many other available frameworks (e.g., Bottle, Web2Py, CherryPy). A technical reviewer also noted we should mention Falcon, which is a framework specifically for building RESTful APIs (i.e., not for serving HTML).

All of the libraries in Table 7-3 can be installed using pip:

$ pip install Django
$ pip install Flask
$ pip install tornado
$ pip install pyramid

Table 7-3. Web frameworks
Python library	License	Reasons to use
Django	BSD license	It provides structure—a mostly prebuilt site where you design the layout and the underlying data and logic. It also autogenerates an administrative web interface, where nonprogrammers can add or delete data (like news articles). It is integrated with Django’s object-relational mapping (ORM) tool.
Flask	BSD license	It allows you total control over what is in your stack. It provides elegant decorators that add URL routing to any function you choose. It frees you from the structure provided by either Django or Pyramid.
Tornado	Apache 2.0 license	It provides excellent asynchronous event handling—Tornado uses its own HTTP server. It also gives you a way to handle many WebSockets (full duplex, persistent communication over TCP^a) or other long-duration connection out of the box.
Pyramid	Modified BSD license	It provides some prebuilt structure—called a scaffolding—but less than Django, allowing you to use any database interface or templating library you wish (if any). It is based on the popular the Zope framework, and on Pylons, both predecessors of Pyramid.
^a Transmission Control Protocol (TCP) is a standard protocol that defines a way for two computers to establish a connection and communicate with each other.

The following sections provide a more detailed look at the web frameworks in Table 7-3.

Django

Django is a “batteries included” web application framework and is an excellent choice for creating content-oriented websites. By providing many utilities and patterns out of the box, Django aims to make it possible to build complex, database-backed web applications quickly while encouraging best practices in code that uses it.

Django has a large and active community, and many reusable modules that can be directly incorporated into a new project or customized to fit your needs.

There are annual Django conferences in the United States and in Europe—the majority of new Python web applications today are built with Django.

Flask

Flask is a microframework for Python, and is an excellent choice for building smaller applications, APIs, and web services. Rather than aiming to provide everything you could possibly need, Flask implements the most commonly used core components of a web application framework, like URL routing, HTTP request and response objects, and templates. Building an app with Flask is a lot like writing standard Python modules, except some functions have routes attached to them (via a decorator, like in the code sample shown here). It’s really beautiful:

@app.route('/deep-thought')
def answer_the_question():
    return 'The answer is 42.'

If you use Flask, it is up to you to choose other components for your application, if any. For example, database access or form generation/validation are not built into Flask. This is great, because many web applications don’t need those features. If yours do, there are many available extensions, such as SQLAlchemy for a database, or pyMongo for MongoDB and WTForms for forms.

Flask is the default choice for any Python web application that isn’t a good fit for Django’s prebuilt scaffolding. Try these example Flask applications for a good introduction. If you want to run multiple applications (the default for Django), use application dispatching. If instead you want to duplicate behaviors for sets of subpages within an app, try Flask’s Blueprints.

Tornado

Tornado is an asynchronous (event-driven and nonblocking, like Node.js) web framework for Python that has its own event loop.⁶ This allows it to natively support the WebSockets communication protocol, for example. Unlike the other frameworks in this section, Tornado is not a WSGI application. It can be made to run either as a WSGI application or as a WSGI server through their module tornado.wsgi, but even the authors ask “what’s the point?”⁷ seeing as WSGI is a synchronous interface and the purpose of Tornado is to provide an asynchronous framework.

Tornado is a more difficult and less frequently used web framework than Django or Flask—use it only if you know the performance gain of using an asynchronous framework is worth the additional time you will spend programming. If you do, a good place to start is with their demo applications. Well-written Tornado applications are known to have excellent performance.

Pyramid

Pyramid is a lot like Django, except with a heavier focus on modularity. It comes with a smaller number of built-in libraries (fewer “batteries” included), and encourages users to extend its base functionality through shareable templates they call scaffolds. You register the scaffold, and then invoke it when creating a new project using their command pcreate to build your project’s scaffolding—like Django’s django-admin startproject project-name command, but with options for different structures, different database backends, and URL routing options.

Pyramid does not have a large user base, unlike Django and Flask, but those who use it are passionate about it. It’s a very capable framework, but not as popular a choice for new Python web applications right now.

Here are a few Pyramid tutorials to start with. Or, for a page you can use to sell Pyramid to your boss, try this portal to all things Pyramid.

Web Template Engines

Most WSGI applications exist to respond to HTTP requests and serve content in HTML or other markup languages. Template engines are in charge of rendering this content: they manage a suite of templating files, with a system of hierarchy and inclusion to avoid unnecessary repetition, and fill the static content of the templates with the dynamic content generated by the application. This helps us adhere to the concept of separation of concerns⁸—keeping only the application logic in the code, and delegating presentation to the templates.

Template files are sometimes written by designers or frontend developers, and complexity in the pages can make coordination difficult. Here are some good practices both for the application passing dynamic content to the template engine, and for the templates themselves:

Never is often better than right now: Template files should be passed only the dynamic content that is needed for rendering the template. Avoid the temptation to pass additional content “just in case”: it is easier to add some missing variable when needed than to remove a likely unused variable later.
Try to keep logic out of the template: Many template engines allow for complex statements or assignments in the template itself, and many allow some Python code to be evaluated in the templates. This convenience can lead to uncontrolled growth in complexity and often makes it harder to find bugs. We’re not 100% against it—practicality beats purity—just contain yourself.
Keep the JavaScript separate from the HTML: It is often necessary to mix JavaScript templates with HTML templates. Preserve your sanity, and isolate the parts where the HTML template passes variables to the JavaScript code.

All of the template engines listed in Table 7-4 are second-generation, with good rendering speed⁹ and features added thanks to experience with older template languages.

Table 7-4. Template engines
Python library	License	Reasons to use
Jinja2	BSD license	It’s Flask’s default and bundled with Django. It is based on the Django Template Language, with just a little more logic allowed in the templates. Jinja2 is the default engine for Sphinx, Ansible, and Salt—if you’ve used them, you know Jinja2.
Chameleon	Modified BSD license	The templates are themselves valid XML/HTML. It is similar to the Template Attribute Language (TAL) and its derivatives.
Mako	MIT license	It’s Pyramid’s default. It’s designed for speed—for when your template rendering is actually a bottleneck. It allows you to put a lot of code in your templates—Mako is kind of like a Python version of PHP.

The following sections describe the libraries in Table 7-4 in more detail.

Jinja2

Jinja2 is our recommended templating library for new Python web applications. It is Flask’s default engine, the default engine for the Python documentation generator Sphinx and can be used in Django, Pyramid, and Tornado. It uses a text-based template language and can thus be used to generate any type of markup, not just HTML. It allows customization of filters, tags, tests, and globals. Inspired by the Django template language, it adds features like a little bit of in-template logic, which saves tons of code.

Here are some important Jinja2 tags:

{# This is a comment -- because of the hash + curly braces. #}

{# This is how to insert a variable: #}
{{title}}

{# This defines a named block, replaceable by a child template. #}
{% block head %}
<h1>This is the default heading.</h1>
{% endblock %}

{# This is how to do iteration: #}
{% for item in list %}
<li>{{ item }}</li>
{% endfor %}

Here’s an example of a website in combination with the Tornado web server, described in “Tornado”:

# import Jinja2
from jinja2 import Environment, FileSystemLoader

# import Tornado
import tornado.ioloop
import tornado.web

# Load template file templates/site.html
TEMPLATE_FILE = "site.html"
templateLoader = FileSystemLoader( searchpath="templates/" )
templateEnv = Environment( loader=templateLoader )
template = templateEnv.get_template(TEMPLATE_FILE)

# List for famous movie rendering
movie_list = [
    [1,"The Hitchhiker's Guide to the Galaxy"],
    [2,"Back to the Future"],
    [3,"The Matrix"]
]

# template.render() returns a string containing the rendered HTML
html_output = template.render(list=movie_list, title="My favorite movies")

# Handler for main page
class MainHandler(tornado.web.RequestHandler):
    def get(self):
        # Returns rendered template string to the browser request
        self.write(html_output)

# Assign handler to the server root  (127.0.0.1:PORT/)
application = tornado.web.Application([
    (r"/", MainHandler),
])
PORT=8884
if __name__ == "__main__":
    # Set up the server
    application.listen(PORT)
    tornado.ioloop.IOLoop.instance().start()

A base.html file can be used as base for all site pages. In this example, they would be implemented in the (currently empty) content block:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <link rel="stylesheet" href="style.css" />
    <title>{{title}} - My Web Page</title>
</head>
<body>
<div id="content">
    {# In the next line, the content from the site.html template will be added #}
    {% block content %}{% endblock %}
</div>
<div id="footer">
    {% block footer %}
    &copy; Copyright 2013 by <a href="http://domain.invalid/">you</a>.
    {% endblock %}
</div>
</body>

The next code example is our site page (site.html), which extends base.html. The content block here will be automatically inserted into the corresponding block in base.html:

<!{% extends "base.html" %}
{% block content %}
    <p class="important">
    <div id="content">
        <h2>{{title}}</h2>
        <p>{{ list_title }}</p>
        <ul>
             {% for item in list %}
             <li>{{ item[0]}} :  {{ item[1]}}</li>
             {% endfor %}
        </ul>
    </div>
    </p>
{% endblock %}

Chameleon

Chameleon Page Templates (with file extension *.pt) are an HTML/XML template engine implementation of the Template Attribute Language (TAL), TAL Expression Syntax (TALES), and Macro Expansion TAL (Metal) syntaxes. Chameleon parses the Page Templates and “compiles” them into Python bytecode to increase loading speed. It is available for Python 2.5 and up (including 3.x and PyPy) and is one of the two default rendering engines used by “Pyramid”. (The other is Mako, described in the next section.)

Page Templates add special element attributes and text markup to your XML document: a set of simple language constructs let you control the document flow, element repetition, text replacement, and translation. Because of the attribute-based syntax, unrendered page templates are valid HTML and can be viewed in a browser and even edited in WYSIWYG (what you see is what you get) editors. This can make round-trip collaboration with designers and prototyping with static files in a browser easier. The basic TAL language is simple enough to grasp from an example:

<html>
  <body>
  <h1>Hello, <span tal:replace="context.name">World</span>!</h1>
    <table>
      <tr tal:repeat="row 'apple', 'banana', 'pineapple'">
        <td tal:repeat="col 'juice', 'muffin', 'pie'">
           <span tal:replace="row.capitalize()" /> <span tal:replace="col" />
        </td>
      </tr>
    </table>
  </body>
</html>

The <span tal:replace="expression" /> pattern for text insertion is common enough that if you do not require strict validity in your unrendered templates, you can replace it with a more terse and readable syntax using the pattern ${expression}, as follows:

<html>
  <body>
    <h1>Hello, ${world}!</h1>
    <table>
      <tr tal:repeat="row 'apple', 'banana', 'pineapple'">
        <td tal:repeat="col 'juice', 'muffin', 'pie'">
           ${row.capitalize()} ${col}
        </td>
      </tr>
    </table>
  </body>
</html>

But remember the full <span tal:replace="expression">Default Text</span> syntax also allows for default content in the unrendered template.

Being from the Pyramid world, Chameleon is not widely used.

Mako

Mako is a template language that compiles to Python for maximum performance. Its syntax and API are borrowed from the best parts of other templating languages like Django and Jinja2 templates. It is the default template language included with the Pyramid web framework (discussed in “Pyramid”) web framework. An example template in Mako looks like this:

<%inherit file="base.html"/>
<%
    rows = [[v for v in range(0,10)] for row in range(0,10)]
%>
<table>
    % for row in rows:
        ${makerow(row)}
    % endfor
</table>

<%def name="makerow(row)">
    <tr>
    % for name in row:
        <td>${name}</td>\
    % endfor
    </tr>
</%def>

It is a text markup language, like Jinja2, so it can be used for anything, not just XML/HTML documents. To render a very basic template, you can do the following:

from mako.template import Template
print(Template("hello ${data}!").render(data="world"))

Mako is well respected within the Python web community. It is fast and allows developers to embed a lot of Python logic into the page, which we know we cautioned against—but in times when you think it’s necessary, this is the tool that allows it.

Web Deployment

The two options we’ll cover for web deployment are either to use web hosting (i.e., pay a vendor like Heroku, Gondor, or PythonAnywhere to manage your server and your database for you), or to set up your own infrastructure on a machine provided by a virtual private server (VPS) host like Amazon Web Services or Rackspace. We’ll quickly cover both.

Hosting

Platform as a Service (PaaS) is a type of cloud computing infrastructure which abstracts and manages infrastructure (e.g., setting up the database and web server, and keeping up with security patches), routing, and scaling of web applications. When using a PaaS, application developers can focus on writing application code rather than concerning themselves with deployment details.

There are dozens of competing PaaS providers, but the ones in this list specifically focus on the Python community. Most offer some sort of free tier or trial to start:

Heroku: Heroku is our recommended PaaS for deploying Python web applications. It supports Python 2.7–3.5 applications of all types: web applications, servers, and frameworks. A set of command-line tools is provided for interfacing with both your Heroku account and the actual database and web servers that support your application, so you can make changes without using a web interface. Heroku maintains detailed articles on using Python with Heroku, as well as step-by-step instructions on how to set up your first application.
Gondor: Gondor is run by a small company and focuses on helping businesses find success with Python and Django. Its platform is specialized for deploying Django and Pinax¹⁰ applications and uses. Gondor’s platform is Ubuntu 12.04 with Django 1.4, 1.6, and 1.7 and a subset of Python 2 and 3 implementation listed here. It can automatically configure your Django site if you use local_settings.py for site-specific configuration information. For more information, see Gondor’s guide to deploying Django projects; a command-line interface tool is also available.
PythonAnywhere: PythonAnywhere supports Django, Flask, Tornado, Pyramid, and many of the other web application frameworks we didn’t describe, like Bottle (no framework, like Flask, but with a much smaller community) and web2py (great for teaching). Its pricing model is related to compute time—rather than charging more, computations are throttled once they go over a daily maximum—which is good for cost-conscious developers.

Web servers

With the exception of Tornado (which comes with its own HTTP server), all of the web application frameworks we discussed are WSGI applications. This means they must interact with a WSGI server as defined in PEP 3333 in order to receive an HTTP request and send back an HTTP response.

The majority of self-hosted Python applications today are hosted with a WSGI server such as Gunicorn, either by itself—WSGI servers can often be used as standalone HTTP servers—or behind a lightweight web server such as Nginx. When both are used, the WSGI servers interact with the Python applications while the web server handles tasks better suited for it—static file serving, request routing, distributed denial-of-service (DDoS) protection, and basic authentication. The two most popular web servers are Nginx and Apache, described here:

Nginx: Nginx (pronounced “engine-x”) is a web server and reverse proxy¹¹ for HTTP, SMTP, and other protocols. It is known for its high performance, relative simplicity, and compatibility with many application servers (like WSGI servers). It also includes handy features like load balancing,¹² basic authentication, streaming, and others. Designed to serve high-load websites, Nginx is gradually becoming quite popular.
Apache HTTP server: Apache is the most popular HTTP server in the world, but we prefer Nginx. Still, those new to deployment may want to start with Apache and mod_wsgi, which is regarded as the easiest WSGI interface out there. There are tutorials in each framework’s documentation for mod_wsgi with Pyramid, mod_wsgi with Django, and mod_wsgi with Flask.

WSGI servers

Standalone WSGI servers typically use less resources than traditional web servers and provide top performance benchmarks for Python WSGI Servers. They can also be used in conjunction with Nginx or Apache, which would serve as reverse proxies. The most popular WSGI servers are:

Gunicorn (Green Unicorn)

Gunicorn is the recommended choice for new Python web applications—a pure-Python WSGI server used to serve Python applications. Unlike other Python web servers, it has a thoughtful user interface and is extremely easy to use and configure. Gunicorn has sane and reasonable defaults for configurations. However, some other servers, like uWSGI, are tremendously more customizable (but therefore are much more difficult to effectively use).

Waitress

Waitress is a pure-Python WSGI server that claims “very acceptable performance.” Its documentation is not very detailed, but it does offer some nice functionality that Gunicorn doesn’t have (e.g., HTTP request buffering); it doesn’t block when a slow client takes time to respond—hence the name “Wait”-ress. Waitress is gaining popularity within the Python web development community.

uWSGI

uWSGI is a full stack for building hosting services. We don’t recommend using it as a standalone web router unless you know why you need it.

But uWSGI can also run behind a full web server (such as Nginx or Apache)—a web server can configure uWSGI and an application’s operation over the uwsgi protocol. uWSGI’s web server support allows for dynamically configuring Python, passing environment variables and further tuning. For full details, see uWSGI magic variables .

¹ docopt uses neither optparse nor argparse and relies on regular expressions to parse the docstring.

² Tcl, originally Tool Command Language, is a lightweight language created by John Ousterhout in the early 1990s for integrated circuit design.

³ In addition to supporting the usual mouse, it can handle touch: TUIO (an open source touch and gesture protocol and API), Nintendo’s Wii remote, WM_TOUCH (the Windows touch API), USB touchscreens using HidTouch Apple’s products, and others.

⁴ GIMP stands for GNU Image Manipulation Program. GTK+ was built to support drawing in GIMP, but became popular enough that people wanted to make a whole desktop windowing environment with it—hence GNOME.

⁵ But the creation of Swift may have reduced that demand—it’s almost as easy as Python, so if you’re just writing for OS X, why not just use Swift and do everything native (except for calculations, which still would benefit from scientific libraries like NumPy and Pandas)?

⁶ It was inspired by the Twisted project’s Twisted Web server, which is a part of the Tornado networking toolkit. If you wish things existed in Tornado that don’t, look into Twisted, because it’s probably implemented them. But be warned that Twisted is notoriously hard for beginners.

⁷ Actually, their WSGI documentation says “Use WSGIContainer only when there are benefits to combining Tornado and WSGI in the same process that outweigh the reduced scalability.”

⁸ Separation of concerns is a design principle that means good code is modular—each component should do just one thing.

⁹ Rendering is rarely the bottleneck in a web app, though—it’s usually the data access.

¹⁰ Pinax bundles popular Django templates, apps, and infrastructure to make starting a Django project faster.

¹¹ A reverse proxy fetches information from another server on behalf of a client and returns it to the client as if it come from the reverse proxy.

¹² Load balancing optimizes performance by delegating work across multiple computing resources.

Previous Chapter

III. Scenario Guide

Next Chapter

8. Code Management and Improvement

Table of Contents for The Hitchhiker's Guide to Python