Tox

Tox is an automation tool providing packaging, testing, and deployment of Python software right from the console or CI server. It is a generic virtualenv management and test command-line tool that provides the following features:

• Checks that packages install correctly with different Python versions and interpreters

• Runs tests in each of the environments, configuring your test tool of choice

• Acts as a frontend to continuous integration servers, reducing boilerplate and merging CI and shell-based testing

Install it using pip:

$ pip install tox

Travis-CI

Travis-CI is a distributed CI server which builds tests for open source projects for free. It provides multiple workers that run Python tests and seamlessly integrates with GitHub. You can even have it comment on your pull requests² whether this particular set of changes breaks the build or not. So if you are hosting your code on GitHub, Travis-CI is a great and easy way to get started with continuous integration. Travis-CI can build your code on a virtual machine that is running Linux, OS X, or iOS.

To get started, add a .travis.yml file to your repository with this example content:

language: python
python:
  - "2.6"
  - "2.7"
  - "3.3"
  - "3.4"
script: python tests/test_all_of_the_units.py
branches:
  only:
    - master

This will get your project tested on all the listed Python versions by running the given script and will only build the master branch. There are a lot more options you can enable, like notifications, before and after steps, and much more. The Travis-CI docs explain all of these options and are very thorough. To use Tox with Travis-CI, add a Tox script to your repository, and change the line with script: in it to become:

install:
  - pip install tox
script:
  - tox

In order to activate testing for your project, go to the Travis-CI site and log in with your GitHub account. Then activate your project in your profile settings and you’re ready to go. From now on, your project’s tests will be run on every push to GitHub.

Jenkins

Jenkins CI is an extensible continuous integration engine and currently the most popular CI engine. It works on Windows, Linux, and OS X and plugs in to “every Source Code Management (SCM) tool that exists.” Jenkins is a Java servlet (the Java equivalent of a Python WSGI application) that ships with its own servlet container, so you can run it directly using java --jar jenkins.war. For more information, refer to the Jenkins installation instructions; the Ubuntu page has instructions for how to place Jenkins behind an Apache or Nginx reverse proxy.

You interact with Jenkins via a web-based dashboard, or its HTTP-based RESTful API³ (e.g., at http://myServer:8080/api), meaning we can use HTTP to communicate with the Jenkins server from remote machines. For examples, look at Apache’s Jenkins Dashboard or the Pylons project’s Jenkins Dashboard.

The most frequently used Python tool to interact with the Jenkins API is python-jenkins, created by the OpenStack⁴ infrastructure team. Most Python users configure Jenkins to run a Tox script as part of the build process. For more information, see the documentation for using Tox with Jenkins and this guide to set up Jenkins with multiple build machines.

Buildbot

Buildbot is a Python system to automate the compile/test cycle to validate code changes. It works like Jenkins in that it polls your source control manager for changes, builds and test your code on multiple computers according to your instructions (with built-in support for Tox), and then tells you what happened. It runs behind a Twisted web server. For an example of what the web interface looks like, here is Chromium’s public buildbot dashboard (Chromium powers the Chrome browser).

Because Buildbot is pure Python, it’s installed via pip:

$ pip install buildbot

The 0.9 version has a REST API, but it is still in beta, so you won’t be able to use it unless you expressly specify the version number (e.g., pip install buildbot==0.9.00.9.0rc1). Buildbot has a reputation for being the most powerful, but also the most complex of the continuous integration tools. To get started, follow their excellent tutorial.

Server Automation

Salt, Ansible, Puppet, Chef, and CFEngine are server automation tools that provide an elegant way for system administrators to manage their fleet of physical and virtual machines. They all can manage Linux, Unix-like systems, and Windows machines. We’re of course partial to Salt and Ansible, as they’re written in Python. But they’re still new, and the other options are more widely used. The following sections provide a quick summary of these options.

$ pip install salt  # No Python 3 yet ...

$ salt '*' test.ping

$ salt -G 'os:CentOS' test.ping

apache:
  pkg:
    - installed
  service:
    - running
    - enable: True
    - require:
      - pkg: apache

$ pip install ansible  # No Python 3 yet...

[server_name]
127.0.0.1

---
- hosts: all

  tasks:
    - name: ping
      action: ping

$ ansible-playbook ping.yml -i hosts.yml --ask-pass

$ facter kernel
Linux
$
$ facter operatingsystem
Ubuntu

notify { 'Hello World, this message is getting logged into the agent node':

    #As nothing is specified in the body, the resource title
    #is the notification message by default.
}

notify{ 'Mac Warning':
    message => $operatingsystem ? {
        'Darwin' => 'This seems to be a Mac.',
        default  => 'I am a PC.',
    },
}

package { 'openssh-server':
    ensure => installed,
}

file { '/etc/ssh/sshd_config':
    source   => 'puppet:///modules/sshd/sshd_config',
    owner    => 'root',
    group    => 'root',
    mode     => '640',
    notify   =>  Service['sshd'], # sshd will restart
                                  # whenever you edit this
                                  # file
    require  => Package['openssh-server'],

}

service { 'sshd':
    ensure    => running,
    enable    => true,
    hasstatus => true,
    hasrestart=> true,
}

$ knife cookbook create cookbook_name

System and Task Monitoring

The following libraries all help system administrators monitor running jobs but have very different applications: Psutil provides information in Python that can be obtained by Unix utility functions, Fabric makes it easy to define and execute commands on a list of remote hosts via SSH, and Luigi makes it possible to schedule and monitor long-running batch processes like chained Hadoop commands.

$ pip install psutil

# Functions to get system values:
from psutil import cpu_percent, net_io_counters
# Functions to take a break:
from time import sleep
# Package for email services:
import smtplib
import string

MAX_NET_USAGE = 400000
MAX_ATTACKS = 4
attack = 0
counter = 0
while attack <= MAX_ATTACKS:
    sleep(4)
    counter = counter + 1
    # Check the CPU usage
    if cpu_percent(interval = 1) > 70:
        attack = attack + 1
    # Check the net usage
    neti1 = net_io_counters()[1]
    neto1 = net_io_counters()[0]
    sleep(1)
    neti2 = net_io_counters()[1]
    neto2 = net_io_counters()[0]
    # Calculate the bytes per second
    net = ((neti2+neto2) - (neti1+neto1))/2
    if net > MAX_NET_USAGE:
        attack = attack + 1
    if counter > 25:
        attack = 0
        counter = 0

# Write a very important email if attack is higher than 4
TO = "you@your_email.com"
FROM = "webmaster@your_domain.com"
SUBJECT = "Your domain is out of system resources!"
text = "Go and fix your server!"
BODY = string.join(
        ("From: %s" %FROM,"To: %s" %TO,"Subject: %s" %SUBJECT, "",text), "\r\n")
server = smtplib.SMTP('127.0.0.1')
server.sendmail(FROM, [TO], BODY)
server.quit()

$ pip install fabric

# fabfile.py
from fabric.api import cd, env, prefix, run, task

env.hosts = ['my_server1', 'my_server2']   # Where to SSH

@task
def memory_usage():
    run('free -m')

@task
def deploy():
    with cd('/var/www/project-env/project'):
        with prefix('. ../bin/activate'):
            run('git pull')
            run('touch app.wsgi')

$ ssh hostname cd /var/ww/project-env/project && ../bin/activate && git pull
$ ssh hostname cd /var/ww/project-env/project && ../bin/activate && \
> touch app.wsgi

$ fab memory_usage
[my_server1] Executing task 'memory'
[my_server1] run: free -m
[my_server1] out:              total     used     free   shared  buffers   cached
[my_server1] out: Mem:          6964     1897     5067        0      166      222
[my_server1] out: -/+ buffers/cache:     1509     5455
[my_server1] out: Swap:            0        0        0

[my_server2] Executing task 'memory'
[my_server2] run: free -m
[my_server2] out:              total     used     free   shared  buffers   cached
[my_server2] out: Mem:          1666      902      764        0      180      572
[my_server2] out: -/+ buffers/cache:      148     1517
[my_server2] out: Swap:          895        1      894

$ fab deploy

$ pip install luigi

Threading

Python’s threading library allows you to create multiple threads. Because of the GIL (at least in CPython), there will only be one Python process running per Python interpreter, meaning there will only be a performance gain when at least one thread is blocking (e.g., on I/O). The other option for I/O is to use event handling. For that, see the paragraphs on asyncio in “Performance networking tools in Python’s Standard Library”.

What happens in Python when you have multiple threads is the kernel notices that one thread is blocking on I/O, and it switches to allow the next thread to use the processor until it blocks or is finished. All of this happens automatically when you start your threads. There’s a good example use of threading on Stack Overflow, and the Python Module of the Week series has a great threading introduction. Or see the threading documentation in the Standard Library.

Multiprocessing

The multiprocessing module in Python’s Standard Library provides a way to bypass the GIL—by launching additional Python interpreters. The separate processes can communicate using a multiprocessing.Pipe, or by a multiprocessing.Queue, or share memory via a multiprocessing.Array and multiprocessing.Value, which implement locking automatically. Share data sparingly; these objects implement locking to prevent simultaneous access by different processes.

Here’s an example to show that the speed gain from using a pool of worker processes isn’t always proportional to the number of workers used. There’s a trade-off between the computational time saved and the time it takes to launch another interpeter. The example uses the Monte Carlo method (of drawing random numbers) to estimate the value of Pi:⁷

>>> import multiprocessing
>>> import random
>>> import timeit
>>>
>>> def calculate_pi(iterations):
...     x = (random.random() for i in range(iterations))
...     y = (random.random() for i in range(iterations))
...     r_squared = [xi**2 + yi**2 for xi, yi in zip(x, y)]
...     percent_coverage = sum([r <= 1 for r in r_squared]) / len(r_squared)
...     return 4 * percent_coverage
...
>>>
>>> def run_pool(processes, total_iterations):
...     with multiprocessing.Pool(processes) as pool:  
...         # Divide the total iterations among the processes.
...         iterations = [total_iterations // processes] * processes  
...         result = pool.map(calculate_pi, iterations)  
...     print( "%0.4f" % (sum(result) / processes), end=',  ')
...
>>>
>>> ten_million = 10000000        
>>> timeit.timeit(lambda: run_pool(1, ten_million), number=10)
3.141,  3.142,  3.142,  3.141,  3.141,  3.142,  3.141,  3.141,  3.142,  3.142,
134.48382110201055 
>>>                                
>>> timeit.timeit(lambda: run_pool(10, ten_million), number=10)
3.142,  3.142,  3.142,  3.142,  3.142,  3.142,  3.141,  3.142,  3.142,  3.141,
74.38514468498761  

Using the multiprocessing.Pool within a context manager reinforces that the pool should only be used by the process that creates it.

The total iterations will always be the same; they’ll just be divided between a different number of processes.

pool.map() creates the multiple processes—one per item in the iterations list, up to the maximum number stated when the pool was initialized (in multiprocessing.Pool(processes)).

There is only one process for the first timeit trial.

10 repetitions of one single process running with 10 million iterations took 134 seconds.

There are 10 processes for the second timeit trial.

10 repetitions of 10 processes each running with one million iterations took 74 seconds.

The point of all this was that there is overhead in making the multiple processes, but the tools for runing multiple processes in Python are robust and mature. See the multiprocessing documentation in the Standard Library for more information, and check out Jeff Knupp’s blog post about getting around the GIL, because it has a few paragraphs about multiprocessing.

Subprocess

The subprocess library was introduced into the Standard Library in Python 2.4 and defined in PEP 324. It launches a system call (like unzip or curl) as if called from the command line (by default, without calling the system shell), with the developer selecting what to do with the subprocess’s input and output pipes. We recommend Python 2 users get an updated version with some bugfixes from the subprocess32 package. Install it using pip:

$ pip install subprocess32

There is a great subprocess tutorial on the Python Module of the Week blog.

PyPy

PyPy is a pure-Python implementation of Python. It’s fast, and when it works, you don’t have to do anything to your code, and it just runs faster for free. You should try this option before anything else.

You can’t get it using pip, because it’s actually another implementation of Python. Scroll through the PyPy downloads page for your correct version of Python and your operating system.

Here is a slightly modified version of David Beazley’s CPU bound test code, with an added loop for multiple tests. You can see the difference between PyPy and CPython. First it’s run using the CPython:

$ # CPython
$ ./python -V
Python 2.7.1
$
$ ./python measure2.py
1.06774401665
1.45412397385
1.51485204697
1.54693889618
1.60109114647

And here is the same script, and the only thing different is the Python interpreter—it’s running with PyPy:

$ # PyPy
$ ./pypy -V
Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:10)
[PyPy 1.7.0 with GCC 4.4.3]
$
$ ./pypy measure2.py
0.0683999061584
0.0483210086823
0.0388588905334
0.0440690517426
0.0695300102234

So, just by downloading PyPy, it went from an average of about 1.4 seconds to around 0.05 seconds—more than 20 times faster. Sometimes your code won’t even double in speed, but other times you really do get a big boost. And with no effort outside of downloading the PyPy interpreter. If you want your C library to be compatible with PyPy, follow PyPy’s advice and use the CFFI instead of ctypes in the Standard Library.

Cython

Unfortunately, PyPy doesn’t work with all libraries that use C extensions. For those cases, Cython (pronounced “PSI-thon”—not the same as CPython, the standard C implementation of Python) implements a superset of the Python language that lets you write C and C++ modules for Python. Cython also allows you to call functions from compiled C libraries, and provides a context, nogil, that allows you to release the GIL around a section of code, provided it does not manipulate Python objects in any way. Using Cython allows you to take advantage of Python’s strong typing⁸ of variables and operations.

Here’s an example of strong typing with Cython:

def primes(int kmax):
"""Calculation of prime numbers with additional Cython keywords"""

    cdef int n, k, i
    cdef int p[1000]
    result = []
    if kmax > 1000:
        kmax = 1000
    k = 0
    n = 2
    while k < kmax:
        i = 0
        while i < k and n % p[i] != 0:
            i = i + 1
        if i == k:
            p[k] = n
            k = k + 1
            result.append(n)
        n = n + 1
    return result

This implementation of an algorithm to find prime numbers has some additional keywords compared to the next one, which is implemented in pure Python:

def primes(kmax):
"""Calculation of prime numbers in standard Python syntax"""

    p= range(1000)
    result = []
    if kmax > 1000:
        kmax = 1000
    k = 0
    n = 2
    while k < kmax:
        i = 0
        while i < k and n % p[i] != 0:
            i = i + 1
        if i == k:
            p[k] = n
            k = k + 1
            result.append(n)
        n = n + 1
    return result

Notice that in the Cython version you declare integers and integer arrays to be compiled into C types while also creating a Python list:

# Cython version

def primes(int kmax):  
    """Calculation of prime numbers with additional Cython keywords"""
    cdef int n, k, i  
    cdef int p[1000]  
    result = []

The type is declared to be an integer.

The upcoming variables n, k, and i are declared as integers.

And we then have preallocated a 1000-long array of integers for p.

What is the difference? In the Cython version, you can see the declaration of the variable types and the integer array in a similar way as in standard C. For example, the addtional type declaration (of integer) in the cdef int n,k,i allows the Cython compiler to generate more efficient C code than it could without type hints. Because the syntax is incompatible with standard Python, it is not saved in *.py files—instead, Cython code is saved in *.pyx files.

What’s the difference in speed? Let’s try it!

import time
# activate pyx compiler
import pyximport  
pyximport.install()  
# primes implemented with Cython
import primesCy
# primes implemented with Python
import primes

print("Cython:")
t1 = time.time()
print primesCy.primes(500)
t2 = time.time()
print("Cython time: %s" %(t2-t1))
print("")
print("Python")
t1 = time.time()  
print(primes.primes(500))
t2 = time.time()
print("Python time: {}".format(t2-t1))

The pyximport module allows you to import *.pyx files (e.g., primesCy.pyx) with the Cython-compiled version of the primes function.

The pyximport.install() command allows the Python interpreter to start the Cython compiler directly to generate C-code, which is automatically compiled to a *.so C-library. Cython is then able to import this library for you in your Python code, easily and efficiently.

With the time.time() function, you are able to compare the time between these two different calls to find 500 prime numbers. On a standard notebook (dual-core AMD E-450 1.6 GHz), the measured values are:

Cython time: 0.0054 seconds

Python time: 0.0566 seconds

And here the output of an embedded ARM BeagleBone machine:

Cython time: 0.0196 seconds

Python time: 0.3302 seconds

Numba

Numba is a NumPy-aware Python compiler (just-in-time [JIT] specializing compiler) that compiles annotated Python (and NumPy) code to LLVM (Low-Level Virtual Machine) through special decorators. Briefly, Numba uses LLVM to compile Python down to machine code that can be natively executed at runtime.

If you use Anaconda, install Numba with conda install numba; if not, install it by hand. You must already have NumPy and LLVM installed before installing Numba. Check the LLVM version you need (it’s on the PyPI page for llvmlite), and download that version from whichever place matches your OS:

• LLVM builds for Windows.

• LLVM builds for Debian/Ubuntu.

• LLVM builds for Fedora.

• For a discussion of how to build from source for other Unix systems, see “Building the Clang + LLVM compilers”.

• On OS X, use brew install homebrew/versions/llvm37 (or whatever version number is now current).

Once you have LLVM and NumPy, install Numba using pip. You may need to help the installer find the llvm-config file by providing an environment variable LLVM_CONFIG with the appropriate path, like this:

$ LLVM_CONFIG=/path/to/llvm-config-3.7 pip install numba

Then, to use it in your code, just decorate your functions:

from numba import jit, int32

@jit  
def f(x):
    return x + 3

@jit(int32(int32, int32))  
def g(x, y):
    return x + y

With no arguments, the @jit decorator does lazy compilation—deciding itself whether to optimize the function, and how.

For eager compilation, specify types. The function will be compiled with the given specialization, and no other will be allowed—the return value and the two arguments will all have type numba.int32.

There is a nogil flag that can allow code to ignore the Global Interpreter Lock, and a module numba.pycc that can be used to compile the code ahead of time. For more information, see Numba’s user manual.

GPU libraries

Numba can optionally be built with capacity to run on the computer’s graphics processing unit (GPU), a chip optimized for the fast, parallel computation used in modern video games. You’ll need to have a NVIDIA GPU, with NVIDIA’s CUDA Toolkit installed. Then follow the documentation for using Numba’s CUDA JIT with the GPU.

Outside of Numba, the other popular library with GPU capability is TensorFlow, released by Google under the Apache v2.0 license. It provides tensors (mutidimensional matrices) and a way to chain tensor operations together, for fast matrix math. Currently it can only use the GPU on Linux operating systems. For installation instructions, see the following pages:

• Installing TensorFlow with GPU support

• TensorFlow installation without GPU support

For those not on Linux, Theano, from the University of Montréal, was the de facto matrix-math-over-GPU libary in Python until Google posted TensorFlow. Theano is still under active development. It has a page dedicated to using the GPU. Theano supports Windows, OS X, and Linux operating systems, and is available via pip:

$ pip install Theano

For lower-level interaction with the GPU, you can try PyCUDA.

Finally, people without a NVIDIA GPU can use PyOpenCL, a wrapper for Intel’s OpenCL library, which is compatible with a number of different hardware sets.

Interfacing with C/C++/FORTRAN Libraries

Each of the libraries described in the following sections are very different: both CFFI and ctypes are Python libraries, F2PY is for FORTRAN, SWIG can make C objects available in multiple languages (not just Python), and Boost.Python is a C++ library that can expose C++ objects to Python and vice versa. Table 8-2 goes into a little more detail.

$ pip install cffi

from cffi import FFI
ffi = FFI()
ffi.cdef("size_t strlen(const char*);")  
clib = ffi.dlopen(None)  
length = clib.strlen("String to be evaluated.")  
# prints: 23
print("{}".format(length))

struct my_struct {
    int a;
    int b;
};

import ctypes
class my_struct(ctypes.Structure):
    _fields_ = [("a", c_int),
                ("b", c_int)]

$ pip install numpy

$ f2py -c fortran_code.f -m python_module_name

$ f2py fortran_code.f -m python_module_name -h interface_file.pyf  
$ vim interface_file.pyf  
$ f2py -c interface_file.pyf fortran_code.f  

#include <string>
class MyClass {
private:
    std::string name;
public:
    std::string getName();
};

%include "string.i"

%module myclass
%{
#include <string>
#include "MyClass.h"
%}

%extend MyClass {
    std::string __repr__()
    {
        return $self->getName();
    }
}

%include "MyClass.h"

$ brew install pcre