Python Standard Library (PSL): A Few More Standard Modules

collections (PSL 8.3.)

This module implements specialized container data types as alternatives to Python’s other built-in containers: dict, list, set, and tuple. Some of the containers that tend to be used in data analyses are deque, Counter, defaultdict, and OrderedDict.

random (PSL 9.3.)

This module implements pseudorandom number generators for various distributions. There are functions for selecting a random integer from a range; selecting a random element from a sequence; randomly permuting a sequence; randomly sampling without replacement; and selecting random values from uniform, normal (Gaussian), gamma, beta, and other distributions.

statistics (PSL 9.7.)

This module provides functions for calculating some common statistics of numeric data. There are functions for calculating measures of central location, like mean, median, and mode. There are also functions for calculating measures of spread, like variance and standard deviation.

itertools (PSL 10.1.)

This module provides a set of standardized, fast, memory-efficient iterators (i.e., generators) for several useful data algorithms. There are iterators for merging and splitting sequences, converting input values, producing new values, and filtering and grouping data.

operator (PSL 10.3.)

This module provides a set of efficient functions that correspond to intrinsic operators in Python. There are functions for performing object comparisons, logical operations, mathematical operations, and sequence operations. There are also functions for generalized attribute and item lookups.

These five additional standard modules are a small subset of all of the modules available in Python’s standard library. In fact, there are over 35 sections in the standard library, each providing a wide variety of modules and functions related to specific topics. Because all of these modules are built into Python, you can use them immediately with an import statement, like from itertools import accumulate or from statistics import mode. To learn more about these and other standard modules, peruse the Python Standard Library.

Built-in Functions

Similar to the standard modules just discussed, there are also a few built-in functions that didn’t make it into earlier chapters but are still useful for data processing and analysis. As with the modules, it’s useful to know that these functions are available in case they can help with your specific analysis task. It’s helpful to have the following functions in your Python toolbox:

enumerate()

Expands a sequence into a list of (index, value) tuples

filter()

Applies a function to a sequence and returns the values that are true based on the function call

zip()

Combines two sequences into one sequence by joining pairs of values by position in the sequence

These functions are built into Python, so you can use them immediately. To learn more about these and other built-in functions, peruse the Python list of standard functions. In addition, it always helps to see how people use these functions to accomplish specific analysis tasks. If you’re interested in learning how others have used these functions, perform a quick Google or Bing search like “python enumerate examples” or “python zip examples” to retrieve a set of helpful examples.

NumPy

NumPy (pronounced “Num Pie”) is a foundational Python package that provides the ndarray, a fast, efficient, multidimensional data container for (primarily) numerical data. It also provides vectorized versions of standard mathematical and statistical functions that enable you to operate on arrays without for loops. Some of the helpful functions NumPy provides include functions for reading, reshaping, aggregating, and slicing and dicing structured data (especially numerical data).

As is the case with pandas, which is built on top of NumPy, many of NumPy’s functions encapsulate and simplify techniques you’ve learned in this book. NumPy is a fundamental package that underlies many other add-in packages (and provides the powerful ndarray data structure with vectorized operations), so let’s review some of NumPy’s functionality.

from numpy import loadtxt
my_ndarray = loadtxt('input_file.csv')
print(my_ndarray)

name      age  color      score
clint     32   green      15.6
john      30   blue       22.3
rachel    27   red        31.4

from numpy import dtype, loadtxt
person_dtype = dtype([('name', 'S10'), ('age', int), ('color', 'S6'),\
('score', float)])
people = loadtxt('people.txt', skiprows=1, dtype=person_dtype)
print(people)

from numpy import genfromtxt
name_to_int = dict(rachel=1, john=2, clint=3)
color_to_int = dict(blue=1, green=2, red=3)
def convert_name(n):
    return name_to_int.get(n, -999)
def convert_color(c):
    return color_to_int.get(c, -999)
data = genfromtxt('people.txt', dtype=float, names=True, \
converters={0:convert_name, 2:convert_color})
print(data)

2.1,3.2,4.3
3.2,4.3,5.2
4.3,2.6,1.5

import csv
from numpy import array
file = open('myCSVInputFile.csv', 'r')
file_reader = csv.reader(file)
data = []
for row_list in file_reader:
    row_list_floats = [float(value) for value in row_list]
    data.append(row_list_floats)
file.close()
data = array(data)
print(data)

from pandas import read_excel
from numpy import array
myDataFrame = read_excel('myExcelInputFile.xlsx')
data = array(myDataFrame)
print(data)

from numpy import savetxt
savetxt('output_file.txt', data)

savetxt('output_file.txt', data, fmt='%d')
savetxt('output_file.csv', data, fmt='%.2f', delimiter=',')

column_headings_list = ['var1', 'var2', 'var3']
header_string = ','.join(column_headings_list)
savetxt('output_file.csv', data, fmt='%.2f', delimiter=',', \
comments='', header=header_string)

row_filter1 = (data['Cost'] > 110) & (data['Supplier'] == 3)
data[row_filter1]
row_filter2 = (data['Quantity'] > 55) | (data['Time to Delivery'] > 30)
data[row_filter2]

import numpy as np
def columns_view(arr, fields):
  dtype2 = np.dtype({name:arr.dtype.fields[name] for name in fields})
  return np.ndarray(arr.shape, dtype2, arr, 0, arr.strides)

supplies_view = columns_view(supplies, ['Supplier', 'Cost'])
print(supplies_view)
row_filter = supplies['Cost'] > 1000
supplies_row_column_filters = columns_view(supplies[row_filter],\
['Supplier', 'Cost'])
print(supplies_total_cost_gt_1000_two_columns)

import numpy as np
from numpy import concatenate, vstack, r_
array_concat = np.concatenate([array1, array2], axis=0)
array_concat = np.vstack((array1, array2))
array_concat = np.r_[array1, array2]

import numpy as np
from numpy import concatenate, hstack, c_
array_concat = np.concatenate([array1, array2], axis=1)
array_concat = np.hstack((array1, array2))
array_concat = np.c_[array1, array2]

SciPy

SciPy (pronounced “Sigh Pie”) is another foundational Python package that provides scientific and statistical distributions, functions, and tests for mathematics, science, and engineering. SciPy has a broad scope, so its functionality is organized into different subpackages. Some of the subpackages are:

cluster

Provides clustering algorithms

constants

Provides physical and mathematical constants

interpolate

Provides functions for interpolation and smoothing splines

io

Provides input/output functions

linalg

Provides linear algebra operations

sparse

Provides operations for sparse matrices

spatial

Provides spatial data structures and algorithms

stats

Provides statistical distributions and functions

weave

Provides C/C++ integration

As you can see from this list, SciPy’s subpackages provide functionality for a diverse range of operations and calculations. For example, the linalg package provides functions for performing very fast linear algebra operations on two-dimensional arrays; the interpolate package provides functions for linear and curvilinear interpolation between data points; and the stats package provides functions for working with random variables, calculating descriptive and test statistics, and conducting regression.

SciPy is a fundamental package that underlies many other add-in packages (in addition to providing a variety of useful mathematical and statistical functions), so let’s review some of SciPy’s functionality.

from numpy import array
from scipy import linalg
A = array([[1,2,3], [2,3,1], [5,-1,2]])
b = array([[3], [-10], [14]])
solution = linalg.solve(A, b)
print(solution)

import numpy as np
from scipy import linalg
c1, c2 = 6.0, 3.0
i = np.r_[1:21]
xi = 0.1*i
yi = c1*np.exp(-xi) + c2*xi
zi = yi + 0.05 * np.max(yi) * np.random.randn(len(yi))
A = np.c_[np.exp(-xi)[:, np.newaxis], xi[:, np.newaxis]]
c, resid, rank, sigma = linalg.lstsq(A, zi)
print(c)

from numpy import arange, exp
from scipy import interpolate
import matplotlib.pyplot as plt
x = arange(0, 20)
y = exp(-x/4.5)
interpolation_function = interpolate.interp1d(x, y)
new_x = arange(0, 19, 0.1)
new_y = interpolation_function(new_x)
plt.plot(x, y, 'o', new_x, new_y, '-')
plt.show()

from scipy.stats import norm, describe
x = norm.rvs(loc=5, scale=2, size=1000)
print(x.mean())
print(x.min())
print(x.max())
print(x.var())
print(x.std())
x_nobs, (x_min, x_max), x_mean, x_variance, x_skewness, x_kurtosis = describe(x)
print(x_nobs)

from numpy.random import random
from scipy import stats
x = random(20)
y = random(20)
slope, intercept, r_value, p_value, std_err = stats.linregress(x, y)
print("R-squared:", round(r_value**2, 4))

Scikit-Learn

The Scikit-Learn add-in module provides functions for estimating statistical machine learning models, including regression, classification, and clustering models, as well as data preprocessing, dimensionality reduction, and model selection. Scikit-Learn’s functions handle both supervised models, where the dependent variable’s values or class labels are available, and unsupervised models, where the values or class labels are not available. One of the features of Scikit-Learn that distinguishes it from StatsModels is a set of functions for conducting different types of cross-validation (i.e., testing a model’s performance on data that was not used to fit the model).

Testing a model’s performance on the same data that was used to fit the model is a methodological mistake because it is possible to create models that repeat the dependent variable’s values or class labels perfectly with the data used to fit the model. These models might appear to have excellent performance based on their results with the data at hand, but they’re actually overfitting the model data and would not have good performance or provide useful predictions on new data.

To avoid overfitting and estimate a model that will tend to have good performance on new data, it is common to split a dataset into two pieces, a training set and a test set. The training set is used to formulate and fit the model, and the test set is used to evaluate the model’s performance. Because the data used to fit the model is different than the data used to evaluate the model’s performance, the chances of overfitting are reduced. This process of repeatedly splitting a dataset into two pieces—training a model on the training set and testing the model on the test set—is called cross-validation.

There are many different methods of cross-validation, but one basic method is called k-fold cross-validation. In k-fold cross-validation, the original dataset is split into a training set and a test set and then the training set is again split into k pieces, or “folds” (e.g., five or ten folds). Then, for each of the k folds, k – 1 of the folds are used as training data to fit the model and the remaining fold is used to evaluate the model’s performance. In this way, cross-validation creates several performance values, one for each fold, and the final performance measure for the training set is the average of the values calculated for each fold. Finally, the cross-validated model is run on the test data to calculate the overall performance measure for the model.

To see how straightforward it is to formulate statistical learning models in Scikit-Learn, let’s specify a random forest model with cross-validation. If you are not familiar with random forest models, check out the Wikipedia entry for an overview or, for a more in-depth treatment, see The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman or Applied Predictive Modeling by Max Kuhn and Kjell Johnson (both from Springer), which are excellent resources on the topic. You can formulate a random forest model with cross-validation and evaluate the model’s performance with a few lines of code in Scikit-Learn:

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.cross_validation import KFold
from sklearn.ensemble import RandomForestClassifier as RF

y = data_frame['Purchased?']
y_pred = y.copy()
feature_space = data_frame[numeric_columns]
X = feature_space.as_matrix().astype(np.float)
scaler = StandardScaler()
X = scaler.fit_transform(X)

kf = KFold(len(y), n_folds=5, shuffle=True, random_state=123)
for train_index, test_index in kf:
    X_train, X_test = X[train_index], X[test_index]
    y_train = y[train_index]
    clf = RF()
    clf.fit(X_train, y_train)
    y_pred[test_index] = clf.predict(X_test)

accuracy = np.mean(y == y_pred)
print "Random forest: " + "%.3f" % (accuracy)

The first five lines of code import NumPy, pandas, and three components of Scikit-Learn. In order, the three components enable you to center and scale the explanatory variables, carry out k-fold cross-validation, and use the random forest classifier.

The next block of code handles specifying the dependent variable, y; creating the matrix of explanatory variables, X; and centering and scaling the explanatory variables. This block assumes that you’ve already created a pandas DataFrame called data_frame and the data for the dependent variable is in a column called Purchased?. The copy function makes a copy of the dependent variable, assigning it into y_pred, which will be used to evaluate the model’s performance. The next line assumes you’ve created a list of the numeric variables in data_frame so you can select them as your set of explanatory variables. The following line uses NumPy and pandas functions to transform the feature set into a matrix called X. The last two lines in the block use Scikit-Learn functions to create and use a scaler object to center and scale the explanatory variables.

The next block of code implements k-fold cross-validation with the random forest classifier. The first line uses the KFold function to split the dataset into five different pairs, or folds, of training and test sets. The next line is a for loop for iterating through each of the folds. Within the for loop, for each fold, we assign the training and test sets of explanatory variables, assign the training set for the dependent variable, initialize the random forest classifier, fit the random forest model with the training data, and then use the model and test data to estimate predicted values for the dependent variable.

The final block of code calculates and reports the model’s accuracy. The first line uses NumPy’s mean function to calculate the average number of times the predicted values for the dependent variable equal the actual, original data values. The evaluation in parentheses tests whether the two values are equal (i.e., whether they are both 1 or are both 0), so the mean function averages a series of 1s and 0s. If all of the predicted values match the original data values, then the average will be 1. If all of the predicted values do not match the original values, then the average will be 0. Therefore, we want the cross-validated random forest classifier to produce an average value that is close to 1. The final line prints the model’s accuracy, formatted to three decimal places, to the screen.

This example illustrated how to carry out cross-validation with a random forest classifier model in Scikit-Learn. Scikit-Learn enables you to specify many more regression and classification models than were presented in this section. For example, to implement a support vector machine, all you would need to do is add the following import statement (and change the classifier from clf = RF() to clf = SVC()):

from sklearn.svm import SVC

In addition to other models, Scikit-Learn also has functions for data pre-processing, dimensionality reduction, and model selection.

To learn more about Scikit-Learn and how to estimate other models and use other cross-validation methods in Scikit-Learn, check out the Scikit-Learn documentation.

A Few Additional Add-in Packages

In addition to NumPy, SciPy, and Scikit-Learn, there are a few additional add-in packages that you may want to look into, depending on the type of data analysis you need to do. This list represents a tiny fraction of the thousands of add-in Python packages on the Python Package Index and is simply intended as a suggestion of some packages that you might find intriguing and useful:

xarray

Provides a pandas-like toolkit for analysis on multidimensional arrays

SKLL

Provides command-line utilities for running common Scikit-Learn operations

NetworkX

Provides functions for creating, growing, and analyzing complex networks

PyMC

Provides functions for implementing Bayesian statistics and MCMC

NLTK

Provides text processing and analysis libraries for human language data

Cython

Provides an interface for calling and generating fast C code in Python

These packages do not come preinstalled with Python. You have to download and install them separately. To do so, visit the Python Package Index website or the Unofficial Windows Binaries for Python Extension Packages website.

Stacks

A stack is an ordered collection of items where you add an item to and remove an item from the same end of the stack. You can only add or remove one item at a time. The end where you add and remove items is called the top. The opposite end is called the base. Given a stack’s ordering principle, items near the top have been in the stack for less time than items near the base. In addition, the order in which you remove items from the stack is opposite to the order in which you add them. This property is called LIFO (last in, first out).

Consider a stack of trays in a cafeteria. To create the stack, you place a tray on the counter, then you place another tray on top of the first tray, and so on. To shrink the stack, you take a tray from the top of the stack. You can add or remove a tray at any time, but it must always be added to or removed from the top.

There are lots of data processing and analysis situations in which it’s helpful to use stacks. People implement stacks to allocate and access computer memory, to store command-line and function arguments, to parse expressions, to reverse data items, and to store and backtrack through URLs.

Queues

A queue is an ordered collection of items where you add items to one end of the queue and remove items from the other end of the queue. In a queue, you add items to the rear and they make their way to the front, where they are removed. Given a queue’s ordering principle, items near the rear have been in the queue for less time than items near the front. This property is called FIFO (first in, first out).

Consider any well-maintained queue, or line, you’ve ever waited in. Regardless of whether you’re at a theme park, a movie theater, or a grocery store, you enter the queue at the back and then wait until you make your way to the front of the queue, where you receive your ticket or service; then you leave the queue.

There are lots of data processing and analysis situations in which it’s helpful to use queues. People implement queues to process print jobs on a printer, to hold computer processes waiting for resources, and to optimize queues and network flows.

Graphs

A graph is a set of nodes (a.k.a. vertices) and edges, which connect to nodes. The edges can be directed, representing a direction between two nodes, or undirected, representing a connection between two nodes with no particular direction. They can also have weights that represent some relationship between the two nodes, which depends on the context of the problem.

Consider any graphical representation of the relationship between people, places, or topics. For example, imagine a collection of actors, directors, and movies as nodes on a canvas and edges between the nodes indicating who acted in or directed the movies. Alternatively, imagine cities as nodes on a canvas and the edges between the nodes indicating the paths to get from one city to the next. The edges can be weighted to indicate the distance between two cities.

There are lots of data processing and analysis situations in which it’s helpful to use graphs. People implement graphs to represent relationships among suppliers, customers, and products; to represent relationships between entities; to represent maps; and to represent capacity and demand for different resources.

Trees

A tree is a specific type of graph data structure consisting of a hierarchical set of nodes and edges. In a tree, there is a single topmost node, which is designated as the root node. The root node may have any number of child nodes. Each of these nodes may also have any number of child nodes. A child node may only have one parent node. If each node has a maximum of two child nodes, then the tree is called a binary tree.

Consider elements in HTML. Within the html tags there are head and body tags. Within the head tag, there may be meta and title tags. Within the body tag, there may be h1, div, form, and ul tags. Viewing these tags in a tree structure, you see the html tag as the root node with two child nodes, the head and body tags. Under the node for the head tag, you see meta and title tags. Under the node for the body tag, you see h1, div, form, and ul tags.

There are lots of data processing and analysis situations in which it’s helpful to use trees. People implement trees to create computer filesystems, to manage hierarchical data, to make information easy to search, and to represent the phrase structure of sentences.

This section presented a very brief introduction some classic data structures. A nice resource for learning more about these data structures in Python is Brad Miller and David Ranum’s online book, Problem Solving with Algorithms and Data Structures Using Python.

It’s helpful to know that these data containers exist so you have the option to use them when your initial implementation isn’t performing well. By knowing about these data structures and understanding when to use one type versus another, you’ll be able to solve a variety of large, difficult problems and improve the processing time and memory utilization of many of your scripts.