nltk

The Natural Language ToolKit (nltk) is the Python tool for text analysis: originally released by Steven Bird and Edward Loper to aid students in Bird’s course on Natural Language Processing (NLP) at the University of Pennsylvania in 2001, it has grown to an expansive library covering multiple languages and containing algorithms for recent research in the field. It is available under the Apache 2.0 license and is downloaded from PyPI over 100,000 times per month. Its creators have an accompanying book, Natural Language Processing with Python (O’Reilly), that is accessible as a course text introducing both Python and NLP.

You can install nltk from the command line using pip.³ It also relies on NumPy, so install that first:

$ pip install numpy
$ pip install nltk

If you’re using Windows, and can’t get the NumPy installed using pip to work, you can try following the instructions in this Stack Overflow post. The size and scope of the library may unnecessarily scare some people away, so here’s a tiny example to demonstrate how easy simple uses can be. First, we need to get a dataset from the separately downloadable collection of corpora, including tagging tools for multiple languages and datasets to test algorithms against. These are licensed separate from nltk, so be sure to check your selection’s individual license. If you know the name of the corpus you want to download (in our case, the Punkt tokenizer,⁴ which we can use to split up text files into sentences or words), you can do it on the command line:

$ python3 -m nltk.downloader punkt --dir=/usr/local/share/nltk_data

Or you can download it in an interactive session—“stopwords” contains a list of common words that tend to overpower word counts, such as “the”, “in”, or “and” in many languages:

>>> import nltk
>>> nltk.download('stopwords', download_dir='/usr/local/share/nltk_data')
[nltk_data] Downloading package stopwords to /usr/local/share/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
True

And if you don’t know the name of the corpus you want, you can launch an interactive downloader from the Python interpreter by invoking nltk.download() without its first argument:

>>> import nltk
>>> nltk.download(download_dir='/usr/local/share/nltk_data')

Then we can load the dataset we care about, and process and analyze it. In this code sample, we are loading a saved copy of the Zen of Python:

>>> import nltk
>>> from nltk.corpus import stopwords
>>> import string
>>>
>>> stopwords.ensure_loaded()  
>>> text = open('zen.txt').read()
>>> tokens = [
...     t.casefold() for t in nltk.tokenize.word_tokenize(text)  
...     if t not in string.punctuation
... ]
>>>
>>> counter = {}
>>> for bigram in nltk.bigrams(tokens):  
...     counter[bigram] = 1 if bigram not in counter else counter[bigram] + 1
...
>>> def print_counts(counter):  # We'll reuse this
...     for ngram, count in sorted(
...             counter.items(), key=lambda kv: kv[1], reverse=True):  
...         if count > 1:
...             print ('{:>25}: {}'.format(str(ngram), '*' * count))  
...
>>> print_counts(counter)
       ('better', 'than'): ********  
         ('is', 'better'): *******
        ('explain', 'it'): **
            ('one', '--'): **
        ('to', 'explain'): **
            ('if', 'the'): **
('the', 'implementation'): **
 ('implementation', 'is'): **
>>>
>>> kept_tokens = [t for t in tokens if t not in stopwords.words()]  
>>>
>>> from collections import Counter  
>>> c = Counter(kept_tokens)
>>> c.most_common(5)
[('better', 8), ('one', 3), ('--', 3), ('although', 3), ('never', 3)]

The corpora are loaded lazily, so we need to do this to actually load the stopwords corpus.

The tokenizer requires a trained model—the Punkt tokenizer (default) comes with a model trained on English (also default).

A bigram is a pair of adjacent words. We are iterating over the bigrams and counting how many times they occur.

The sorted() function here is being keyed on the count, and sorted in reverse order.

The '{:>25}' right-justifies the string with a total width of 25 characters.

The most frequently occurring bigram in the Zen of Python is “better than.”

This time, to avoid high counts of “the” and “is”, we remove the stopwords.

In Python 3.1 and later, you can use collections.Counter for the counting.

There’s a lot more in this library—take a weekend and go for it!

SyntaxNet

Google’s SyntaxNet, built on top of TensorFlow, provides a trained English parser (named Parsey McParseface) and the framework to build other models, even in other languages, provided you have labeled data. It is currently only available for Python 2.7; detailed instructions for downloading and using it are on SyntaxNet’s main GitHub page.

Pillow

The Python Imaging Library, or PIL for short, is one of the core libraries for image manipulation in Python. Its was last released in 2009 and was never ported to Python 3. Luckily, there’s an actively developed fork of PIL called Pillow—it’s easier to install, runs on all operating systems, and supports Python 3.

Before installing Pillow, you’ll have to install Pillow’s prerequisites. Find the instructions for your platform in the Pillow installation instructions. After that, it’s straightforward:

$ pip install Pillow

Here is a brief example use of Pillow (yes, the module name to import from is PIL not Pillow):

from PIL import Image, ImageFilter
# Read image
im = Image.open( 'image.jpg' )
# Display image
im.show()

# Applying a filter to the image
im_sharp = im.filter( ImageFilter.SHARPEN )
#Saving the filtered image to a new file
im_sharp.save( 'image_sharpened.jpg', 'JPEG' )

# Splitting the image into its respective bands (i.e., Red, Green,
# and Blue for RGB)
r,g,b = im_sharp.split()

# Viewing EXIF data embedded in image
exif_data = im._getexif()
exif_data

There are more examples of the Pillow library in the Pillow tutorial.

cv2

OpenSource Computer Vision, more commonly known as OpenCV, is a more advanced image manipulation and processing software than PIL. It is written in C and C++, and focuses on real-time computer vision. For example, it has the first model used in real-time face detection (already trained on thousands of faces; this example shows it being used in Python code), a face recognition model, and a person detection model, among others. It has been implemented in several languages and is widely used.

In Python, image processing using OpenCV is implemented using the cv2 and NumPy libraries. OpenCV version 3 has bindings for Python 3.4 and above, but the cv2 library is still linked to OpenCV2, which does not. The installation instructions in the OpenCV tutorial page have explicit details for Windows and Fedora, using Python 2.7. On OS X, you’re on your own.⁵ Finally, here’s an option using Python 3 on Ubuntu. If the installation becomes difficult, you can downlad Anaconda and use that instead; they have cv2 binaries for all platforms, and you consult the blog post “Up & Running: OpenCV3, Python 3, & Anaconda” to use cv2 and Python 3 on Anaconda.

Here’s an example use of cv2:

from cv2 import *
import numpy as np
#Read Image
img = cv2.imread('testimg.jpg')
#Display Image
cv2.imshow('image',img)
cv2.waitKey(0)
cv2.destroyAllWindows()

#Applying Grayscale filter to image
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

#Saving filtered image to new file
cv2.imwrite('graytest.jpg',gray)

There are more Python-implemented examples of OpenCV in this collection of tutorials.

Scikit-Image

A newer library, Scikit-Image, is growing in popularity, thanks partly to having more of its source in Python and also its great documentation. It doesn’t have the full-fledged algorithms like cv2, which you’d still use for algorithms that work on real-time video, but it’s got enough to be useful for scientists—like blob detection and feature detection, plus it has the standard image processing tools like filtering and contrast adjustment. For example, Scikit-image was used to make the image composites of Pluto’s smaller moons. There are many more examples on the main Scikit-Image page.