Training a multilayer perceptron

Multilayer perceptrons are one of the simplest forms of feedforward artificial neural networks. In this section, we will train a multilayer perceptron using the Scikit-Learn library.

Our input data is a series of 18,000 reviews of albums from the website Pitchfork.com¹; each review contains the text of the review, in which the music reviewer discusses the relative merits of the album and the band, as well as a floating-point numeric score between 0 and 10. An excerpt² can be seen in Figure 12-3.

We would like to predict the relative positivity or negativity of a review given the text. Scikit-Learn’s neural net module, sklearn.neural_network, enables us to train a multilayer perceptron to perform classification or regression using the now familiar fit and predict methods. We’ll attempt both a regression to predict the actual numeric score of an album and a classification to predict if the album is “terrible,” “okay,” “good,” or “amazing.”

Figure 12-3. Sample Pitchfork review

First, we create a function, documents, to retrieve the pickled, part-of-speech tagged documents from our corpus reader object, a continuous function to get the original numeric ratings of each album, and a categorical function that uses NumPy’s digitize method to bin the ratings into our four categories:

import numpy as np

def documents(corpus):
    return list(corpus.reviews())

def continuous(corpus):
    return list(corpus.scores())

def make_categorical(corpus):
    """
    terrible : 0.0 < y <= 3.0
    okay     : 3.0 < y <= 5.0
    great    : 5.0 < y <= 7.0
    amazing  : 7.0 < y <= 10.1
    """
    return np.digitize(continuous(corpus), [0.0, 3.0, 5.0, 7.0, 10.1])

Next, we add a train_model function, which will take as input a path to the pickled corpus, a Scikit-Learn estimator, and keyword arguments for whether the labels are continuous, an optional path for storing the fitted model, and the number of folds to use in cross-validation.

Our function instantiates a corpus reader, calls the documents function as well as either continuous or make_categorical to get the input values X and the target values y. We then calculate the cross-validated scores, fit and store the model using the joblib utility from Scikit-Learn, and return the scores:

from sklearn.externals import joblib
from sklearn.model_selection import cross_val_score

def train_model(path, model, continuous=True, saveto=None, cv=12):
    """
    Trains model from corpus at specified path; constructing cross-validation
    scores using the cv parameter, then fitting the model on the full data.
    Returns the scores.
    """
    # Load the corpus data and labels for classification
    corpus = PickledReviewsReader(path)
    X = documents(corpus)
    if continuous:
        y = continuous(corpus)
        scoring = 'r2_score'
    else:
        y = make_categorical(corpus)
        scoring = 'f1_score'

    # Compute cross-validation scores
    scores = cross_val_score(model, X, y, cv=cv, scoring=scoring)

    # Write to disk if specified
    if saveto:
        joblib.dump(model, saveto)

    # Fit the model on entire dataset
    model.fit(X, y)

    # Return scores
    return scores

To create our models for training, we will build two pipelines to streamline the text normalization, vectorization, and modeling steps:

if __name__ == '__main__':
    from transformer import TextNormalizer
    from reader import PickledReviewsReader

    from sklearn.pipeline import Pipeline
    from sklearn.neural_network import MLPRegressor, MLPClassifier
    from sklearn.feature_extraction.text import TfidfVectorizer

    # Path to postpreprocessed, part-of-speech tagged review corpus
    cpath = '../review_corpus_proc'

    regressor = Pipeline([
        ('norm', TextNormalizer()),
        ('tfidf', TfidfVectorizer()),
        ('ann', MLPRegressor(hidden_layer_sizes=[500,150], verbose=True))
    ])
    regression_scores = train_model(cpath, regressor, continuous=True)

    classifier = Pipeline([
        ('norm', TextNormalizer()),
        ('tfidf', TfidfVectorizer()),
        ('ann', MLPClassifier(hidden_layer_sizes=[500,150], verbose=True))
    ])
    classifer_scores = train_model(cpath, classifier, continuous=False)

Scikit-Learn provides many features for tuning neural networks and can be customized. For example, by default, both MLPRegressor and MLPClassifier use the ReLU activation function, which can be specified with the activation param, and stochastic gradient descent to minimize the cost function, which can be specified with the solver param:

Mean score for MLPRegressor: 0.27290534221341
Mean score for MLPClassifier: 0.7115215174722

The MLPRegressor is fairly weak, showing a very low goodness of fit to the data as described by the R² score. Regression benefits from reducing the number of dimensions particularly with respect to the number of instances. We can reason about this through the lens of the curse of dimensionality; the Pitchfork reviews are on average about 1,000 words in length, and each word adds another dimension to our decision space. Since our corpus consists of only about 18,000 reviews total, our MLPRegressor simply doesn’t have enough instances to predict scores to the degree of float-point numeric precision.

However, we can see that the MLPClassifier has much better results, and is probably worth additional tuning. To improve our MLPClassifier’s performance, we can experiment with adding and removing complexity. We can add complexity by adding more layers and neurons to hidden_layer_sizes. We can also increase the max_iter param to increase the number of training epochs and give our model more time to learn from backpropagation.

We can also decrease the complexity of the model by removing layers, decreasing the number of neurons, or adding a regularization term to the loss function using the alpha param, which, similar to a sklearn.linear_model.RidgeRegression, will artificially shrink the parameters to help prevent overfitting.

Using the Scikit-Learn API to construct neural models is very convenient for simple models. However, as we will see in the next section, libraries like TensorFlow provide much more in the way of flexibility and tuning of the model architecture, as well as speed, leveraging GPUs to scale for higher performance on larger datasets.

TensorFlow: A framework for deep learning

TensorFlow is a distributed computation engine that exposes a framework for deep learning. Developed by Google for the purpose of parallelizing models across not only GPUs but also networks of many machines, it was made open source in November 2015 and has since become one of the most popular publicly available deep learning libraries.

TensorFlow presumes the user has fairly substantial familiarity with neural network architectures, and is geared toward building data flow graphs with a significant degree of customization. In the TensorFlow workflow, we begin by specifying each layer and all hyperparameters, then compile those steps into a static graph, and then run a session to begin the training. While this makes deep learning models easier to control and optimize as their complexity increases, it also makes rapid prototyping much more challenging.

In essence, deep learning models are just chains of functions, which means that many deep learning libraries tend to have a functional or verbose, declarative style. As such, an ecosystem of other libraries, including Keras, TF-slim, TFLearn, and SkFlow, have quickly evolved to provide a more abstract, object-oriented interface for deep learning. In the next section, we will demonstrate how to use TensorFlow through the Keras API.

Keras: An API for deep learning

While it is often grouped together with deep learning frameworks like TensorFlow, Caffe, Theano, and PyTorch, Keras exposes a general API spec for deep learning. The original Keras interface was written for a Theano backend, but following TensorFlow’s open sourcing and dramatic popularity, the Keras API quickly became the default for many TensorFlow users, and was pulled into the TensorFlow core in early 2017.

In Keras, everything is an object, which makes it a particularly convenient tool for prototyping. In order to roughly recreate the multilayer perceptron classifier we used in the previous section, we can create a build_network function that instantiates a Sequential Keras model, adds two Dense (meaning fully connected) hidden layers, the first with 500 nodes and the second with 150, and both of which employ rectified linear units for the activation parameter. Note that in our first hidden layer, we are required to pass in a tuple, input_shape for the shape of the input layer.

In the output layer we must specify a function to condense the dimensions of the previous layer into the same space as the number of classes in our classification problem. In this case we use softmax, a popular choice for natural language processing because it represents a categorical distribution that aligns with the tokens of our corpus. The build_network function then calls the compile method, specifying the desired loss and optimizer functions for gradient descent, and finally returns the compiled network:

from keras.layers import Dense
from keras.models import Sequential

N_FEATURES = 5000
N_CLASSES = 4

def build_network():
    """
    Create a function that returns a compiled neural network
    """
    nn = Sequential()
    nn.add(Dense(500, activation='relu', input_shape=(N_FEATURES,)))
    nn.add(Dense(150, activation='relu'))
    nn.add(Dense(N_CLASSES, activation='softmax'))
    nn.compile(
        loss='categorical_crossentropy',
        optimizer='adam',
        metrics=['accuracy']
    )
    return nn

The keras.wrappers.scikit_learn module exposes KerasClassifier and Keras​Regressor, two wrappers that implement the Scikit-Learn interface. This means that we can incorporate Sequential Keras models as part of a Scikit-Learn Pipeline or Gridsearch.

To use the KerasClassifier in our workflow, we begin our pipeline as in the previous section, with a TextNormalizer (as described in “Creating a custom text normalization transformer”) and a Scikit-Learn TfidfVectorizer. We use the vectorizer’s max_features parameter to pass in the N_FEATURES global variable, which will ensure our vectors have the same dimensionality as our compiled neural network’s input_shape.

Finally, we add the KerasClassifier to the pipeline, passing in the network’s build function, desired number of epochs, and an optional batch size. Note that we must pass build_network in as a function, not as an instance of the compiled network:

if __name__ == '__main__':
    from sklearn.pipeline import Pipeline
    from transformer import TextNormalizer
    from keras.wrappers.scikit_learn import KerasClassifier
    from sklearn.feature_extraction.text import TfidfVectorizer

    pipeline = Pipeline([
        ('norm', TextNormalizer()),
        ('vect', TfidfVectorizer(max_features=N_FEATURES)),
        ('nn', KerasClassifier(build_fn=build_network,
                               epochs=200,
                               batch_size=128))
    ])

We can now run and evaluate our pipeline using a slightly modified version of our train_model function. Just as in the previous section, this function will instantiate a corpus reader, call documents and make_categorical get the input and target values X and y, compute cross-validated scores, fit and store the model, and return the scores.

Unfortunately, at the time of this writing, pipeline persistence for a Scikit-Learn wrapped Keras model is somewhat challenging, since the neural network component must be saved using the Keras-specific save method as a hierarchical data format (.h5) file. As a workaround, we use Pipeline indexing to store first the trained neural network and then the remainder of the fitted pipeline using joblib:

def train_model(path, model, saveto=None, cv=12):
    """
    Trains model from corpus at specified path and fits on full data.
    If a saveto dictionary is specified, writes Keras and Sklearn
    pipeline components to disk separately. Returns the scores.
    """
    corpus = PickledReviewsReader(path)
    X = documents(corpus)
    y = make_categorical(corpus)

    scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy', n_jobs=-1)
    model.fit(X, y)

    if saveto:
        model.steps[-1][1].model.save(saveto['keras_model'])
        model.steps.pop(-1)
        joblib.dump(model, saveto['sklearn_pipe'])

    return scores

Now, back in our if-main statement, we provide the path to the corpus and the dictionary of paths where our serialized model will be stored:

    cpath = '../review_corpus_proc'
    mpath = {
        'keras_model'  : 'keras_nn.h5',
        'sklearn_pipe' : 'pipeline.pkl'
    }
    scores = train_model(cpath, pipeline, saveto=mpath, cv=12)

With 5,000 input features to our neural network, our preliminary Keras classifier performed fairly well; on average, our model is able to predict whether a Pitchfork review considered an album “terrible,” “okay,” “great,” or “amazing”:

Mean score for KerasClassifier: 0.70533893018807

While the mean score of our Scikit-Learn classifier was slightly higher, using Keras, the training took only two hours on a MacBook Pro with all available cores, or roughly one-sixth the training time of the Scikit-Learn MLPClassifier. As a result, we can see that tuning the Keras model (e.g., adding more hidden layers and nodes, adjusting the activation or cost functions, randomly “dropping out,” or setting a fraction of inputs to zero to avoid overfitting, etc.) will allow for more rapid improvements to our model.

However, one of the main challenges for our model is the small size of the dataset; neural networks generally outperform other machine learning model families, but only beyond some threshold of available training data (for more discussion on this, see Andrew Ng’s “Why Is Deep Learning Taking Off?”³). In the next section of this chapter, we will explore a much larger dataset, as well as experimenting with using the kinds of syntactic features we saw in Chapters 7 and 10 to improve the signal-to-noise ratio.

Predicting sentiment with a bag-of-keyphrases

In the previous section, we used Keras to train a simple multilayer perceptron to fairly successfully predict how a Pitchfork music critic would score an album, based on the words contained in their review. In this section, we’ll attempt to build a more complex model with a larger corpus and a “bag-of-keyphrases” approach.

For our new corpus, we will use a subset of the Amazon product reviews corpus compiled by Julian McAuley of the University of California at San Diego.⁶ The full dataset contains over one million reviews of movies and television, each of which is comprised of the text of the review and its score. The scores are categorical, ranging from the lowest rating of 1 to the highest rating of 5.

The premise of our model is that most of the semantic information in our text will be contained in small syntactic substructures within the sentences. Instead of using a bag-of-words approach, we will implement a lightweight method to leverage the syntactic structure of our review text by adding a new KeyphraseExtractor class, which will modify the keyphrase extraction technique from Chapter 7 to transform our review corpus documents in a vector representation of the document keyphrases.

In particular, we will use a regular expression to define a grammar that uses the part-of-speech tags to identify adverbial phrases (“without care”) and adjective phrases (“terribly helpful”). We can then chunk our text into keyphrases using the NLTK RegexpParser. We will be using a neural network model that requires us to know in advance the total number of features (in other words, the lexicon) and the length of each document, so we’ll add these parameters to our __init__ function.

class KeyphraseExtractor(BaseEstimator, TransformerMixin):
    """
    Extract adverbial and adjective phrases, and transform
    documents into lists of these keyphrases, with a total
    keyphrase lexicon limited by the nfeatures parameter
    and a document length limited/padded to doclen
    """
    def __init__(self, nfeatures=100000, doclen=60):
        self.grammar = r'KT: {(<RB.> <JJ.*>|<VB.*>|<RB.*>)|(<JJ> <NN.*>)}'
        self.chunker = RegexpParser(self.grammar)
        self.nfeatures = nfeatures
        self.doclen = doclen

To further reduce complexity, we’ll add a normalize method that removes punctuation from each tokenized, tagged sentence and lowercases the words, and a extract_candidate_phrases that uses our grammar to extract keyphrases from each sentence:

...
    def normalize(self, sent):
        is_punct = lambda word: all(unicat(c).startswith('P') for c in word)
        sent = filter(lambda t: not is_punct(t[0]), sent)
        sent = map(lambda t: (t[0].lower(), t[1]), sent)
        return list(sent)

    def extract_candidate_phrases(self, sents):
        """
        For a document, parse sentences using our chunker created by
        our grammar, converting the parse tree into a tagged sequence.
        Extract phrases, rejoin with a space, and yield the document
        represented as a list of its keyphrases.
        """
        for sent in sents:
            sent = self.normalize(sent)
            if not sent: continue
            chunks = tree2conlltags(self.chunker.parse(sent))
            phrases = [
                " ".join(word for word, pos, chunk in group).lower()
                for key, group in groupby(
                    chunks, lambda term: term[-1] != 'O'
                ) if key
            ]
            for phrase in phrases:
                yield phrase

To pass the size of the input layer to our neural network, we use a get_lexicon method to extract the keyphrases from each review and build a lexicon with the desired number of features. Finally, a clip method will ensure each document includes only keyphrases from the lexicon:

...
    def get_lexicon(self, keydocs):
        """
        Build a lexicon of size nfeatures
        """
        keyphrases = [keyphrase for doc in keydocs for keyphrase in doc]
        fdist = FreqDist(keyphrases)
        counts = fdist.most_common(self.nfeatures)
        lexicon = [phrase for phrase, count in counts]
        return {phrase: idx+1 for idx, phrase in enumerate(lexicon)}

    def clip(self, keydoc, lexicon):
        """
        Remove keyphrases from documents that aren't in the lexicon
        """
        return [lexicon[keyphrase] for keyphrase in keydoc
                if keyphrase in lexicon.keys()]

While our fit method is a no-op, simply returning self, the transform method will do the heavy lifting of extracting the keyphrases, building the lexicons, clipping the documents, and then padding each using Keras’ sequence.pad_sequences function so that they’re all of the same desired length:

from keras.preprocessing import sequence

...
    def fit(self, documents, y=None):
        return self

    def transform(self, documents):
        docs = [list(self.extract_candidate_phrases(doc)) for doc in documents]
        lexicon = self.get_lexicon(docs)
        clipped = [list(self.clip(doc, lexicon)) for doc in docs]
        return sequence.pad_sequences(clipped, maxlen=self.doclen)

Now we will write a function to build our neural network; in this case we’ll be building a long short-term memory (LSTM) network.

Our LSTM will begin with an Embedding layer that will build a vector embedding from our keyphrase documents, specifying three parameters: the total number of features (e.g., the total size of our keyphrase lexicon), the desired dimensionality of the embeddings, and the input_length of each keyphrase document. Our 200-node LSTM layer is nested between two Dropout layers, which will randomly set a fraction of the input units to 0 during each training cycle to help prevent overfitting. Our final layer specifies the number of expected targets for our sentiment classification:

N_FEATURES = 100000
N_CLASSES = 2
DOC_LEN = 60


def build_lstm():
    lstm = Sequential()
    lstm.add(Embedding(N_FEATURES, 128, input_length=DOC_LEN))
    lstm.add(Dropout(0.4))
    lstm.add(LSTM(units=200, recurrent_dropout=0.2, dropout=0.2))
    lstm.add(Dropout(0.2))
    lstm.add(Dense(N_CLASSES, activation='sigmoid'))
    lstm.compile(
        loss='categorical_crossentropy',
        optimizer='adam',
        metrics=['accuracy']
    )
    return lstm

We will be training our sentiment model as a binary classification problem (as Socher did in his implementation), so we will add a binarize function to bin our labels into two categories for use in our train_model function. The two classes will roughly correspond to “liked it” or “hated it”:

def binarize(corpus):
    """
    hated it : 0.0 < y <= 3.0
    liked it : 3.0 < y <= 5.1
    """
    return np.digitize(continuous(corpus), [0.0, 3.0, 5.1])

def train_model(path, model, cv=12, **kwargs):
    corpus = PickledAmazonReviewsReader(path)
    X = documents(corpus)
    y = binarize(corpus)
    scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy')
    model.fit(X, y)

    ...

    return scores

Finally, we’ll specify our Pipeline input and components, and call our train_model function to get our cross-validated scores:

if __name__ == '__main__':
    am_path = '../am_reviews_proc'
    pipeline = Pipeline([
        ('keyphrases', KeyphraseExtractor()),
        ('lstm', KerasClassifier(build_fn=build_nn,
                                 epochs=20,
                                 batch_size=128))
    ])

    scores = train_model(am_path, pipeline, cv=12)

Mean score: 0.8252357452734355

The preliminary results of our model are surprisingly effective, suggesting that keyphrase extraction is a useful way to reduce dimensionality for text data without totally discarding the semantic information encoded in syntactic structures.