k-means clustering

Because it has implementations in familiar libraries like NLTK and Scikit-Learn, k-means is a convenient place to start. A popular method for unsupervised learning tasks, the k-means clustering algorithm starts with an arbitrarily chosen number of clusters, k, and partitions the vectorized instances into clusters according to their proximity to the centroids, which are computed to minimize the within-cluster sum of squares.

We will begin by defining a class, KMeansClusters, which inherits from BaseEstimator and TransformerMixin so it will function inside a Scikit-Learn Pipeline. We’ll use NLTK’s implementation of k-means for our model, since it allows us to define our own distance metric. We initialize the NLTK KMeansClusterer with our desired number of clusters (k) and our preferred distance measure (cosine_distance), and avoid a result with clusters that contain no documents.

from nltk.cluster import KMeansClusterer
from sklearn.base import BaseEstimator, TransformerMixin


class KMeansClusters(BaseEstimator, TransformerMixin):

    def __init__(self, k=7):
        """
        k is the number of clusters
        model is the implementation of Kmeans
        """
        self.k = k
        self.distance = nltk.cluster.util.cosine_distance
        self.model = KMeansClusterer(self.k, self.distance,
                                     avoid_empty_clusters=True)

Now we’ll add our no-op fit() method and a transform() method that calls the internal KMeansClusterer model’s cluster() method, specifying that each document should be assigned a cluster. Our transform() method expects one-hot encoded documents, and by setting assign_clusters=True, it will return a list of the cluster assignments for each document:

    def fit(self, documents, labels=None):
        return self

    def transform(self, documents):
        """
        Fits the K-Means model to one-hot vectorized documents.
        """
        return self.model.cluster(documents, assign_clusters=True)

To prepare our documents for our KMeansClusters class, we need to normalize and vectorize them first. For normalization, we’ll use a version of the TextNormalizer class we defined in “Creating a custom text normalization transformer”, with one small change to the tranform() method. Instead of returning a representation of documents as bags-of-words, this version of the TextNormalizer will perform stopwords removal and lemmatization and return a string for each document:

class TextNormalizer(BaseEstimator, TransformerMixin):

    ...

    def transform(self, documents):
        return [' '.join(self.normalize(doc)) for doc in documents]

To vectorize our documents after normalization and ahead of clustering, we’ll define a OneHotVectorizer class. For vectorization, we’ll use Scikit-Learn’s CountVectorizer with binary=True, which will wrap both frequency encoding and binarization. Our transform() method will return a representation of each document as a one-hot vectorized array:

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.feature_extraction.text import CountVectorizer


class OneHotVectorizer(BaseEstimator, TransformerMixin):

    def __init__(self):
        self.vectorizer = CountVectorizer(binary=True)

    def fit(self, documents, labels=None):
        return self

    def transform(self, documents):
        freqs = self.vectorizer.fit_transform(documents)
        return [freq.toarray()[0] for freq in freqs]

Now, we can create a Pipeline inside our main() execution to perform k-means clustering. We’ll initialize a PickledCorpusReader as defined in “Reading the Processed Corpus”, specifying that we want to use only the “news” category of our corpus. Then we’ll initialize a pipeline to streamline our custom TextNormalizer, OneHotVectorizer, and KMeansClusters classes. By calling fit_transform() on the pipeline, we perform each of these steps in sequence:

from sklearn.pipeline import Pipeline

corpus = PickledCorpusReader('../corpus')
docs = corpus.docs(categories=['news'])

model = Pipeline([
    ('norm', TextNormalizer()),
    ('vect', OneHotVectorizer()),
    ('clusters', KMeansClusters(k=7))
])

clusters = model.fit_transform(docs)
pickles = list(corpus.fileids(categories=['news']))
for idx, cluster in enumerate(clusters):
    print("Document '{}' assigned to cluster {}.".format(pickles[idx],cluster))

Our result is a list of cluster assignments corresponding to each of the documents from our news category, which we can easily map back to the fileids of the pickled documents:

Document 'news/56d62554c1808113ffb87492.pickle' assigned to cluster 0.
Document 'news/56d6255dc1808113ffb874f0.pickle' assigned to cluster 5.
Document 'news/56d62570c1808113ffb87557.pickle' assigned to cluster 4.
Document 'news/56d625abc1808113ffb87625.pickle' assigned to cluster 2.
Document 'news/56d63a76c1808113ffb8841c.pickle' assigned to cluster 0.
Document 'news/56d63ae1c1808113ffb886b5.pickle' assigned to cluster 3.
Document 'news/56d63af0c1808113ffb88745.pickle' assigned to cluster 5.
Document 'news/56d64c7ac1808115036122b4.pickle' assigned to cluster 6.
Document 'news/56d64cf2c1808115036125f5.pickle' assigned to cluster 2.
Document 'news/56d65c2ec1808116aade2f8a.pickle' assigned to cluster 2.
...

We now have a preliminary model for clustering documents based on text similarity; now we should consider what we can do to optimize our results. However, unlike the case of classification, we don’t have a convenient measure to tell us if the “news” corpus has been correctly partitioned into subcategories—when it comes to unsupervised learning, we don’t have a ground truth. Instead, generally we would look to some human validation to see if our clusters make sense—are they meaningfully distinct? Are they sufficiently focused? In the next section, we’ll discuss what it might mean to “optimize” a clustering model.

Optimizing k-means

How can we “improve” a clustering model? In our case, this amounts to asking how we can make our results more interpretable and more useful. First, we can often make our results more interpretable by experimenting with different values of k. With k-means clustering, k-selection is often an iterative process; while there are rules of thumb, initial selection is often somewhat arbitrary.

We can also tune other parts of our pipeline; for example, instead of using one-hot encoding, we could switch to TF–IDF vectorization. Alternatively, in place of our TextNormalizer, we could introduce a feature selector that would select only a subset of the entire feature set (e.g., only the 5,000 most common tokens, excluding stopwords).

Keep in mind that k-means is not a lightweight algorithm and can be particularly slow with high-dimensional data such as text. If your clustering pipeline is very slow, you can optimize for speed by switching from the nltk.cluster module to using sklearn.cluster’s MiniBatchKMeans implementation. MiniBatchKMeans is a k-means variant that uses randomly sampled subsets (or “mini-batches”) of the entire training dataset to optimize the same objective function, but with a much-reduced computation time.

from sklearn.cluster import MiniBatchKMeans
from sklearn.base import BaseEstimator, TransformerMixin


class KMeansClusters(BaseEstimator, TransformerMixin):

    def __init__(self, k=7):
        self.k = k
        self.model = MiniBatchKMeans(self.k)

    def fit(self, documents, labels=None):
        return self

    def transform(self, documents):
        return self.model.fit_predict(documents)

The trade-off is that our MiniBatchKMeans implementation, while faster, will use Euclidean distance, which is less effective for text. At the time of this writing, the Scikit-Learn implementations of KMeans and MiniBatchKMeans do not support the use of non-Euclidean distance measures.

Handling uneven geometries

The k-means algorithm makes several naive assumptions about data, presuming that it will be well-distributed, that clusters will have roughly comparable degrees of variance, and that they will be fundamentally spherical in nature. As such, there are many cases where k-means clustering will be unsuccessful; for instance, when outlier data points interfere with cluster coherence and when clusters have markedly different, or nonspherical densities. Using alternative distance measures, such as cosine or Mahalanobis distance, can help to address these cases.

Ultimately, the advantages of using k-means make it an important tool in the natural language processing toolkit; k-means offers conceptual simplicity, producing tight, spherical clusters, convenient centroids that support model interpretability, and the guarantee of eventual convergence. In the rest of this chapter, we’ll explore some of the other, more sophisticated techniques we have found useful for text, but there’s no silver bullet, and a simple k-means approach is often a convenient place to start.

Agglomerative clustering

Agglomerative clustering iteratively combines the closest instances into clusters until all the instances belong to a single group. In the context of text data, the result is a hierarchy of variable-sized groups that describe document similarities at different levels or granularities.

We can switch our clustering implementation to an agglomerative approach fairly easily. We first define a HierarchicalClusters class, which initializes a Scikit-Learn AgglomerativeClustering model.

from sklearn.cluster import AgglomerativeClustering


class HierarchicalClusters(object):

    def __init__(self):
        self.model = AgglomerativeClustering()

We add our no-op fit() method and a tranform() method, which calls the internal model’s fit_predict method, saving the resulting children and labels attributes for later use and returning the clusters.

...

    def fit(self, documents, labels=None):
        return self

    def transform(self, documents):
        """
        Fits the agglomerative model to the given data.
        """
        clusters = self.model.fit_predict(documents)
        self.labels = self.model.labels_
        self.children = self.model.children_

        return clusters

Next, we’ll put the pieces together into a Pipeline and inspect the cluster labels and membership of each of the children of each nonleaf node.

model = Pipeline([
    ('norm', TextNormalizer()),
    ('vect', OneHotVectorizer()),
    ('clusters', HierarchicalClusters())
])

model.fit_transform(docs)
labels = model.named_steps['clusters'].labels
pickles = list(corpus.fileids(categories=['news']))

for idx, fileid in enumerate(pickles):
    print("Document '{}' assigned to cluster {}.".format(fileid,labels[idx]))

The results appear as follows:

Document 'news/56d62554c1808113ffb87492.pickle' assigned to cluster 1.
Document 'news/56d6255dc1808113ffb874f0.pickle' assigned to cluster 0.
Document 'news/56d62570c1808113ffb87557.pickle' assigned to cluster 1.
Document 'news/56d625abc1808113ffb87625.pickle' assigned to cluster 1.
Document 'news/56d63a76c1808113ffb8841c.pickle' assigned to cluster 1.
Document 'news/56d63ae1c1808113ffb886b5.pickle' assigned to cluster 0.
Document 'news/56d63af0c1808113ffb88745.pickle' assigned to cluster 1.
Document 'news/56d64c7ac1808115036122b4.pickle' assigned to cluster 1.
Document 'news/56d64cf2c1808115036125f5.pickle' assigned to cluster 0.
Document 'news/56d65c2ec1808116aade2f8a.pickle' assigned to cluster 0.
...

One of the challenges with agglomerative clustering is that we don’t have the benefit of centroids to use in labeling our document clusters as we did with the k-means example. Therefore, to enable us to visually explore the resultant clusters of our AgglomerativeClustering model, we will define a method plot_dendrogram to create a visual representation.

Our plot_dendrogram method will use the dendrogram method from SciPy and will also require Matplotlib’s pyplot. We use NumPy to compute a distance range between the child leaf nodes, and make an equal range of values to represent each child position. We then create a linkage matrix to hold the positions between each child and their distances. Finally we use SciPy’s dendrogram method, passing in the linkage matrix and any keyword arguments that can later be passed in to modify the figure.

import numpy as np
from matplotlib import pyplot as plt
from scipy.cluster.hierarchy import dendrogram

def plot_dendrogram(children, **kwargs):
    # Distances between each pair of children
    distance = position = np.arange(children.shape[0])

    # Create linkage matrix and then plot the dendrogram
    linkage_matrix = np.column_stack([
        children, distance, position]
    ).astype(float)

    # Plot the corresponding dendrogram
    fig, ax = plt.subplots(figsize=(10, 5))  # set size
    ax = dendrogram(linkage_matrix, **kwargs)
    plt.tick_params(axis='x', bottom='off', top='off', labelbottom='off')
    plt.tight_layout()
    plt.show()

children = model.named_steps['clusters'].children
plot_dendrogram(children)

We can see the resulting plot in Figure 6-9. The first documents to be aggregated into the same cluster, those with the shortest branches, are the ones that exhibited the least variations. Those with the longest branches illustrate those with more variance, which were clustered later in the process.

Just as there are multiple ways of quantifying the difference between any two documents, there are also multiple criteria for establishing the linkages between them. Agglomerative clustering requires both a distance function and a linkage criterion. Scikit-Learn’s implementation defaults to the Ward criterion, which minimizes the within-cluster variance as each are successively merged. At each aggregation step, the algorithm finds the pair of clusters that contributes the least increase in total within-cluster variance after merging.

Figure 6-9. Dendrogram plot

As we can see from the results of our KMeansClusters and HierarchicalClusters, one of the challenges with both partitive and agglomerative clustering is that they don’t give much insight into why a document ended up in a particular cluster. In the next section, we’ll explore a different set of methods that expose strategies we can leverage not only to rapidly group our documents, but also to effectively describe their contents.

In Scikit-Learn

We begin by creating a class, SklearnTopicModels. The __init__ function instantiates a pipeline with our TextNormalizer, CountVectorizer, and Scikit-Learn’s implementation of LatentDirichletAllocation. We must specify a number of topics (here 50), just as we did with k-means clustering.

from sklearn.pipeline import Pipeline
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

class SklearnTopicModels(object):

    def __init__(self, n_topics=50):
        """
        n_topics is the desired number of topics
        """
        self.n_topics = n_topics
        self.model = Pipeline([
            ('norm', TextNormalizer()),
            ('vect', CountVectorizer(tokenizer=identity,
                                     preprocessor=None, lowercase=False)),
            ('model', LatentDirichletAllocation(n_topics=self.n_topics)),
        ])

We then create a fit_transform method, which will call the internal fit and transform methods of each step of our pipeline:

    def fit_transform(self, documents):
        self.model.fit_transform(documents)

        return self.model

Now that we have a way to create and fit the pipeline, we want some mechanism to inspect our topics. The topics aren’t labeled, and we don’t have a centroid with which to produce a label as we would with centroidal clustering. Instead, we will inspect each topic in terms of the words it has the highest probability of generating.

We create a get_topics method, which steps through our pipeline object to retrieve the fitted vectorizer and extracts the tokens from its get_feature_names() attribute. We loop through the components_ attribute of the LDA model, and for each of the topics and its corresponding index, we reverse-sort the numbered tokens by weight such that the 25 highest weighted terms are ranked first. We then retrieve the corresponding tokens from the feature names and store our topics as a dictionary where the key is the index of one of the 50 topics and the values are the top words associated with that topic.

    def get_topics(self, n=25):
        """
        n is the number of top terms to show for each topic
        """
        vectorizer = self.model.named_steps['vect']
        model = self.model.steps[-1][1]
        names = vectorizer.get_feature_names()
        topics = dict()

        for idx, topic in enumerate(model.components_):
            features = topic.argsort()[:-(n - 1): -1]
            tokens = [names[i] for i in features]
            topics[idx] = tokens

        return topics

We can now instantiate a SklearnTopicModels object, and fit and transform the pipeline on our corpus documents. We assign the result of the get_topics() attribute (a Python dictionary) to a topics variable and unpack the dictionary, printing out the corresponding topics and their most informative terms:

if __name__ == '__main__':
    corpus = PickledCorpusReader('corpus/')

    lda       = SklearnTopicModels()
    documents = corpus.docs()

    lda.fit_transform(documents)
    topics = lda.get_topics()
    for topic, terms in topics.items():
        print("Topic #{}:".format(topic+1))
        print(terms)

The results appear as follows:

Topic #1:
['science', 'scientist', 'data', 'daviau', 'human', 'earth', 'bayesian',
'method', 'scientific', 'jableh', 'probability', 'inference', 'crater',
'transhumanism', 'sequence', 'python', 'engineer', 'conscience',
'attitude', 'layer', 'pee', 'probabilistic', 'radio']
Topic #2:
['franchise', 'rhoden', 'rosemary', 'allergy', 'dewine', 'microwave',
'charleston', 'q', 'pike', 'relmicro', '($', 'wicket', 'infant',
't20', 'piketon', 'points', 'mug', 'snakeskin', 'skinnytaste',
'frankie', 'uninitiated', 'spirit', 'kosher']
Topic #3:
['cosby', 'vehicle', 'moon', 'tesla', 'module', 'mission', 'hastert',
'air', 'mars', 'spacex', 'kazakhstan', 'accuser', 'earth', 'makemake',
'dragon', 'model', 'input', 'musk', 'recall', 'buffon', 'stage',
'journey', 'capsule']
...

The Gensim way

Gensim also exposes an implementation for Latent Dirichlet Allocation, which offers some convenient attributes over Scikit-Learn. Conveniently, Gensim (starting with version 2.2.0) provides a wrapper for its LDAModel, called ldamodel.LdaTransformer, which makes integration with a Scikit-Learn pipeline that much more convenient.

To use Gensim’s LdaTransformer, we need to create a custom Scikit-Learn wrapper for Gensim’s TfidfVectorizer so that it can function inside a Scikit-Learn Pipeline. GensimTfidfVectorizer will vectorize our documents ahead of LDA, as well as saving, holding, and loading a custom-fitted lexicon and vectorizer for later use.

class GensimTfidfVectorizer(BaseEstimator, TransformerMixin):

    def __init__(self, dirpath=".", tofull=False):
        """
        Pass in a directory that holds the lexicon in corpus.dict and the
        TF-IDF model in tfidf.model.

        Set tofull = True if the next thing is a Scikit-Learn estimator
        otherwise keep False if the next thing is a Gensim model.
        """
        self._lexicon_path = os.path.join(dirpath, "corpus.dict")
        self._tfidf_path = os.path.join(dirpath, "tfidf.model")

        self.lexicon = None
        self.tfidf = None
        self.tofull = tofull

        self.load()

    def load(self):
        if os.path.exists(self._lexicon_path):
            self.lexicon = Dictionary.load(self._lexicon_path)

        if os.path.exists(self._tfidf_path):
            self.tfidf = TfidfModel().load(self._tfidf_path)

    def save(self):
        self.lexicon.save(self._lexicon_path)
        self.tfidf.save(self._tfidf_path)

If the model has already been fit, we can initialize the GensimTfidfVectorizer with a lexicon and vectorizer that can be loaded from disk using the load method. We also implement a save() method, which we will call after fitting the vectorizer.

Next, we implement fit() by creating a Gensim Dictionary object, which takes as an argument a list of normalized documents. We instantiate a Gensim TfidfModel, passing in as an argument the list of documents, each of which have been passed through lexicon.doc2bow, and been transformed into bags of words. We then call the save method, which serializes our lexicon and vectorizer and saves them to disk. Finally, the fit() method returns self to conform with the Scikit-Learn API.

    def fit(self, documents, labels=None):
        self.lexicon = Dictionary(documents)
        self.tfidf = TfidfModel([
            self.lexicon.doc2bow(doc)
            for doc in documents],
            id2word=self.lexicon)
        self.save()
        return self

We then implement our transform() method, which creates a generator that loops through each of our normalized documents and vectorizes them using the fitted model and their bag-of-words representation. Because the next step in our pipeline will be a Gensim model, we initialized our vectorizer to set tofull=False, so that it would output a sparse document format (a sequence of 2-tuples). However, if we were going to use a Scikit-Learn estimator next, we would want to initialize our GensimTfidfVectorizer with tofull=True, which here in our transform method would convert the sparse format into the needed dense representation for Scikit-Learn, an np array.

    def transform(self, documents):
        def generator():
            for document in documents:
                vec = self.tfidf[self.lexicon.doc2bow(document)]
                if self.tofull:
                    yield sparse2full(vec)
                else:
                    yield vec
        return list(generator())

We now have a custom wrapper for our Gensim vectorizer, and here in GensimTopicModels, we put all of the pieces together:

from sklearn.pipeline import Pipeline
from gensim.sklearn_api import ldamodel


class GensimTopicModels(object):

    def __init__(self, n_topics=50):
        """
        n_topics is the desired number of topics
        """
        self.n_topics = n_topics
        self.model = Pipeline([
            ('norm', TextNormalizer()),
            ('vect', GensimTfidfVectorizer()),
            ('model', ldamodel.LdaTransformer(num_topics = self.n_topics))
        ])

    def fit(self, documents):
        self.model.fit(documents)

        return self.model

We can now fit our pipeline with our corpus.docs:

if __name__ == '__main__':
    corpus = PickledCorpusReader('../corpus')

    gensim_lda = GensimTopicModels()

    docs = [
        list(corpus.docs(fileids=fileid))[0]
        for fileid in corpus.fileids()
    ]

    gensim_lda.fit(docs)

In order to inspect the topics, we can retrieve them from the LDA step, which is the gensim_model attribute from the last step of our pipeline. We can then use the Gensim LDAModel show_topics method to view the topics and the token-weights for the top ten most influential tokens:

    lda = gensim_lda.model.named_steps['model'].gensim_model
    print(lda.show_topics())

We can also define a function get_topics, which given the fitted LDAModel and vectorized corpus, will retrieve the highest-weighted topic for each of the documents in the corpus:

    def get_topics(vectorized_corpus, model):
        from operator import itemgetter

        topics = [
            max(model[doc], key=itemgetter(1))[0]
            for doc in vectorized_corpus
        ]

        return topics

    lda = gensim_lda.model.named_steps['model'].gensim_model

    corpus = [
        gensim_lda.model.named_steps['vect'].lexicon.doc2bow(doc)
        for doc in gensim_lda.model.named_steps['norm'].transform(docs)
    ]

    topics = get_topics(corpus,lda)

    for topic, doc in zip(topics, docs):
        print("Topic:{}".format(topic))
        print(doc)

Visualizing topics

Oftentimes with unsupervised learning techniques, it is helpful to be able to visually explore the results of a model, since traditional model evaluation techniques are useful only for supervised learning problems. Visualization techniques for text analytics will be discussed in greater detail in Chapter 8, but here we will briefly explore the use of the pyLDAvis library, which is designed to provide a visual interface for interpreting the topics derived from a topic model.

PyLDAvis works by extracting information from fitted LDA topic models to inform an interactive web-based visualization, which can easily be run from inside a Jupyter notebook or saved as HTML. In order visualize our document topics with pyLDAvis, we can fit our pipeline inside a Jupyter notebook as follows:

import pyLDAvis
import pyLDAvis.gensim

lda = gensim_lda.model.named_steps['model'].gensim_model

corpus = [
    gensim_lda.model.named_steps['vect'].lexicon.doc2bow(doc)
    for doc in gensim_lda.model.named_steps['norm'].transform(docs)
]

lexicon = gensim_lda.model.named_steps['vect'].lexicon

data = pyLDAvis.gensim.prepare(model,corpus,lexicon)
pyLDAvis.display(data)

The key method pyLDAvis.gensim.prepare takes as an argument the LDA model, the vectorized corpus, and the derived lexicon and produces, upon calling display, visualizations like the one shown in Figure 6-11.

Figure 6-11. Interactive topic model visualization with pyLDAvis

In Scikit-Learn

To do Latent Semantic Analysis with Scikit-Learn, we will make a pipeline that normalizes our text, creates a term-document matrix using a CountVectorizer, and then employs TruncatedSVD, which is the Scikit-Learn implementation of Singular Value Decomposition. Scikit-Learn’s implementation only computes the k largest singular values, where k is a hyperparameter that we must specify via the n_components attribute. Fortunately, this requires very little refactoring of the __init__ method we created for our SklearnTopicModels class!

class SklearnTopicModels(object):

    def __init__(self, n_topics=50, estimator='LDA'):
        """
        n_topics is the desired number of topics
        To use Latent Semantic Analysis, set estimator to 'LSA',
        otherwise, defaults to Latent Dirichlet Allocation ('LDA').
        """
        self.n_topics = n_topics

        if estimator == 'LSA':
            self.estimator = TruncatedSVD(n_components=self.n_topics)
        else:
            self.estimator = LatentDirichletAllocation(n_topics=self.n_topics)

        self.model = Pipeline([
            ('norm', TextNormalizer()),
            ('tfidf', CountVectorizer(tokenizer=identity,
                                      preprocessor=None, lowercase=False)),
            ('model', self.estimator)
        ])

Our original fit_transform and get_topics methods that we defined for our Scikit-Learn implementation of the Latent Dirichlet Allocation pipeline will not require any modification to work with our newly refactored SklearnTopicModels class, so we can easily switch between the two algorithms to see which performs best according to our application and document context.

The Gensim way

Updating our GensimTopicModels class to enable a Latent Semantic Analysis pipeline is much the same. We again use our TextNormalizer, followed by the GensimTfidfVectorizer we used in our Latent Dirichlet Allocation pipeline, and finally the Gensim LsiModel wrapper exposed through the gensim.sklearn_api module, lsimodel.LsiTransformer:

from gensim.sklearn_api import lsimodel, ldamodel


class GensimTopicModels(object):

    def __init__(self, n_topics=50, estimator='LDA'):
        """
        n_topics is the desired number of topics

        To use Latent Semantic Analysis, set estimator to 'LSA'
        otherwise defaults to Latent Dirichlet Allocation.
        """
        self.n_topics = n_topics

        if estimator == 'LSA':
            self.estimator = lsimodel.LsiTransformer(num_topics=self.n_topics)
        else:
            self.estimator = ldamodel.LdaTransformer(num_topics=self.n_topics)

        self.model = Pipeline([
            ('norm', TextNormalizer()),
            ('vect', GensimTfidfVectorizer()),
            ('model', self.estimator)
        ])

We can now switch between the two Gensim algorithms with the estimator keyword argument.

In Scikit-Learn

Scikit-Learn’s implementation of non-negative matrix factorization is the sklearn.decomposition.NMF class. With a minor amount of refactoring to the __init__ method of our SklearnTopicModels class, we can easily implement NNMF in our pipeline without any additional changes needed:

from sklearn.decomposition import NMF


class SklearnTopicModels(object):

    def __init__(self, n_topics=50, estimator='LDA'):
        """
        n_topics is the desired number of topics
        To use Latent Semantic Analysis, set estimator to 'LSA',
        To use Non-Negative Matrix Factorization, set estimator to 'NMF',
        otherwise, defaults to Latent Dirichlet Allocation ('LDA').
        """
        self.n_topics = n_topics

        if estimator == 'LSA':
            self.estimator = TruncatedSVD(n_components=self.n_topics)
        elif estimator == 'NMF':
            self.estimator = NMF(n_components=self.n_topics)
        else:
            self.estimator = LatentDirichletAllocation(n_topics=self.n_topics)

        self.model = Pipeline([
            ('norm', TextNormalizer()),
            ('tfidf', CountVectorizer(tokenizer=identity,
                                      preprocessor=None, lowercase=False)),
            ('model', self.estimator)
        ])

So which topic modeling algorithm is best? Anecdotally,⁵ LSA is sometimes considered better for learning descriptive topics, which is helpful with longer documents and more diffuse corpora. Latent Dirichlet Allocation and non-negative matrix factorization, on the other hand, can be better for learning compact topics, which is useful for creating succinct labels from topics.

Ultimately, the best model will depend a great deal on the corpus you are working with and the goals of your application. Now you are equipped with the tools to experiment with multiple models to determine which works best for your use case!