Build the Components of the RBM Class

Like we did in Chapter 10, let’s walk through the various components of the RBM class.

First, we will initialize the class with a few parameters; these are the input size of the RBM, the output size, the learning rate, the number of epochs to train for, and the batch size during the training process.

We will also create zero matrices for the weight matrix, the hidden bias vector, and the visible bias vector.

# Define RBM class
class RBM(object):

    def __init__(self, input_size, output_size,
                 learning_rate, epochs, batchsize):
        # Define hyperparameters
        self._input_size = input_size
        self._output_size = output_size
        self.learning_rate = learning_rate
        self.epochs = epochs
        self.batchsize = batchsize

        # Initialize weights and biases using zero matrices
        self.w = np.zeros([input_size, output_size], "float")
        self.hb = np.zeros([output_size], "float")
        self.vb = np.zeros([input_size], "float")

Next, let’s define functions for the forward pass, the backward pass, and the sampling of data during each of these passes back and forth.

Here is the forward pass, where h is the hidden layer and v is the visible layer.

def prob_h_given_v(self, visible, w, hb):
    return tf.nn.sigmoid(tf.matmul(visible, w) + hb)

Here is the backward pass.

def prob_v_given_h(self, hidden, w, vb):
    return tf.nn.sigmoid(tf.matmul(hidden, tf.transpose(w)) + vb)

Here is the sampling function.

def sample_prob(self, probs):
    return tf.nn.relu(tf.sign(probs - tf.random_uniform(tf.shape(probs))))

Now, we need a function that performs that training. Since we are using TensorFlow, we first need to create placeholders for the TensorFlow graph, which we will use when we feed data into the TensorFlow session.

We will have placeholders for the weights matrix, the hidden bias vector, and the visible bias vector.

We will also need to initialize the values for these three using zeros. And, we will need one set to hold the current values and one set to hold the previous values.

_w = tf.placeholder("float", [self._input_size, self._output_size])
_hb = tf.placeholder("float", [self._output_size])
_vb = tf.placeholder("float", [self._input_size])

prv_w = np.zeros([self._input_size, self._output_size], "float")
prv_hb = np.zeros([self._output_size], "float")
prv_vb = np.zeros([self._input_size], "float")

cur_w = np.zeros([self._input_size, self._output_size], "float")
cur_hb = np.zeros([self._output_size], "float")
cur_vb = np.zeros([self._input_size], "float")

Likewise, we need a placeholder for the visible layer. The hidden layer is derived from matrix multiplication of the visible layer and the weights matrix and the matrix addition of the hidden bias vector.

v0 = tf.placeholder("float", [None, self._input_size])
h0 = self.sample_prob(self.prob_h_given_v(v0, _w, _hb))

During the backward pass, we take the hidden layer output, multiply it with the transpose of the weights matrix used during the forward pass and add the visible bias vector. Note that the weights matrix is the same weights matrix during both the forward and the backward pass.

And, then, we perform the forward pass again.

v1 = self.sample_prob(self.prob_v_given_h(h0, _w, _vb))
h1 = self.prob_h_given_v(v1, _w, _hb)

To update the weights, we perform constrastive divergence, which we introduced in Chapter 10. We also define the error as mean squared error.

positive_grad = tf.matmul(tf.transpose(v0), h0)
negative_grad = tf.matmul(tf.transpose(v1), h1)

update_w = _w + self.learning_rate * \
    (positive_grad - negative_grad) / tf.to_float(tf.shape(v0)[0])
update_vb = _vb +  self.learning_rate * tf.reduce_mean(v0 - v1, 0)
update_hb = _hb +  self.learning_rate * tf.reduce_mean(h0 - h1, 0)

err = tf.reduce_mean(tf.square(v0 - v1))

With this, we are ready to initialize the TensorFlow session with the variables we have just defined.

Once we call sess.run, we can feed in batches of data to begin the training. During the training, forward and backward passes will be made, and the RBM will update weights based on how the generated data compares to the original input. We will print the reconstruction error from each epoch.

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for epoch in range(self.epochs):
        for start, end in zip(range(0, len(X), self.batchsize), \
                range(self.batchsize,len(X), self.batchsize)):
            batch = X[start:end]
            cur_w = sess.run(update_w, \
                feed_dict={v0: batch, _w: prv_w, \
                           _hb: prv_hb, _vb: prv_vb})
            cur_hb = sess.run(update_hb, \
                feed_dict={v0: batch, _w: prv_w, \
                           _hb: prv_hb, _vb: prv_vb})
            cur_vb = sess.run(update_vb, \
                feed_dict={v0: batch, _w: prv_w, \
                           _hb: prv_hb, _vb: prv_vb})
            prv_w = cur_w
            prv_hb = cur_hb
            prv_vb = cur_vb
        error = sess.run(err, feed_dict={v0: X, _w: cur_w, \
                                        _vb: cur_vb, _hb: cur_hb})
        print ('Epoch: %d' % epoch,'reconstruction error: %f' % error)
    self.w = prv_w
    self.hb = prv_hb
    self.vb = prv_vb

Generate Images Using the RBM Model

Let’s also define a function to generate new images from the generative model that the RBM has learned.

def rbm_output(self, X):

    input_X = tf.constant(X)
    _w = tf.constant(self.w)
    _hb = tf.constant(self.hb)
    _vb = tf.constant(self.vb)
    out = tf.nn.sigmoid(tf.matmul(input_X, _w) + _hb)
    hiddenGen = self.sample_prob(self.prob_h_given_v(input_X, _w, _hb))
    visibleGen = self.sample_prob(self.prob_v_given_h(hiddenGen, _w, _vb))
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        return sess.run(out), sess.run(visibleGen), sess.run(hiddenGen)

We feed the original matrix of images, called X, into the function.

We create TensorFlow placeholders for the original matrix of images, the weights matrix, the hidden bias vector, and the visible bias vector.

Then, we push the input matrix to produce the output of a forward pass (out), a sample of the hidden layer (hiddenGen), and a sample of the reconstructed images generated by the model (visibleGen).

View the Intermediate Feature Detectors

Finally, let’s define a function to show the feature detectors of the hidden layer.

def show_features(self, shape, suptitle, count=-1):
    maxw = np.amax(self.w.T)
    minw = np.amin(self.w.T)
    count = self._output_size if count == -1 or count > \
            self._output_size else count
    ncols = count if count < 14 else 14
    nrows = count//ncols
    nrows = nrows if nrows > 2 else 3
    fig = plt.figure(figsize=(ncols, nrows), dpi=100)
    grid = Grid(fig, rect=111, nrows_ncols=(nrows, ncols), axes_pad=0.01)

    for i, ax in enumerate(grid):
        x = self.w.T[i] if i<self._input_size else np.zeros(shape)
        x = (x.reshape(1, -1) - minw)/maxw
        ax.imshow(x.reshape(*shape), cmap=mpl.cm.Greys)
        ax.set_axis_off()

    fig.text(0.5,1, suptitle, fontsize=20, horizontalalignment='center')
    fig.tight_layout()
    plt.show()
    return

We will use this and the other functions on the MNIST dataset now.

Examine Feature Detectors

Now, let’s view the learned features from each of the RBMs using the rbm.show_features function we defined earlier.

rbm_shapes = [(28,28),(25,24),(25,20)]
for i in range(0,len(rbm_list)):
    rbm = rbm_list[i]
    print(rbm.show_features(rbm_shapes[i],
     "RBM learned features from MNIST", 56))

Figure 11-4 displays the learned features for the various RBMs.

Figure 11-4. Learned features of the RBMs

As you can see, each RBM learns increasingly abstract features from the MNIST data. The features of the first RBM vaguely resemble digits, and the features of the second and the third RBMs are increasingly nuanced and less discernible. This is pretty typical of how feature detectors work on image data; the deeper layers of the neural network recognize increasingly abstract elements from the original images.

View Generated Images

Before we build the full DBN, let’s view some of the generated images from one of the RBMs we just trained.

To keep things simple, we will feed the original MNIST training matrix into the first RBM we trained, perform a forward pass and then a backward pass, and this will produce the generated images we need.

We will compare the first ten images of the MNIST dataset with the newly generated images.

inputX = np.array(X_train)
rbmOne = rbm_list[0]

print('RBM 1')
outputX_rbmOne, reconstructedX_rbmOne, hiddenX_rbmOne =
 rbmOne.rbm_output(inputX)
reconstructedX_rbmOne = pd.DataFrame(data=reconstructedX_rbmOne,
 index=X_train.index)
for j in range(0,10):
    example = j
    view_digit(reconstructedX, y_train, example)
    view_digit(X_train, y_train, example)

Figure 11-5 shows the first image produced by the RBM compared to the first original image.

Figure 11-5. First generated image of the first RBM

As you can see, the generated image is similar to the original image — both display the digit five.

Let’s view a few more images like this to compare the RBM generated one with the original ones (see Figures 11-6 through 11-911-9).

Figure 11-6. Second generated image of the first RBM

Figure 11-7. Third generated image of the first RBM

Figure 11-8. Fourth generated image of the first RBM

Figure 11-9. Fifth generated image of the first RBM

These digits are zero, four, one, and nine, respectively, and the generated images look reasonably similar to the original images.

How Training of a DBN Works

Each of the three RBMs we have trained already has its own weights matrix, hidden bias vector, and visible bias vector.

During the training of the fourth RBM as part of the DBN, we will not adjust the weights matrix, hidden bias vector, and visible bias vector of those first three RBMs.

Rather, we will use the first three RBMs as fixed components of the DBN. We will call upon the first three RBMs just to do the forward and backward passes (and use samples of the data these three generate).

During the training of the fourth RBM in the DBN, we will adjust weights and biases of just the fourth RBM. In other words, the fourth RBM in the DBN takes the output of the first three RBMs as given and performs forward and backward passes to learn a generative model that minimizes the reconstruction error between its generated images and the original images.

Another way to train the DBNs would be to allow the DBN to learn and adjust weights for all four RBMs as its performs forward and backward passes through the entire network.

But, training of the DBN would be very computationally expensive (perhaps not so with computers of today but certainly by the standards of 2006, when DBNs were first introduced).

That being said, if we wish to perform more nuanced pre-training, we could allow the weights of the individual RBMs to be adjusted - one RBM at a time - as we perform batches of forward and backward passes through the network. We will not delve into this, but I encourage you to experiment on your own time.

Train the DBN

We will now train the DBN. We set the original image dimensions as 784, the dimensions of the third RBM output as 500, and the desired dimensions of the DBN as 500. We will use a learning rate of 1.0, train for 50 epochs, and use a batch size of 200. Finally, we will call the first three trained RBMs as part of the DBN.

# Instantiate DBN Class
dbn = DBN(784, 500, 500, 1.0, 50, 200, rbm_list[0], rbm_list[1], rbm_list[2])

Now, let’s train.

inputX = np.array(X_train)
error_list = []
error_list = dbn.train(inputX)

Figure 11-10 displays the reconstruction errors of the DBN over the course of the training.

Figure 11-10. Reconstruction errors of the DBN

Figure 11-11 displays the learned features from the last layer of the DBN — the hidden layer of the fourth RBM.

Figure 11-11. Learned features of the fourth RBM in the DBN

Both the reconstruction errors and the learned features look reasonable and similar to the ones from the individual RBMs we analyzed earlier.

Generate Images to Build a Better Image Classifier

To simulate this benefit that unsupervised learning is able to provide, let’s reduce our MNIST training dataset to just five thousand labeled examples. We will store the first five thousand images in a dataframe called inputXReduced.

Then, from these five thousand labeled images, we will generate new images from the generative model we just built using a DBN. And, we will do this 20 times over. In other words, we will generate five thousand new images 20 times to create a dataset that is 100,000 large, all of which will be labeled. Technically, we are storing the final hidden layer outputs not the reconstructed images directly, although we will store the reconstructed images, too, so we can evaluate them soon.

We will store these 100,000 outputs in a NumPy array called generatedImages.

# Generate images and store them
inputXReduced = X_train.loc[:4999]
for i in range(0,20):
    print("Run ",i)
    finalOutput_DBN, reconstructedOutput_DBN = dbn.dbn_output(inputXReduced)
    if i==0:
        generatedImages = finalOutput_DBN
    else:
        generatedImages = np.append(generatedImages, finalOutput_DBN, axis=0)

We will loop through the first five thousand labels from the training labels, called y_train, 20 times to generate an array of labels called labels.

# Generate a vector of labels for the generated images
for i in range(0,20):
    if i==0:
        labels = y_train.loc[:4999]
    else:
        labels = np.append(labels,y_train.loc[:4999])

Finally, we will generate the output on the validation set, which we will need to evaluate the image classifier we build soon.

# Generate images based on the validation set
inputValidation = np.array(X_validation)
finalOutput_DBN_validation, reconstructedOutput_DBN_validation = \
    dbn.dbn_output(inputValidation)

Before we use the data we just generated, let’s view a few of the reconstructed images.

# View reconstructed images
for i in range(0,10):
    example = i
    reconstructedX = pd.DataFrame(data=reconstructedOutput_DBN, \
                                  index=X_train[0:5000].index)
    view_digit(reconstructedX, y_train, example)
    view_digit(X_train, y_train, example)

Figure 11-12. First generated image of the DBN

As you can see in Figure 11-12, the generated image is very similar to the original image — both display the digit five. Unlike the RBM generated images we saw earlier, these are more similar to the original MNIST images, including the pixelated bits.

Let’s view a few more images like this to compared the DBN generated ones with the original MNIST ones (see Figures 11-13 through 11-16).

Figure 11-13. Second generated image of the DBN

Figure 11-14. Third generated image of the DBN

Figure 11-15. Fourth generated image of the DBN

Figure 11-16. Fifth generated image of the DBN

Also, note that the DBN model (as well as the RBM models) is generative and, therefore, the images are produced using a stochastic process. The images are not produced using a deterministic process, and, therefore, the images of a single example vary from one DBN run to another.

To simulate this, we will take the first MNIST image and use the DBN to generate a new one and do this 10 times over.

# Generate the first example 10 times
inputXReduced = X_train.loc[:0]
for i in range(0,10):
    example = 0
    print("Run ",i)
    finalOutput_DBN_fives, reconstructedOutput_DBN_fives = \
        dbn.dbn_output(inputXReduced)
    reconstructedX_fives = pd.DataFrame(data=reconstructedOutput_DBN_fives, \
                                        index=[0])
    print("Generated")
    view_digit(reconstructedX_fives, y_train.loc[:0], example)

As you see from Figures 11-17 through 11-21, all the generated images display the number five, but they vary from image to image even though they all were generated using the same original MNIST image.

Figure 11-17. First two generated images of the digit five

Figure 11-18. Second two generated images of the digit five

Figure 11-19. Third two generated images of the digit five

Figure 11-20. Fourth two generated images of the digit five

Figure 11-21. Fifth two generated images of the digit five

Supervised Only

The first image classifier will rely on just the first five thousand labeled MNIST images. This is the reduced set from the original 50,000 labeled MNIST training set; we designed this to simulate real world problems where we have relatively few labeled examples.

Since we covered gradient boosting and the LightGBM algorithm in depth earlier in the book, we will not go into a lot of details.

Let’s set the parameters for the algorithm.

predictionColumns = ['0','1','2','3','4','5','6','7','8','9']

params_lightGB = {
    'task': 'train',
    'application':'binary',
    'num_class':10,
    'boosting': 'gbdt',
    'objective': 'multiclass',
    'metric': 'multi_logloss',
    'metric_freq':50,
    'is_training_metric':False,
    'max_depth':4,
    'num_leaves': 31,
    'learning_rate': 0.1,
    'feature_fraction': 1.0,
    'bagging_fraction': 1.0,
    'bagging_freq': 0,
    'bagging_seed': 2018,
    'verbose': 0,
    'num_threads':16
}

Next, we will train on the 5,000 labeled MNIST training set - the reduced set - and validate on the 10,000 labeled MNIST validation set.

trainingScore = []
validationScore = []
predictionsLightGBM = pd.DataFrame(data=[], \
                        index=y_validation.index, \
                        columns=predictionColumns)

lgb_train = lgb.Dataset(X_train.loc[:4999], y_train.loc[:4999])
lgb_eval = lgb.Dataset(X_validation, y_validation, reference=lgb_train)
gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
                   valid_sets=lgb_eval, early_stopping_rounds=200)

loglossTraining = log_loss(y_train.loc[:4999], \
    gbm.predict(X_train.loc[:4999], num_iteration=gbm.best_iteration))
trainingScore.append(loglossTraining)

predictionsLightGBM.loc[X_validation.index,predictionColumns] = \
    gbm.predict(X_validation, num_iteration=gbm.best_iteration)
loglossValidation = log_loss(y_validation,
    predictionsLightGBM.loc[X_validation.index,predictionColumns])
validationScore.append(loglossValidation)

print('Training Log Loss: ', loglossTraining)
print('Validation Log Loss: ', loglossValidation)

loglossLightGBM = log_loss(y_validation, predictionsLightGBM)
print('LightGBM Gradient Boosting Log Loss: ', loglossLightGBM)

Figure 11-22 shows the training and the validation log loss from this supervised-only solution.

Figure 11-22. Supervised only log loss

Figure 11-23 shows the overall accuracy of this supervised-only image classification solution.

predictionsLightGBM_firm = np.argmax(np.array(predictionsLightGBM), axis=1)
accuracyValidation_lightGBM = accuracy_score(np.array(y_validation), \
                                            predictionsLightGBM_firm)
print("Supervised-Only Accuracy: ", accuracyValidation_lightGBM)

Figure 11-23. Supervised only accuracy

Unsupervised and Supervised Solution

Now, instead of training on the 5,000 labeled MNIST images, let’s train on the 100,000 generated images from the DBN.

# Prepare DBN-based DataFrames for LightGBM use
generatedImagesDF = pd.DataFrame(data=generatedImages,index=range(0,100000))
labelsDF = pd.DataFrame(data=labels,index=range(0,100000))

X_train_lgb = pd.DataFrame(data=generatedImagesDF,
                           index=generatedImagesDF.index)
X_validation_lgb = pd.DataFrame(data=finalOutput_DBN_validation,
                                index=X_validation.index)

# Train LightGBM
trainingScore = []
validationScore = []
predictionsDBN = pd.DataFrame(data=[],index=y_validation.index,
                              columns=predictionColumns)

lgb_train = lgb.Dataset(X_train_lgb, labels)
lgb_eval = lgb.Dataset(X_validation_lgb, y_validation, reference=lgb_train)
gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
                   valid_sets=lgb_eval, early_stopping_rounds=200)

loglossTraining = log_loss(labelsDF, gbm.predict(X_train_lgb, \
                            num_iteration=gbm.best_iteration))
trainingScore.append(loglossTraining)

predictionsDBN.loc[X_validation.index,predictionColumns] = \
    gbm.predict(X_validation_lgb, num_iteration=gbm.best_iteration)
loglossValidation = log_loss(y_validation,
    predictionsDBN.loc[X_validation.index,predictionColumns])
validationScore.append(loglossValidation)

print('Training Log Loss: ', loglossTraining)
print('Validation Log Loss: ', loglossValidation)

loglossDBN = log_loss(y_validation, predictionsDBN)
print('LightGBM Gradient Boosting Log Loss: ', loglossDBN)

Figure 11-24 displays the log loss of this unsupervised-enchanced image classification solution.

Figure 11-24. Unsupervised plus supervised log loss

Figure 11-25 shows the overall accuracy of this unsupervised-enchanced image classification solution.

Figure 11-25. Unsupervised plus supervised accuracy

As you see, the solution improves by nearly one percentage point, which is considerable.