The Problem with Traditional NN

In the scenario just mentioned, a traditional neural network would highlight the image as a 1 only if the pixels around the middle are highlighted and the rest of the pixels in the image are not highlighted (since most people have highlighted the pixels in the middle).

To better understand this problem, let’s go through the code we went through in Chapter 7 (code available as “issue with traditional NN.ipynb” in github):

1. 1.

   Download the dataset and extract the train and test datasets:

   from keras.datasets import mnist

   import matplotlib.pyplot as plt

   %matplotlib inline

   # load (downloaded if needed) the MNIST dataset

   (X_train, y_train), (X_test, y_test) = mnist.load_data()

   # plot 4 images as gray scale

   plt.subplot(221)

   plt.imshow(X_train[0], cmap=plt.get_cmap('gray'))

   plt.subplot(222)

   plt.imshow(X_train[1], cmap=plt.get_cmap('gray'))

   plt.subplot(223)

   plt.imshow(X_train[2], cmap=plt.get_cmap('gray'))

   plt.subplot(224)

   plt.imshow(X_train[3], cmap=plt.get_cmap('gray'))

   # show the plot

   plt.show()

   

    
2. 2.

   Import the relevant packages:

   import numpy as np

   from keras.datasets import mnist

   from keras.models import Sequential

   from keras.layers import Dense

   from keras.layers import Dropout

   from keras.layers import Flatten

   from keras.layers.convolutional import Conv2D

   from keras.layers.convolutional import MaxPooling2D

   from keras.utils import np_utils

   from keras import backend as K
    
3. 3.

   Fetch the training set corresponding to the label 1 only:

   X_train1 = X_train[y_train==1]
    
4. 4.

   Reshape and normalize the dataset:

   num_pixels = X_train.shape[1] * X_train.shape[2]

   X_train = X_train.reshape(X_train.shape[0],num_pixels ).astype('float32')

   X_test = X_test.reshape(X_test.shape[0],num_pixels).astype('float32')

   X_train = X_train / 255

   X_test = X_test / 255
    
5. 5.

   One-hot-encode the labels:

   y_train = np_utils.to_categorical(y_train)

   y_test = np_utils.to_categorical(y_test)

   num_classes = y_train.shape[1]
    
6. 6.

   Build a model and run it:

   model = Sequential()

   model.add(Dense(1000, input_dim=num_pixels, activation="relu"))

   model.add(Dense(num_classes, activation="softmax"))

   model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=[''accuracy'])

   model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5, batch_size=1024, verbose=1)
    

Let’s plot what an average 1 label looks like:

Figure 9-2 shows the result.

Scenario 1

In this scenario, a new image is created (Figure 9-3) in which the original image is translated by 1 pixel toward the left:

for i in range(pic.shape[0]):

  if i<20:

    pic[:,i]=pic[:,i+1]

plt.imshow(pic)

Let’s go ahead and predict the label of the image in Figure 9-3 using the built model:

model.predict(pic.reshape(1,784))

We see the wrong prediction of 8 as output.

Scenario 2

A new image is created (Figure 9-4) in which the pixels are not translated from the original average 1 image:

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

plt.imshow(pic)

The prediction of this image is as follows:

model.predict(pic.reshape(1,784))

We see a correct prediction of 1 as output.

Scenario 3

A new image is created (Figure 9-5) in which the pixels of the original average 1 image are shifted by 1 pixel to the right:

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

pic2=np.copy(pic)

for i in range(pic.shape[0]):

  if ((i>6) and (i<26)):

    pic[:,i]=pic2[:,(i-1)]

plt.imshow(pic)

Let us go ahead and predict the label of the above image using the built model:

model.predict(pic.reshape(1,784))

We have a correct prediction of 1 as output.

Scenario 4

A new image is created (Figure 9-6) in which the pixels of the original average 1 image are shifted by 2 pixels to the right:

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

pic2=np.copy(pic)

for i in range(pic.shape[0]):

  if ((i>6) and (i<26)):

    pic[:,i]=pic2[:,(i-2)]

plt.imshow(pic)

We’ll predict the label of the image using the built model:

model.predict(pic.reshape(1,784))

And we see a wrong prediction of 3 as output.

From the preceding scenarios, you can see that traditional NN fails to produce good results the moment there is translation in the data. These scenarios call for a different way of dealing with the network to address translation variance. And this is where a convolutional neural network (CNN) comes in handy.

Understanding the Convolutional in CNN

You already have a good idea of how a typical neural network works. In this section, let’s explore what the word convolutional means in CNN. A convolution is a multiplication between two matrices, with one matrix being big and the other smaller.

To see convolution, consider the following example.

Matrix A is as follows:

Matrix B is as follows:

The result of the preceding steps would be a matrix, as follows:

Conventionally, the smaller matrix is called a filter or kernel, and the smaller matrix values are arrived at statistically through gradient descent (more on gradient descent a little later). The values within the filter can be considered as the constituent weights.

From Convolution Activation to Pooling

So far, we have looked at how convolutions work. In this section, we will consider the typical next step after a convolution: pooling.

Let’s say the output of the convolution step is as follows (we are not considering the preceding example—this is a new example to illustrate pooling, and the rationale will be explained in a later section):

In this case, the output of a convolution step is a 2 × 2 matrix. Max pooling considers the 2 × 2 block and gives the maximum value as output—similarly if the output of a convolution step is a bigger matrix, as follows:

Max pooling divides the big matrix into non-overlapping blocks of size 2 × 2 each, as follows:

From each block, only the element that has the highest value is chosen. So, the output of the max pooling operation on the preceding matrix would be the following:

Note that, in practice, it is not necessary to always have a 2 × 2 filter.

The other types of pooling involved are sum and average. Again, in practice we see a lot of max pooling when compared to other types of pooling.

Creating CNNs with Code

From the preceding traditional NN scenario, we saw that a NN does not work if the pixels are translated by 1 unit to the left. Practically, we can consider the convolution step as identifying the pattern and pooling step as the one that results in translation variance.

For the preceding convolution, where one convolution is followed by one pooling layer, the output prediction works out well if the pixels are translated by 1 unit to the left or right, but does not work when the pixels are translated by more than 1 unit (Figure 9-7):

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

for i in range(pic.shape[0]):

  if i<20:

    pic[:,i]=pic[:,i+1]

plt.imshow(pic)

Let’s go ahead and predict the label of Figure 9-7:

model.predict(pic.reshape(1,28,28,1))

We see a correct prediction of 1 as output.

In the next scenario (Figure 9-8), we move the pixels by 2 units to the left:

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

for i in range(pic.shape[0]):

  if i<20:

    pic[:,i]=pic[:,i+2]

plt.imshow(pic)

Let’s predict the label of Figure 9-8 per the CNN model we built earlier:

model.predict(pic.reshape(1,28,28,1))

We have an incorrect prediction when the image is translated by 2 pixels to the left.

Note that when the number of convolution pooling layers in the model is the same as the amount of translation in an image, the prediction is correct. But prediction is more likely to be incorrect if there are less convolution pooling layers when compared to the translation in image.

Working Details of CNN

The various layers of the preceding model are as follows:

model.layers

The name and shape corresponding to various layers are as follows:

names = [weight.name for layer in model.layers for weight in layer.weights]

weights = model.get_weights()

for name, weight in zip(names, weights):

    print(name, weight.shape)

The weights corresponding to a given layer can be extracted as follows:

model.layers[0].get_weights()

The prediction for the first input can be calculated as follows:

model.predict(X_train[0].reshape(1,4,4,1))

Now that we know the probability of 0 for the preceding prediction is 0.89066, let’s validate our intuition of CNN so far by matching the preceding prediction in Excel (available as “CNN simple example.xlsx” in github).

The first input and its corresponding scaled version, along with convolution weights and bias (that came out from the model), are as follows:

The output of convolution is as follows (please check cells L4 to M5 in the ‘CNN simple example.xlsx’ file):

The calculation of convolution is per the following formula:

After the convolution layer, we perform the max pooling as follows:

Once the pooling is performed, all the outputs are flattened (per the specification in our model). However, given that our pooling layer has only one output, flattening would also result in a single output.

In the next step, the flattened layer is connected to the hidden dense layer (which in our model specification has ten neurons). The weights and bias corresponding to each neuron are as follows:

The matrix multiplication and the ReLU activation after the multiplication would be as follows:

The formulas for the preceding output are as follows:

Now let’s look at the calculations from hidden layer to output layer. Note that there are two outputs given for each input (output for every row has two columns in dimension: probability of 0 and probability of 1). The weights from hidden layer to output layer are as follows:

Now that each neuron is connected to two weights (where each weight gives its connection to the two outputs), let’s look at the calculation from hidden to output layer:

The calculation of the output layer is as follows:

Now that we have some output values, let’s calculate the softmax part of the output:

The output would now be exactly the same as what we saw in the output from the keras model:

Thus, we have a validation about the intuition laid out in the previous sections.

Deep Diving into Convolutions/Kernels

To see how kernels/filters help, let’s go through another scenario. From the MNIST dataset, let’s modify the objective in such a way that we are only interested in predicting whether an image is a 1 or not a 1:

We will come up with a simple CNN where there are only two convolution filters:

model = Sequential()

model.add(Conv2D(2, (3,3), input_shape=(28, 28,1), activation="relu"))

model.add(Flatten())

model.add(Dense(1000, activation="relu"))

model.add(Dense(num_classes, activation="softmax"))

model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=['accuracy'])

model.summary()

Now we’ll go ahead and run the model as follows:

model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5, batch_size=1024, verbose=1)

We can extract the weights corresponding to the filters in the following way:

Let’s manually convolve and apply the activation by using the weights derived in the preceding step (Figure 9-9):

from scipy import signal

from scipy import misc

import numpy as np

import pylab

for j in range(2):

    gradd=np.zeros((30,30))

    for i in range(6000):

        grad = signal.convolve2d(X_train1[i,:,:,0], model.layers[0].get_weights()[0].T[j][0])+model.layers[0].get_weights()[1][j]

        grad = np.where(grad<0,0,grad)

        gradd=grad+gradd

    grad2=np.where(gradd<0,0,gradd)

    pylab.imshow(grad2/6000)

    pylab.gray()

    pylab.show()

In the figure, note that the filter on the left activates a 1 image a lot more than the filter on the right. Essentially, the first filter helps in predicting label 1 more, and the second filter augments in predicting the rest.

From Convolution and Pooling to Flattening: Fully Connected Layer

The outputs we have seen so far until pooling layer are images . In traditional neural network, we would consider each pixel as an independent variable. This is precisely what we are going to perform in the flattening process .

Each pixel of the image is unrolled, and so the process is called flattening. For example, the output image after convolution and pooling looks like this:

The output of flattening looks like this:

Connecting the Dots: Feed Forward Network

A typical CNN looks is shown in Figure 9-10 (the most famous—the one developed by the inventor himself, LeNet, as an example):

The subsample written in Figure 9-10 is equivalent to the max pooling step we saw earlier.

Other Details of CNN

In Figure 9-10, we see that the conv1 step has six channels or convolutions of the original image. Let’s look at this in detail:

1. 1.

   Let’s say we have a greyscale image that is 28 × 28 in dimension. Six filters that are 3 × 3 in size would generate images that are 26 × 26 in size. Thus, we are left with six images of size 26 × 26.

    
2. 2.

   A typical color image would have three channels (RGB). For simplicity, we can assume that the output image we had in step 1 has six channels - one each for the six filters (though we can’t name them as RGB like the three-channel version). In this step, we would perform max pooling on each of the six channels separately. This would result in six images (channels) that are 13 × 13 in dimension.

    
3. 3.

   In the next convolution step, we multiply the six channels of 13 × 13 images with weights of dimensions 3 × 3 × 6. That’s a 3-dimensional weight matrix convolving over a 3-dimensional image (where the image has dimensions 13 × 13 × 6). This would result in an image of 11 × 11 in dimension for each filter.

   Let’s say we’ve considered ten different weight matrices (cubes, to be precise). This would result in an image that is 11 × 11 × 10 in dimension.

    
4. 4.

   Max pooling on each of the 11 × 11 images (which are ten in number) would result in a 5 × 5 image. Note that, when the max pooling is performed on an image that has odd number of dimensions, pooling gives us the rounded-down image—that is, 11/2 is rounded down to 5.

    

A stride is the amount by which the filter that convolves over the original image moves from one step to the next step. For example, if the stride value is 2, the distance between 2 consecutive convolutions is 2 pixels. When the stride value is 2, the multiplication would happen as follows, where A is the bigger matrix and B is the filter:

The first convolution would be between:

The second convolution would be between:

The third convolution would be between:

The final convolution would be between:

Note that the output of the convolution is a 2 × 2 matrix when the stride is 2 for the matrices of the given dimensions here.

Backward Propagation in CNN

Backward propagation in CNN is done in similarly to a typical NN, where the impact of changing a weight by a small amount on the overall weight is calculated. But in place of weights, as in NN, we have filters/matrices of weights that need to be updated to minimize the overall loss.

Sometimes, given that there are typically millions of parameters in a CNN, having regularization can be helpful. Regularization in CNN can be achieved using the dropout method or the L1 and L2 regularizations. Dropout is done by choosing not to update some weights (typically a randomly chosen 20% of total weights) and training the entire network over the whole number of epochs .

Putting It All Together

The following code implements a three-convolution pooling layer followed by flattening and a fully connected layer:

In the next step, we build the model, as follows:

model = Sequential()

model.add(Conv2D(32, (3,3), input_shape=(28, 28,1), activation="relu"))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3,3), activation="relu"))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())

model.add(Dense(1000, activation="relu"))

model.add(Dense(num_classes, activation="softmax"))

model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=['accuracy'])

model.summary()

Finally, we fit the model, as follows:

Note that the accuracy of the model trained using the preceding code is ~98.8%. But note that although this model works best on the test dataset, an image that is translated or rotated from the test MNIST dataset would not be classified correctly (In general, CNN could only help when the image is translated by the number of convolution pooling layers). That can be verified by looking at the prediction when the average 1 image is translated by 2 pixels to the left once, and in another scenario, 3 pixels to the left, as follows:

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:,0]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

for i in range(pic.shape[0]):

  if i<20:

    pic[:,i]=pic[:,i+2]

model.predict(pic.reshape(1,28,28,1))

Note that, in this case, where the image is translated by 2 units to the left, the predictions are accurate:

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:,0]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

for i in range(pic.shape[0]):

  if i<20:

    pic[:,i]=pic[:,i+3]

model.predict(pic.reshape(1,28,28,1))

Note that here, when the image is translated by more pixels than convolution pooling layers, the prediction is not accurate. This issue is solved by using data augmentation, the topic of the next section.

Data Augmentation

Technically, a translated image is the same as a new image that is generated from the original image . New data can be generated by using the ImageDataGenerator function in keras:

From that code, we have generated 50,000 random shufflings from our original data , where the pixels are shuffled by 20%.

As we plot the image of 1 now (Figure 9-11), note that there is a wider spread for the image:

y_train1=np.argmax(y_train,axis=1)

X_train1=X_train[y_train1==1]

pic=np.zeros((28,28))

pic2=np.copy(pic)

for i in range(X_train1.shape[0]):

  pic2=X_train1[i,:,:,0]

  pic=pic+pic2

pic=(pic/X_train1.shape[0])

plt.imshow(pic)

Now the predictions will work even when we don’t do convolution pooling for the few pixels that are to the left or right of center. However, for the pixels that are far away from the center, correct predictions will come once the model is built using the convolution and pooling layers.

So, data augmentation helps in further generalizing for variations of the image across the image boundaries when using the CNN model, even with fewer convolution pooling layers.

Implementing CNN in R

To implement CNN in R, we will leverage the same package we used to implement neural network in R—kerasR (code available as “kerasr_cnn_code.r” in github):

The preceding code results in an accuracy of ~97%.

Summary

In this chapter, we saw how convolutions help us identify the structure of interest and how pooling helps ensure that the image is recognized even when translation happens in the original image. Given that CNN is able to adapt to image translation through convolution and pooling, it’s in a position to give better results than the traditional neural network.