MNIST Tutorial

Tasks

Setup your development environment
This tutorial uses the Keras package, which relies on numpy, scipy, and tensorflow. You can install them on your computer using the pip3 python package manager, which will also install any other package dependencies.

pip3 install keras tensorflow

Once these packages are installed, move to the next step.

Test your development environment

Create a new python code file, referred to as mnist_train.py. Then add the following import statements.

# load packages
import sys
import keras
import numpy as np
import matplotlib.pyplot as plt

# import specific models from Keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.utils import np_utils

# load the data (MNIST is included in Keras)
from keras.datasets import mnist

# make the results reproducible
rand_seed = 42
np.random.seed(rand_seed)
print("Random seed: %d" % (rand_seed))

Save and run the minst_train.py file to ensure all of the necessary packages are working. It should print out the random seed. By setting the seed to a fixed value, the network you create will be reproducible.

Look at the data

Whenever you start working with data, look at it. The MNIST digits data set are pictures of individual handwritten digits. Use the following code to load the training set and testing sets of the MNIST data set. Each set consists of a numpy array of images and an array of labels. After loading the data set, print out the shape of the arrays and plot some of the training examples using pyplot.

To follow good coding style, put all of the remaining code inside a main function

def main(argv):
    ################# Load and view the data
    # load the data sets
    (xtrain, ytrain), (xtest, ytest) = mnist.load_data()

    print("Training set size:   %d" % (xtrain.shape[0]) )
    print("Test set size:     %d" % (xtest.shape[0]) )

    print("Training set shape :", xtrain.shape)
    print("Training set labels:", ytrain.shape )
    print("Test set shape     :", xtest.shape)
    print("Test set labels    :", ytest.shape)

    # display some of the images, note the dark background
    plt.figure()
    for i in range(16):
        plt.subplot( 4, 4, i+1 )
        plt.title( ytrain[i] )
        plt.imshow( xtrain[i] )

    plt.show( )

    # put all of the remaining code into the main function here

# execute the main function if executed
if __name__ == "__main__":
    main(sys.argv)

In the plots, note that the images are inverted: the digits are white on black. This has implications if you decide to generate your own data.

How big are the testing and training sets? how precisely can we characterize accuracy with this test set?

Transform the data
Often, the raw data isn't quite in the right form to pass to a network. For images being applied to a convolutional network, it's necessary to indicate the number of channels in the image. In this case, there is only one channel as the images are greyscale. It's also usually a good idea to convert image data from bytes or integers to floats and scale the values to the range [0, 1]. Finally, the labels in their raw form are integers, but they need to be a categorical numpy type.

Numpy provides a reshape function which allows you to add or modify the dimensions of an array. In this case, we want to keep the same ordering of the existing dimensions and simply add one more dimension.

Insert the following code into your main function.
```
    ################# Reshape and transform the data
    # reshape so the channel is last to match the Conv2D arguments
    xtrain = xtrain.reshape( xtrain.shape[0], xtrain.shape[1], xtrain.shape[2], 1 )
    xtest = xtest.reshape( xtest.shape[0], xtest.shape[1], xtest.shape[2], 1 )

    # print the new shape of the data
    print( "reshaped data: ", xtrain.shape )

    # convert the input data to floats and normalize to [0, 1]
    xtrain = xtrain.astype('float32')
    xtest = xtest.astype('float32')
    xtrain /= 255.0
    xtest /= 255.0

    # the output data needs to in one-hot form
    ytrain = np_utils.to_categorical( ytrain, 10 )
    ytest = np_utils.to_categorical( ytest, 10 )

    # look at the revised data
    print(xtrain.shape)
    print(xtrain[0][6])
```
Run this code and look at the values for the 6th row of the digit. Note the range of the values is now [0,1] and that most of the background pixels are 0.

Build the network

Building a network in Keras requires creating an empty network model and then adding layers to it. We're going to build a model with two convolution layers, one fully connected layer, and one output layer. In between the first two layers we'll insert a max pooling layer to shrink the spatial aspect and a dropout layer to reduce the chances of overfitting. We'll also put a dropout layer between the second and third layers. We'll use the ReLU activation function for the first three layers and the softmax function (build on a sigmoid) for the final layer.

The number of nodes in the Dense layer, the number of channels in the Conv2D layers, the filter sizes in the Conv2D layers, and the percentages in the Dropout layers are all parameters that could potentially be improved with experimentation.

    ################# Build the network

    # define an empty sequential model
    model = Sequential()

    print("Adding the input layer")
    # define the input layer
    # 32 3x3 filters with relu activation
    # step size is (1, 1) by default
    model.add( Conv2D(32, (3, 3), activation='relu', data_format="channels_last", input_shape=(28, 28, 1) ) )

    print("Adding second convolution layer")
    # define an additional 32 filter layer with pooling and dropout
    model.add( Conv2D(32, (3, 3), activation='relu', data_format="channels_last" ) )
    model.add( MaxPooling2D( pool_size=(2, 2) ) )
    model.add( Dropout( 0.25) )

    print("Adding fully connected layer")
    # define a fully connected layer
    # have to flatten the convolution layer prior to a dense layer (make it 1D)
    model.add( Flatten() )
    model.add( Dense( 128, activation='relu' ) )
    model.add( Dropout( 0.5 ) )

    print("Adding the output layer")
    # define the output layer
    model.add( Dense( 10, activation='softmax') )
    
    # compile the model
    # define the loss function as categorical cross-entropy
    # define the optimizer as adam
    # define the metrics (what is accuracy?)
    model.compile( loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'] )

    # print the model
    model.summary()

The summary function at the end prints the whole model to the terminal. It's useful to go through this in detail and understand how the spatial extent of each layer changes as the data moves through the network.

Train the network
We use the model.fit() function to train a network. The function requires the training set and the training labels. We can also specify the batch size and the number of epochs. The batch size is an important training parameter because it specifies how often the weights of the network get updated. Processing all of the training set is one epoch, and training a network often requires many epochs, depending on the network and training set sizes.

Put the following code at the end of your main function. Start by training for just one epoch. After each epoch, run the model on the test set to see how well it is generalizing to unseen data.
```
    # define the batch size (number of examples to run between updates)
    # define the number of epochs (runs through the entire data)
    # can use callbacks to look at the proces and save data
    numEpochs = 1
    trainscores = []
    testscores = []
    
    for i in range(numEpochs):
        print("Epoch", i)
        model.fit( xtrain, ytrain, batch_size=32, epochs=1, verbose=1 )

        # evaluate the model on the training and testing sets
        score = model.evaluate( xtrain, ytrain, verbose=1 )
        trainscores.append(score)
    
        score = model.evaluate( xtest, ytest, verbose=1 )
        testscores.append(score)
        print("Test score:", score)
```
Run the code. The number on the left indicates the batch being processed, and the loss (error) and accuracy are also shown.
Train for multiple epochs
Update the code to train for multiple epochs, such as 5 or 6. How long you should train a network is called a meta-parameters. Technically, we should be using a validation set, rather than a test set, to evaluate how long to train a network. The test set values should not be used to control anything about the training process or else the training process becomes biased.

Using a command line parameter to control the number of epochs can be useful. You can add the following after the assignment to numEpochs.
```
    numEpochs = 1 # initialize numEpochs to a default of 1
    if len(sys.argv) > 1:
        try:
            numEpochs = int(sys.argv[1])
        except:
            print("Invalid number of epochs given on command line")
    print("Training for %d epochs" % (numEpochs) )
    
```

Look at the scores

Finally, we might want to print out the loss and accuracy during training. Loop over the trainscores and testscores and print them out in CSV format for easy copy and paste into a spreadsheet.

    ################# Look at the scores
    print("\nEpoch,Train Loss,Train Acc,Test Loss,Test Acc")
    data = []
    for i in range(len(trainscores)):
        print("%d,%.4f,%.4f,%.4f,%.4f" % (i, trainscores[i][0], trainscores[i][1],
                                          testscores[i][0], testscores[i][1] ) )
        data.append((i+1, trainscores[i][0], trainscores[i][1],
                     testscores[i][0], testscores[i][1] ) )
    print()

    data = np.array( data )
    plt.plot( data[:,[1,3]] )
    plt.show()

Run the code for 3 epochs and look at the results. How good is the network?

Experiment
Think about how you might test different parameters for the network. What are the things you can control? What do you think will improve the performance?