Introduction to Tensorflow¶

Organized by Kayan Irani

Recording¶

1. What is Tensorflow?¶

1.1 Definition¶

Tensorflow is an open source library made by google that allows us to easily create powerful deep learning models and structures.

1.2 Deep Learning¶

Deep learning is a type of algorithm that learns information using these rough steps.

Get a labeled dataset of inputs mapped to outputs(think of this as example question and answers before a test).
Use a model to try and predict outputs based on inputs.
Get feedback on how close you are to the correct answer.
Update your knowledge and continue learning.

2. Breakdown of the process¶

We now breakdown the steps mentioned above and talk about how we can use them on a problem. To follow the basics visually, you can go to the tensorflow playground and view the training process.

2.1. Dataset¶

Your dataset is just a collection of examples where you take input data and tell the computer what the correct outputs for these examples are.

On the tensorflow playground we can select a dataset of our choice under the "DATA" section on the left-hand side.

2.2. Model¶

Deep learning models learn using a structure known as a Neural Network. It has this name because visualizations of these models look like neural pathways in our brains. While this sounds complicated, the model construction follows a fairly easy procedure. To construct this model we stack 'layers' on top of each other. While advancements in research have created many new types of layers for computation, many networks simply operate on a very simple layer known as a Dense layer. This layer performs three operations:

Multiplies the inputs with a set of coefficients (these coefficients automatically learned as the model trains).
Add all of these terms together.
Pass this number through a non-linear function (also known as an activation fucntion). For most models using the function ReLU will work very well for all types of problems and TaNH can be good for some classification datasets.

The tensorflow playground will allow you to select these parameters using the GUI options in the center of your screen.

2.3 Evaluation of the Model¶

To give feedback to our model we will run it on the dataset and compare it's outputs to the results it was supposed to predict. We do this to find the error (difference) between the two values. Common evaluation functions for regression are Mean Squared Error and Mean Average Error. Common evaluation functions for classification are binary crossentropy (if you only have two classes) and categorical crossentropy (if you have many classes).

In tensorflow playground you will find this loss below the output category.

2.4 Learning¶

After the model is evaluated it must take this feedback and work on correcting it's mistakes. This is done using an optimizer which tells the model how to update the coefficients in the layers to reduce the error from the evaluation function. You will need to select two parts of this process :

Choice of Optimizer : the Adam and RMSprop optimizers work best for most problems. We will train our models using these in this tutorial.
Learning rate : This is how fast the model learns from it's mistakes as well as how big of a change it will make to correct them. If this number is too big, it will keep overcorrecting and never get that good. On the other hand, if this is too small it will take too long to train. The best value is in the middle of these extremes. A value of 0.001 is a good starting point for most problems.

Tensorflow playground uses RMSprop but allows you to select the learning rate.

3. Writing the Code¶

Running a model on example data is fun and good for learning, but to utilize these models on real-world problems we require code. Let us begin with importing the libraries.

In [ ]:

  Copied!     
 
import tensorflow as tf # Tensorflow Import
import matplotlib.pyplot as plt # We use this to plot graphs
import tensorflow as tf # Tensorflow Import import matplotlib.pyplot as plt # We use this to plot graphs

3.1 Coding the Dataset¶

We can import any dataset to use for our model. However, tensorflow already comes with pre-loaded datasets that we can use for learning purposes. To show you a simple neural network we will use the boston housing dataset. The aim of this dataset is to predict the average price of a home in an area of Boston given some information about the area. We store the information about the areas in train_x and test_x. The coresponding values for housing prices are stored in train_y and test_y.

In [ ]:

  Copied!     
 
housing_train_data, housing_test_data = tf.keras.datasets.boston_housing.load_data()

housing_train_x, housing_train_y = housing_train_data

housing_test_x, housing_test_y = housing_test_data
housing_train_data, housing_test_data = tf.keras.datasets.boston_housing.load_data() housing_train_x, housing_train_y = housing_train_data housing_test_x, housing_test_y = housing_test_data

In [ ]:

  Copied!     
 
print(housing_train_x.shape, housing_train_y.shape, housing_test_x.shape, housing_test_y.shape)
print(housing_train_x.shape, housing_train_y.shape, housing_test_x.shape, housing_test_y.shape)

(404, 13) (404,) (102, 13) (102,)

3.2 Constructing the Model¶

To begin we create a Sequential model. This means that the output of the previous layer is passed as input to the next layer in a linear order. After this we add some layers into our model in the order we intend to use them. This model will be made of dense layers like the ones we discussed in the prior section. The only new layer is known as the Input layer. This tells tensorflow the size of an input example so that it knows how big of an input to expect.

In [ ]:

  Copied!     
 
import os
os.environ['TF_DETERMINISTIC_OPS'] = '1'
tf.random.set_seed(1)

housing_model = tf.keras.models.Sequential()

housing_model.add(tf.keras.layers.Input(shape = (13, ) )) # this says that each input is a list of 13 numerical values

# Hidden Layers
housing_model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
housing_model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu))
housing_model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu))

# This is the output layer. It's size must match the output shape. As we are only trying to predict a single number, we set this to be one.
housing_model.add(tf.keras.layers.Dense(1, activation = tf.nn.relu))

housing_model.summary()
import os os.environ['TF_DETERMINISTIC_OPS'] = '1' tf.random.set_seed(1) housing_model = tf.keras.models.Sequential() housing_model.add(tf.keras.layers.Input(shape = (13, ) )) # this says that each input is a list of 13 numerical values # Hidden Layers housing_model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu)) housing_model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu)) housing_model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu)) # This is the output layer. It's size must match the output shape. As we are only trying to predict a single number, we set this to be one. housing_model.add(tf.keras.layers.Dense(1, activation = tf.nn.relu)) housing_model.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 128)               1792      
                                                                 
 dense_1 (Dense)             (None, 64)                8256      
                                                                 
 dense_2 (Dense)             (None, 32)                2080      
                                                                 
 dense_3 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 12,161
Trainable params: 12,161
Non-trainable params: 0
_________________________________________________________________

In [ ]:

  Copied!     
 
# Visulizing the model we created in an image
tf.keras.utils.plot_model(housing_model)
# Visulizing the model we created in an image tf.keras.utils.plot_model(housing_model)

Out[ ]:

No description has been provided for this image

3.3 Setting Training and Evaluation Parameters¶

For tensorflow, this is the stage where we need to define how our model is evaluated and how it will learn. For this we specify three things:

Loss : This will be our evaluation function which gives us the error and the model it's feedback. We have selected mean squared error (mse).
Optimizer : The optimizer our model uses to learn. We also specify our learning rate here. The Adam optimizer has been selected here.
Metrics : This is like an evaluation function as it tells us the error the model achieves. However, the difference is that the model will not recive feedback from this; it is only for our information. Mean average error (mae) has been selected.

In [ ]:

  Copied!     
 
housing_model.compile(loss = 'mse', optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001), metrics = ['mae'])
housing_model.compile(loss = 'mse', optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001), metrics = ['mae'])

3.4 Training the Model¶

To train our model we simply give it the following information:

X and y training data.
Epochs : The number of times it runs the training algorithm.
Batch size : The number of examples it learns from at once. For example, if we set this to 32 the training data is split into batches where each batch has 32 training examples. We then take the examples one by one to run the model, evaluate it and use the optimizer. A single epoch is completed when all training batches have been seen once. This is especially useful with large datasets where tey may be too big to evaluate and optimize all at once.
Validation data : This data consists of new examples not present in the training set.This is used to verify that the model will perform well on new examples it has not seen before.

In [ ]:

  Copied!     
 
history = housing_model.fit(housing_train_x, housing_train_y, epochs = 200, batch_size = 32, 
                            validation_data = (housing_test_x, housing_test_y))
history = housing_model.fit(housing_train_x, housing_train_y, epochs = 200, batch_size = 32, validation_data = (housing_test_x, housing_test_y))

Epoch 1/200
13/13 [==============================] - 3s 20ms/step - loss: 158.8385 - mae: 9.6381 - val_loss: 95.7954 - val_mae: 7.8068
Epoch 2/200
13/13 [==============================] - 0s 6ms/step - loss: 76.3933 - mae: 6.3137 - val_loss: 65.9840 - val_mae: 6.0853
Epoch 3/200
13/13 [==============================] - 0s 6ms/step - loss: 65.9594 - mae: 5.7446 - val_loss: 66.3625 - val_mae: 6.0314
Epoch 4/200
13/13 [==============================] - 0s 5ms/step - loss: 62.5118 - mae: 5.5272 - val_loss: 74.6212 - val_mae: 6.8268
Epoch 5/200
13/13 [==============================] - 0s 7ms/step - loss: 74.0129 - mae: 6.4932 - val_loss: 69.1981 - val_mae: 6.3276
Epoch 6/200
13/13 [==============================] - 0s 6ms/step - loss: 59.4877 - mae: 5.2731 - val_loss: 60.4491 - val_mae: 5.7421
Epoch 7/200
13/13 [==============================] - 0s 6ms/step - loss: 57.2982 - mae: 5.2019 - val_loss: 64.5019 - val_mae: 5.7485
Epoch 8/200
13/13 [==============================] - 0s 6ms/step - loss: 55.2949 - mae: 5.2820 - val_loss: 60.4656 - val_mae: 5.5768
Epoch 9/200
13/13 [==============================] - 0s 7ms/step - loss: 54.4415 - mae: 5.0364 - val_loss: 59.9733 - val_mae: 5.9685
Epoch 10/200
13/13 [==============================] - 0s 9ms/step - loss: 55.4086 - mae: 5.2899 - val_loss: 61.1802 - val_mae: 5.5353
Epoch 11/200
13/13 [==============================] - 0s 8ms/step - loss: 52.3622 - mae: 5.1678 - val_loss: 70.7138 - val_mae: 6.0207
Epoch 12/200
13/13 [==============================] - 0s 9ms/step - loss: 54.7812 - mae: 5.1739 - val_loss: 60.4563 - val_mae: 5.5465
Epoch 13/200
13/13 [==============================] - 0s 10ms/step - loss: 51.1010 - mae: 4.9313 - val_loss: 56.2302 - val_mae: 5.8327
Epoch 14/200
13/13 [==============================] - 0s 8ms/step - loss: 54.6022 - mae: 5.4264 - val_loss: 50.6665 - val_mae: 5.1169
Epoch 15/200
13/13 [==============================] - 0s 7ms/step - loss: 49.1987 - mae: 4.8514 - val_loss: 49.9302 - val_mae: 5.1431
Epoch 16/200
13/13 [==============================] - 0s 8ms/step - loss: 44.1350 - mae: 4.6618 - val_loss: 54.3591 - val_mae: 5.6313
Epoch 17/200
13/13 [==============================] - 0s 11ms/step - loss: 44.7293 - mae: 4.5990 - val_loss: 47.4475 - val_mae: 4.9372
Epoch 18/200
13/13 [==============================] - 0s 8ms/step - loss: 48.6658 - mae: 5.1030 - val_loss: 45.9619 - val_mae: 4.8511
Epoch 19/200
13/13 [==============================] - 0s 16ms/step - loss: 44.6938 - mae: 4.7826 - val_loss: 50.4378 - val_mae: 5.1573
Epoch 20/200
13/13 [==============================] - 0s 19ms/step - loss: 42.7974 - mae: 4.5628 - val_loss: 48.9445 - val_mae: 5.0729
Epoch 21/200
13/13 [==============================] - 0s 16ms/step - loss: 40.4660 - mae: 4.4248 - val_loss: 44.1847 - val_mae: 4.7969
Epoch 22/200
13/13 [==============================] - 0s 16ms/step - loss: 42.0077 - mae: 4.5317 - val_loss: 49.4147 - val_mae: 5.4573
Epoch 23/200
13/13 [==============================] - 0s 16ms/step - loss: 42.6910 - mae: 4.5960 - val_loss: 45.0685 - val_mae: 5.0135
Epoch 24/200
13/13 [==============================] - 0s 12ms/step - loss: 43.0620 - mae: 4.8228 - val_loss: 69.4898 - val_mae: 6.1312
Epoch 25/200
13/13 [==============================] - 0s 13ms/step - loss: 45.0583 - mae: 4.8055 - val_loss: 50.0795 - val_mae: 5.6526
Epoch 26/200
13/13 [==============================] - 0s 13ms/step - loss: 39.8796 - mae: 4.6951 - val_loss: 47.7645 - val_mae: 5.4730
Epoch 27/200
13/13 [==============================] - 0s 12ms/step - loss: 40.4625 - mae: 4.6206 - val_loss: 66.1004 - val_mae: 6.0231
Epoch 28/200
13/13 [==============================] - 0s 11ms/step - loss: 44.0332 - mae: 4.8351 - val_loss: 41.6119 - val_mae: 4.7721
Epoch 29/200
13/13 [==============================] - 0s 11ms/step - loss: 37.6091 - mae: 4.3522 - val_loss: 43.1239 - val_mae: 4.9717
Epoch 30/200
13/13 [==============================] - 0s 9ms/step - loss: 40.1845 - mae: 4.7036 - val_loss: 46.3408 - val_mae: 4.7283
Epoch 31/200
13/13 [==============================] - 0s 9ms/step - loss: 40.7183 - mae: 4.3376 - val_loss: 71.5646 - val_mae: 7.3153
Epoch 32/200
13/13 [==============================] - 0s 10ms/step - loss: 44.9594 - mae: 5.0427 - val_loss: 47.3080 - val_mae: 5.1347
Epoch 33/200
13/13 [==============================] - 0s 7ms/step - loss: 36.1740 - mae: 4.2906 - val_loss: 37.3797 - val_mae: 4.3156
Epoch 34/200
13/13 [==============================] - 0s 7ms/step - loss: 35.1721 - mae: 4.2808 - val_loss: 43.5961 - val_mae: 5.1065
Epoch 35/200
13/13 [==============================] - 0s 9ms/step - loss: 42.3842 - mae: 4.6613 - val_loss: 39.9437 - val_mae: 4.6140
Epoch 36/200
13/13 [==============================] - 0s 14ms/step - loss: 34.9352 - mae: 4.2898 - val_loss: 36.6850 - val_mae: 4.3644
Epoch 37/200
13/13 [==============================] - 0s 8ms/step - loss: 37.5937 - mae: 4.5599 - val_loss: 46.0385 - val_mae: 4.8115
Epoch 38/200
13/13 [==============================] - 0s 7ms/step - loss: 38.6579 - mae: 4.5573 - val_loss: 54.7305 - val_mae: 5.7872
Epoch 39/200
13/13 [==============================] - 0s 15ms/step - loss: 39.0535 - mae: 4.5922 - val_loss: 36.8828 - val_mae: 4.3484
Epoch 40/200
13/13 [==============================] - 0s 10ms/step - loss: 32.1071 - mae: 4.1278 - val_loss: 36.2320 - val_mae: 4.4966
Epoch 41/200
13/13 [==============================] - 0s 10ms/step - loss: 31.5088 - mae: 4.0360 - val_loss: 36.7014 - val_mae: 4.1789
Epoch 42/200
13/13 [==============================] - 0s 10ms/step - loss: 29.2168 - mae: 3.8722 - val_loss: 35.3969 - val_mae: 4.1913
Epoch 43/200
13/13 [==============================] - 0s 13ms/step - loss: 32.2159 - mae: 4.1641 - val_loss: 36.9562 - val_mae: 4.2649
Epoch 44/200
13/13 [==============================] - 0s 9ms/step - loss: 29.7467 - mae: 3.9412 - val_loss: 36.2007 - val_mae: 4.1624
Epoch 45/200
13/13 [==============================] - 0s 14ms/step - loss: 28.2262 - mae: 3.8844 - val_loss: 34.9140 - val_mae: 4.3519
Epoch 46/200
13/13 [==============================] - 0s 10ms/step - loss: 27.3710 - mae: 3.8364 - val_loss: 39.7883 - val_mae: 4.3774
Epoch 47/200
13/13 [==============================] - 0s 14ms/step - loss: 27.0967 - mae: 3.7600 - val_loss: 34.0516 - val_mae: 4.3883
Epoch 48/200
13/13 [==============================] - 0s 10ms/step - loss: 25.9610 - mae: 3.7246 - val_loss: 31.9078 - val_mae: 4.1271
Epoch 49/200
13/13 [==============================] - 0s 10ms/step - loss: 27.8723 - mae: 3.9180 - val_loss: 34.9717 - val_mae: 4.1329
Epoch 50/200
13/13 [==============================] - 0s 10ms/step - loss: 32.7865 - mae: 4.1370 - val_loss: 36.5629 - val_mae: 4.7346
Epoch 51/200
13/13 [==============================] - 0s 9ms/step - loss: 31.7112 - mae: 4.0968 - val_loss: 35.9101 - val_mae: 4.2142
Epoch 52/200
13/13 [==============================] - 0s 9ms/step - loss: 31.5274 - mae: 4.0991 - val_loss: 35.5170 - val_mae: 4.6127
Epoch 53/200
13/13 [==============================] - 0s 9ms/step - loss: 35.3326 - mae: 4.3822 - val_loss: 36.5821 - val_mae: 4.2943
Epoch 54/200
13/13 [==============================] - 0s 10ms/step - loss: 28.3815 - mae: 3.9179 - val_loss: 31.7608 - val_mae: 4.0195
Epoch 55/200
13/13 [==============================] - 0s 8ms/step - loss: 26.7372 - mae: 3.7736 - val_loss: 31.3684 - val_mae: 3.9846
Epoch 56/200
13/13 [==============================] - 0s 15ms/step - loss: 27.3837 - mae: 4.0081 - val_loss: 31.9295 - val_mae: 4.1485
Epoch 57/200
13/13 [==============================] - 0s 5ms/step - loss: 23.7578 - mae: 3.5441 - val_loss: 32.2378 - val_mae: 4.0319
Epoch 58/200
13/13 [==============================] - 0s 10ms/step - loss: 23.5053 - mae: 3.5031 - val_loss: 31.2508 - val_mae: 4.0442
Epoch 59/200
13/13 [==============================] - 0s 13ms/step - loss: 23.4275 - mae: 3.5010 - val_loss: 31.8437 - val_mae: 4.1435
Epoch 60/200
13/13 [==============================] - 0s 10ms/step - loss: 21.5507 - mae: 3.4121 - val_loss: 29.1006 - val_mae: 3.9623
Epoch 61/200
13/13 [==============================] - 0s 9ms/step - loss: 22.4740 - mae: 3.4556 - val_loss: 28.5392 - val_mae: 3.8531
Epoch 62/200
13/13 [==============================] - 0s 11ms/step - loss: 22.5698 - mae: 3.5308 - val_loss: 32.2885 - val_mae: 4.0641
Epoch 63/200
13/13 [==============================] - 0s 10ms/step - loss: 21.4554 - mae: 3.3937 - val_loss: 30.3071 - val_mae: 4.1957
Epoch 64/200
13/13 [==============================] - 0s 10ms/step - loss: 21.8027 - mae: 3.4242 - val_loss: 27.1902 - val_mae: 3.8086
Epoch 65/200
13/13 [==============================] - 0s 10ms/step - loss: 24.7308 - mae: 3.6522 - val_loss: 39.3288 - val_mae: 4.9832
Epoch 66/200
13/13 [==============================] - 0s 12ms/step - loss: 22.9252 - mae: 3.5862 - val_loss: 32.6013 - val_mae: 4.0197
Epoch 67/200
13/13 [==============================] - 0s 9ms/step - loss: 22.9201 - mae: 3.5575 - val_loss: 32.5449 - val_mae: 3.9819
Epoch 68/200
13/13 [==============================] - 0s 9ms/step - loss: 26.7672 - mae: 3.7019 - val_loss: 41.9045 - val_mae: 4.8744
Epoch 69/200
13/13 [==============================] - 0s 10ms/step - loss: 21.1358 - mae: 3.3544 - val_loss: 30.2412 - val_mae: 4.2482
Epoch 70/200
13/13 [==============================] - 0s 12ms/step - loss: 18.4177 - mae: 3.1963 - val_loss: 32.0725 - val_mae: 3.9863
Epoch 71/200
13/13 [==============================] - 0s 9ms/step - loss: 19.6991 - mae: 3.3073 - val_loss: 28.9564 - val_mae: 4.0845
Epoch 72/200
13/13 [==============================] - 0s 9ms/step - loss: 18.8953 - mae: 3.2107 - val_loss: 28.8556 - val_mae: 3.8060
Epoch 73/200
13/13 [==============================] - 0s 8ms/step - loss: 19.9072 - mae: 3.2788 - val_loss: 29.5674 - val_mae: 3.7913
Epoch 74/200
13/13 [==============================] - 0s 13ms/step - loss: 16.7959 - mae: 3.0089 - val_loss: 29.3315 - val_mae: 3.8642
Epoch 75/200
13/13 [==============================] - 0s 11ms/step - loss: 16.5847 - mae: 3.0035 - val_loss: 29.0402 - val_mae: 3.7806
Epoch 76/200
13/13 [==============================] - 0s 12ms/step - loss: 18.7005 - mae: 3.1929 - val_loss: 29.0227 - val_mae: 3.8439
Epoch 77/200
13/13 [==============================] - 0s 14ms/step - loss: 16.6677 - mae: 2.9877 - val_loss: 37.2215 - val_mae: 4.4234
Epoch 78/200
13/13 [==============================] - 0s 12ms/step - loss: 17.5018 - mae: 3.0431 - val_loss: 31.3399 - val_mae: 4.2431
Epoch 79/200
13/13 [==============================] - 0s 10ms/step - loss: 19.6106 - mae: 3.2082 - val_loss: 32.4476 - val_mae: 4.2819
Epoch 80/200
13/13 [==============================] - 0s 11ms/step - loss: 36.2121 - mae: 4.7760 - val_loss: 28.4692 - val_mae: 3.9099
Epoch 81/200
13/13 [==============================] - 0s 12ms/step - loss: 40.8866 - mae: 4.8306 - val_loss: 35.2170 - val_mae: 4.3141
Epoch 82/200
13/13 [==============================] - 0s 11ms/step - loss: 25.5971 - mae: 3.7871 - val_loss: 31.8203 - val_mae: 3.9623
Epoch 83/200
13/13 [==============================] - 0s 12ms/step - loss: 20.3588 - mae: 3.3205 - val_loss: 30.0706 - val_mae: 3.9036
Epoch 84/200
13/13 [==============================] - 0s 6ms/step - loss: 25.3911 - mae: 3.8649 - val_loss: 33.7465 - val_mae: 4.5236
Epoch 85/200
13/13 [==============================] - 0s 6ms/step - loss: 19.7263 - mae: 3.2715 - val_loss: 32.0944 - val_mae: 3.9505
Epoch 86/200
13/13 [==============================] - 0s 7ms/step - loss: 16.5362 - mae: 3.0233 - val_loss: 29.3023 - val_mae: 3.9333
Epoch 87/200
13/13 [==============================] - 0s 20ms/step - loss: 17.9351 - mae: 3.1035 - val_loss: 34.1284 - val_mae: 4.0650
Epoch 88/200
13/13 [==============================] - 0s 11ms/step - loss: 16.2137 - mae: 2.9476 - val_loss: 27.3214 - val_mae: 3.7984
Epoch 89/200
13/13 [==============================] - 0s 6ms/step - loss: 18.8782 - mae: 3.1469 - val_loss: 35.5682 - val_mae: 4.2400
Epoch 90/200
13/13 [==============================] - 0s 14ms/step - loss: 17.1858 - mae: 3.0329 - val_loss: 31.3671 - val_mae: 3.9237
Epoch 91/200
13/13 [==============================] - 0s 17ms/step - loss: 15.3767 - mae: 2.9254 - val_loss: 30.9757 - val_mae: 4.0625
Epoch 92/200
13/13 [==============================] - 0s 10ms/step - loss: 14.6350 - mae: 2.8037 - val_loss: 33.2747 - val_mae: 3.9690
Epoch 93/200
13/13 [==============================] - 0s 11ms/step - loss: 18.9898 - mae: 3.1626 - val_loss: 34.8545 - val_mae: 4.2942
Epoch 94/200
13/13 [==============================] - 0s 15ms/step - loss: 17.2571 - mae: 3.0950 - val_loss: 37.2295 - val_mae: 4.2279
Epoch 95/200
13/13 [==============================] - 0s 24ms/step - loss: 17.2022 - mae: 3.0691 - val_loss: 33.6021 - val_mae: 4.0912
Epoch 96/200
13/13 [==============================] - 0s 22ms/step - loss: 14.8782 - mae: 2.8536 - val_loss: 28.8832 - val_mae: 3.9681
Epoch 97/200
13/13 [==============================] - 0s 17ms/step - loss: 16.0852 - mae: 2.9510 - val_loss: 35.1912 - val_mae: 4.0916
Epoch 98/200
13/13 [==============================] - 0s 15ms/step - loss: 15.8149 - mae: 2.8870 - val_loss: 30.4375 - val_mae: 4.0452
Epoch 99/200
13/13 [==============================] - 0s 24ms/step - loss: 15.2301 - mae: 2.8610 - val_loss: 30.6453 - val_mae: 3.7918
Epoch 100/200
13/13 [==============================] - 0s 26ms/step - loss: 15.6172 - mae: 2.8977 - val_loss: 38.3264 - val_mae: 4.5286
Epoch 101/200
13/13 [==============================] - 0s 16ms/step - loss: 16.5772 - mae: 3.0132 - val_loss: 32.3224 - val_mae: 4.0342
Epoch 102/200
13/13 [==============================] - 0s 23ms/step - loss: 16.5946 - mae: 3.0472 - val_loss: 29.5199 - val_mae: 3.8470
Epoch 103/200
13/13 [==============================] - 0s 26ms/step - loss: 14.4194 - mae: 2.7852 - val_loss: 32.6959 - val_mae: 3.9352
Epoch 104/200
13/13 [==============================] - 0s 10ms/step - loss: 13.8321 - mae: 2.7185 - val_loss: 32.7366 - val_mae: 3.9338
Epoch 105/200
13/13 [==============================] - 0s 9ms/step - loss: 14.4389 - mae: 2.7212 - val_loss: 32.4684 - val_mae: 4.1554
Epoch 106/200
13/13 [==============================] - 0s 10ms/step - loss: 14.4000 - mae: 2.7979 - val_loss: 37.3536 - val_mae: 4.4501
Epoch 107/200
13/13 [==============================] - 0s 14ms/step - loss: 17.8691 - mae: 3.1741 - val_loss: 29.1527 - val_mae: 3.7640
Epoch 108/200
13/13 [==============================] - 0s 9ms/step - loss: 16.2034 - mae: 3.0437 - val_loss: 32.7455 - val_mae: 3.9841
Epoch 109/200
13/13 [==============================] - 0s 10ms/step - loss: 15.4557 - mae: 3.0047 - val_loss: 31.6899 - val_mae: 3.9538
Epoch 110/200
13/13 [==============================] - 0s 12ms/step - loss: 13.9040 - mae: 2.7296 - val_loss: 32.0091 - val_mae: 4.2775
Epoch 111/200
13/13 [==============================] - 0s 10ms/step - loss: 15.7991 - mae: 2.9598 - val_loss: 35.2154 - val_mae: 4.2501
Epoch 112/200
13/13 [==============================] - 0s 13ms/step - loss: 14.4490 - mae: 2.8153 - val_loss: 29.9829 - val_mae: 3.8391
Epoch 113/200
13/13 [==============================] - 0s 10ms/step - loss: 13.2926 - mae: 2.7048 - val_loss: 31.8703 - val_mae: 3.8924
Epoch 114/200
13/13 [==============================] - 0s 11ms/step - loss: 13.3824 - mae: 2.6615 - val_loss: 41.0271 - val_mae: 4.8232
Epoch 115/200
13/13 [==============================] - 0s 13ms/step - loss: 17.4602 - mae: 3.0191 - val_loss: 30.4805 - val_mae: 3.7799
Epoch 116/200
13/13 [==============================] - 0s 15ms/step - loss: 21.0600 - mae: 3.4008 - val_loss: 32.8811 - val_mae: 4.0718
Epoch 117/200
13/13 [==============================] - 0s 12ms/step - loss: 20.6031 - mae: 3.3344 - val_loss: 29.2403 - val_mae: 4.1208
Epoch 118/200
13/13 [==============================] - 0s 16ms/step - loss: 16.8213 - mae: 3.0334 - val_loss: 31.8631 - val_mae: 3.9916
Epoch 119/200
13/13 [==============================] - 0s 14ms/step - loss: 15.0947 - mae: 2.8785 - val_loss: 35.9509 - val_mae: 4.4248
Epoch 120/200
13/13 [==============================] - 0s 10ms/step - loss: 16.5802 - mae: 3.0263 - val_loss: 34.4360 - val_mae: 4.0188
Epoch 121/200
13/13 [==============================] - 0s 12ms/step - loss: 17.7656 - mae: 3.0269 - val_loss: 42.7722 - val_mae: 5.0620
Epoch 122/200
13/13 [==============================] - 0s 13ms/step - loss: 17.3118 - mae: 3.0337 - val_loss: 32.5483 - val_mae: 3.9267
Epoch 123/200
13/13 [==============================] - 0s 10ms/step - loss: 13.7797 - mae: 2.7566 - val_loss: 32.5470 - val_mae: 4.0013
Epoch 124/200
13/13 [==============================] - 0s 11ms/step - loss: 20.4752 - mae: 3.3211 - val_loss: 35.2720 - val_mae: 4.0995
Epoch 125/200
13/13 [==============================] - 0s 12ms/step - loss: 16.4471 - mae: 2.9872 - val_loss: 36.7188 - val_mae: 4.1868
Epoch 126/200
13/13 [==============================] - 0s 15ms/step - loss: 15.9363 - mae: 2.9171 - val_loss: 44.0103 - val_mae: 4.5155
Epoch 127/200
13/13 [==============================] - 0s 11ms/step - loss: 15.3808 - mae: 2.8384 - val_loss: 31.5518 - val_mae: 3.8308
Epoch 128/200
13/13 [==============================] - 0s 15ms/step - loss: 14.5555 - mae: 2.8034 - val_loss: 38.3403 - val_mae: 4.6483
Epoch 129/200
13/13 [==============================] - 0s 10ms/step - loss: 15.5323 - mae: 3.0320 - val_loss: 33.0989 - val_mae: 3.9910
Epoch 130/200
13/13 [==============================] - 0s 12ms/step - loss: 17.3143 - mae: 3.0361 - val_loss: 33.5947 - val_mae: 4.0584
Epoch 131/200
13/13 [==============================] - 0s 18ms/step - loss: 14.3783 - mae: 2.7693 - val_loss: 39.4443 - val_mae: 4.2810
Epoch 132/200
13/13 [==============================] - 0s 16ms/step - loss: 17.3261 - mae: 3.0688 - val_loss: 30.0193 - val_mae: 3.7745
Epoch 133/200
13/13 [==============================] - 0s 12ms/step - loss: 15.3965 - mae: 2.8311 - val_loss: 32.8228 - val_mae: 4.2744
Epoch 134/200
13/13 [==============================] - 0s 11ms/step - loss: 15.8023 - mae: 2.8975 - val_loss: 34.4812 - val_mae: 4.0377
Epoch 135/200
13/13 [==============================] - 0s 11ms/step - loss: 16.3663 - mae: 2.9170 - val_loss: 34.0779 - val_mae: 4.2226
Epoch 136/200
13/13 [==============================] - 0s 10ms/step - loss: 13.5982 - mae: 2.7267 - val_loss: 35.1694 - val_mae: 4.2569
Epoch 137/200
13/13 [==============================] - 0s 15ms/step - loss: 17.2159 - mae: 2.9728 - val_loss: 30.5006 - val_mae: 3.9121
Epoch 138/200
13/13 [==============================] - 0s 13ms/step - loss: 13.9763 - mae: 2.7302 - val_loss: 31.9929 - val_mae: 3.9713
Epoch 139/200
13/13 [==============================] - 0s 15ms/step - loss: 13.7487 - mae: 2.6992 - val_loss: 31.3816 - val_mae: 3.8321
Epoch 140/200
13/13 [==============================] - 0s 10ms/step - loss: 13.7296 - mae: 2.6848 - val_loss: 35.0499 - val_mae: 4.3181
Epoch 141/200
13/13 [==============================] - 0s 11ms/step - loss: 13.9495 - mae: 2.7698 - val_loss: 31.8743 - val_mae: 3.9288
Epoch 142/200
13/13 [==============================] - 0s 13ms/step - loss: 12.6621 - mae: 2.6362 - val_loss: 33.0965 - val_mae: 3.9352
Epoch 143/200
13/13 [==============================] - 0s 11ms/step - loss: 15.9332 - mae: 2.9238 - val_loss: 31.0388 - val_mae: 4.2493
Epoch 144/200
13/13 [==============================] - 0s 25ms/step - loss: 17.0841 - mae: 3.0638 - val_loss: 32.0788 - val_mae: 4.0152
Epoch 145/200
13/13 [==============================] - 0s 14ms/step - loss: 12.8091 - mae: 2.7026 - val_loss: 34.9562 - val_mae: 4.1135
Epoch 146/200
13/13 [==============================] - 0s 14ms/step - loss: 13.6347 - mae: 2.7887 - val_loss: 32.1059 - val_mae: 3.9237
Epoch 147/200
13/13 [==============================] - 0s 18ms/step - loss: 14.1129 - mae: 2.7363 - val_loss: 35.5106 - val_mae: 4.1788
Epoch 148/200
13/13 [==============================] - 0s 14ms/step - loss: 14.8108 - mae: 2.7969 - val_loss: 33.9997 - val_mae: 4.1948
Epoch 149/200
13/13 [==============================] - 0s 15ms/step - loss: 12.7958 - mae: 2.7145 - val_loss: 31.5091 - val_mae: 4.0978
Epoch 150/200
13/13 [==============================] - 0s 11ms/step - loss: 12.2240 - mae: 2.6315 - val_loss: 41.7839 - val_mae: 4.8633
Epoch 151/200
13/13 [==============================] - 0s 14ms/step - loss: 15.2069 - mae: 2.8599 - val_loss: 34.5472 - val_mae: 4.0323
Epoch 152/200
13/13 [==============================] - 0s 14ms/step - loss: 16.9133 - mae: 2.9558 - val_loss: 29.3688 - val_mae: 4.2467
Epoch 153/200
13/13 [==============================] - 0s 11ms/step - loss: 15.1710 - mae: 2.8159 - val_loss: 33.1368 - val_mae: 4.0092
Epoch 154/200
13/13 [==============================] - 0s 8ms/step - loss: 14.3021 - mae: 2.7724 - val_loss: 33.4760 - val_mae: 4.1566
Epoch 155/200
13/13 [==============================] - 0s 9ms/step - loss: 17.3418 - mae: 3.1447 - val_loss: 28.3930 - val_mae: 3.7028
Epoch 156/200
13/13 [==============================] - 0s 10ms/step - loss: 13.6271 - mae: 2.7637 - val_loss: 32.2386 - val_mae: 4.0141
Epoch 157/200
13/13 [==============================] - 0s 10ms/step - loss: 12.9707 - mae: 2.6545 - val_loss: 30.4212 - val_mae: 3.7932
Epoch 158/200
13/13 [==============================] - 0s 9ms/step - loss: 12.6811 - mae: 2.6814 - val_loss: 31.3527 - val_mae: 3.9395
Epoch 159/200
13/13 [==============================] - 0s 10ms/step - loss: 12.1345 - mae: 2.5292 - val_loss: 31.3304 - val_mae: 3.8144
Epoch 160/200
13/13 [==============================] - 0s 10ms/step - loss: 13.3987 - mae: 2.6795 - val_loss: 37.0845 - val_mae: 4.2673
Epoch 161/200
13/13 [==============================] - 0s 11ms/step - loss: 14.3757 - mae: 2.8472 - val_loss: 33.8015 - val_mae: 4.2567
Epoch 162/200
13/13 [==============================] - 0s 14ms/step - loss: 15.8548 - mae: 2.9494 - val_loss: 40.5819 - val_mae: 4.5811
Epoch 163/200
13/13 [==============================] - 0s 10ms/step - loss: 16.9006 - mae: 2.9989 - val_loss: 30.7470 - val_mae: 3.8467
Epoch 164/200
13/13 [==============================] - 0s 14ms/step - loss: 12.3910 - mae: 2.6357 - val_loss: 30.2299 - val_mae: 3.7768
Epoch 165/200
13/13 [==============================] - 0s 21ms/step - loss: 12.0933 - mae: 2.5328 - val_loss: 30.3041 - val_mae: 3.8112
Epoch 166/200
13/13 [==============================] - 0s 20ms/step - loss: 12.4885 - mae: 2.6463 - val_loss: 31.1604 - val_mae: 3.8742
Epoch 167/200
13/13 [==============================] - 0s 19ms/step - loss: 12.6699 - mae: 2.5701 - val_loss: 34.6526 - val_mae: 4.0566
Epoch 168/200
13/13 [==============================] - 0s 18ms/step - loss: 12.9657 - mae: 2.6245 - val_loss: 28.6603 - val_mae: 3.7383
Epoch 169/200
13/13 [==============================] - 0s 18ms/step - loss: 12.4112 - mae: 2.5634 - val_loss: 30.3964 - val_mae: 3.8123
Epoch 170/200
13/13 [==============================] - 0s 17ms/step - loss: 11.6612 - mae: 2.5292 - val_loss: 32.4453 - val_mae: 3.7918
Epoch 171/200
13/13 [==============================] - 0s 17ms/step - loss: 13.1072 - mae: 2.6638 - val_loss: 32.1502 - val_mae: 4.0244
Epoch 172/200
13/13 [==============================] - 0s 23ms/step - loss: 13.1694 - mae: 2.7193 - val_loss: 35.4646 - val_mae: 4.0583
Epoch 173/200
13/13 [==============================] - 0s 18ms/step - loss: 17.1581 - mae: 3.0186 - val_loss: 31.8929 - val_mae: 3.9824
Epoch 174/200
13/13 [==============================] - 0s 10ms/step - loss: 12.2085 - mae: 2.6285 - val_loss: 31.8660 - val_mae: 3.9625
Epoch 175/200
13/13 [==============================] - 0s 8ms/step - loss: 12.0036 - mae: 2.5467 - val_loss: 31.5978 - val_mae: 3.8880
Epoch 176/200
13/13 [==============================] - 0s 6ms/step - loss: 13.1227 - mae: 2.6581 - val_loss: 31.2217 - val_mae: 3.7960
Epoch 177/200
13/13 [==============================] - 0s 6ms/step - loss: 12.8984 - mae: 2.6586 - val_loss: 28.2532 - val_mae: 3.6868
Epoch 178/200
13/13 [==============================] - 0s 6ms/step - loss: 14.6840 - mae: 2.8977 - val_loss: 36.5319 - val_mae: 4.3848
Epoch 179/200
13/13 [==============================] - 0s 6ms/step - loss: 14.6307 - mae: 2.8265 - val_loss: 38.5169 - val_mae: 4.4178
Epoch 180/200
13/13 [==============================] - 0s 6ms/step - loss: 16.5457 - mae: 2.8798 - val_loss: 31.0762 - val_mae: 4.2741
Epoch 181/200
13/13 [==============================] - 0s 6ms/step - loss: 16.6882 - mae: 3.0610 - val_loss: 29.4484 - val_mae: 3.9237
Epoch 182/200
13/13 [==============================] - 0s 6ms/step - loss: 14.6878 - mae: 2.7954 - val_loss: 31.3718 - val_mae: 3.8418
Epoch 183/200
13/13 [==============================] - 0s 6ms/step - loss: 12.9565 - mae: 2.6387 - val_loss: 35.0675 - val_mae: 4.1496
Epoch 184/200
13/13 [==============================] - 0s 6ms/step - loss: 11.8077 - mae: 2.5009 - val_loss: 31.5590 - val_mae: 3.8201
Epoch 185/200
13/13 [==============================] - 0s 6ms/step - loss: 13.9091 - mae: 2.7684 - val_loss: 29.9861 - val_mae: 3.8698
Epoch 186/200
13/13 [==============================] - 0s 6ms/step - loss: 14.9959 - mae: 2.7887 - val_loss: 33.8696 - val_mae: 4.0010
Epoch 187/200
13/13 [==============================] - 0s 7ms/step - loss: 12.6529 - mae: 2.6726 - val_loss: 35.0206 - val_mae: 4.2086
Epoch 188/200
13/13 [==============================] - 0s 6ms/step - loss: 15.5221 - mae: 2.8949 - val_loss: 29.4232 - val_mae: 4.1253
Epoch 189/200
13/13 [==============================] - 0s 6ms/step - loss: 18.6872 - mae: 3.1408 - val_loss: 44.2267 - val_mae: 4.6163
Epoch 190/200
13/13 [==============================] - 0s 7ms/step - loss: 19.8354 - mae: 3.2660 - val_loss: 29.9729 - val_mae: 3.9121
Epoch 191/200
13/13 [==============================] - 0s 6ms/step - loss: 11.5201 - mae: 2.5467 - val_loss: 35.5785 - val_mae: 4.0348
Epoch 192/200
13/13 [==============================] - 0s 7ms/step - loss: 12.0410 - mae: 2.5484 - val_loss: 28.0710 - val_mae: 3.7645
Epoch 193/200
13/13 [==============================] - 0s 6ms/step - loss: 10.7334 - mae: 2.4454 - val_loss: 31.1684 - val_mae: 3.8540
Epoch 194/200
13/13 [==============================] - 0s 6ms/step - loss: 11.7818 - mae: 2.5695 - val_loss: 30.4062 - val_mae: 3.9568
Epoch 195/200
13/13 [==============================] - 0s 6ms/step - loss: 12.3549 - mae: 2.5756 - val_loss: 31.9517 - val_mae: 3.9071
Epoch 196/200
13/13 [==============================] - 0s 7ms/step - loss: 12.5529 - mae: 2.6248 - val_loss: 29.7121 - val_mae: 3.7148
Epoch 197/200
13/13 [==============================] - 0s 6ms/step - loss: 12.6280 - mae: 2.6674 - val_loss: 31.2437 - val_mae: 4.0034
Epoch 198/200
13/13 [==============================] - 0s 6ms/step - loss: 11.1278 - mae: 2.4925 - val_loss: 32.3009 - val_mae: 3.8607
Epoch 199/200
13/13 [==============================] - 0s 6ms/step - loss: 12.6047 - mae: 2.6339 - val_loss: 29.1286 - val_mae: 3.7785
Epoch 200/200
13/13 [==============================] - 0s 6ms/step - loss: 11.6160 - mae: 2.5763 - val_loss: 33.1137 - val_mae: 3.8703

3.4 Evaluating The Model¶

In [ ]:

  Copied!     
 
history = history.history
history = history.history

In [ ]:

  Copied!     
 
print(history.keys())
print(history.keys())

dict_keys(['loss', 'mae', 'val_loss', 'val_mae'])

In [ ]:

  Copied!     
 
plt.plot(range(1, 201), history['loss'])
plt.plot(range(1, 201), history['val_loss'])

plt.title('MSE Loss While Training')
plt.xlabel('Epoch Number')
plt.ylabel('MSE Loss')
plt.legend(['Training Loss', 'Testing Loss'])
plt.show()
plt.plot(range(1, 201), history['loss']) plt.plot(range(1, 201), history['val_loss']) plt.title('MSE Loss While Training') plt.xlabel('Epoch Number') plt.ylabel('MSE Loss') plt.legend(['Training Loss', 'Testing Loss']) plt.show()

In [ ]:

  Copied!     
 
plt.plot(range(1, 201), history['loss'])
plt.plot(range(1, 201), history['val_loss'])

plt.title('MSE Loss While Training')
plt.xlabel('Epoch Number')
plt.ylabel('MSE Loss')
plt.legend(['Training Loss', 'Testing Loss'])
plt.ylim(0, 100)
plt.show()
plt.plot(range(1, 201), history['loss']) plt.plot(range(1, 201), history['val_loss']) plt.title('MSE Loss While Training') plt.xlabel('Epoch Number') plt.ylabel('MSE Loss') plt.legend(['Training Loss', 'Testing Loss']) plt.ylim(0, 100) plt.show()

In [ ]:

  Copied!     
 
# We can also evaluate our model after training

housing_model.evaluate(housing_train_x, housing_train_y)
# We can also evaluate our model after training housing_model.evaluate(housing_train_x, housing_train_y)

13/13 [==============================] - 0s 3ms/step - loss: 10.2159 - mae: 2.3925

Out[ ]:

[10.215892791748047, 2.3924641609191895]

In [ ]:

  Copied!     
 
housing_model.evaluate(housing_test_x, housing_test_y)
housing_model.evaluate(housing_test_x, housing_test_y)

4/4 [==============================] - 0s 6ms/step - loss: 33.1137 - mae: 3.8703

Out[ ]:

[33.11373519897461, 3.8702917098999023]

4. Neural Networks for Image Datasets¶

In computer programing images are represented as 3D array where the dimensions are width, height, 3. The 3 at the end represents the RGB (red, green, blue) value of the colour at that pixel.

As these images can often be large, we don't normally use dense layers on them directly. We first simplify the images using convolutional and pooling layers. The goal of these layers is to look at sections of an image (groups of pixels) and then condensing this group into a single value. Below are graphic representations of how these layers work.

Convolutional layers work by multiplying the group of pixels by an array (also called a filter) of numbers and then adding them together. Like Dense layers the value of these arrays are learnt while training. We often multiply a group by multiple pixels and save all the results.

Pooling layers work by finding a metric such as the mean, max or minimum value of the pixels in this group.

4.1 Convolutional Layer Format¶

4.2 Pooling Layer Format¶

4.3 Overall Model Structure¶

4.4 Cifar 10 Image Dataset¶

This is an image dataset with 10 classes of images of common pictures such as dogs or cats. Click on the link to the documentation to learn more cifar 10.

NN meme 4.jpg

In [ ]:

  Copied!     
 
cf_10_train_data, cf_10_test_data = tf.keras.datasets.cifar10.load_data()

cf_10_train_x, cf_10_train_y = cf_10_train_data

cf_10_test_x, cf_10_test_y = cf_10_test_data

cf_10_train_y = tf.one_hot(cf_10_train_y.reshape((-1, )), 10)
cf_10_test_y = tf.one_hot(cf_10_test_y.reshape((-1, )), 10)
cf_10_train_data, cf_10_test_data = tf.keras.datasets.cifar10.load_data() cf_10_train_x, cf_10_train_y = cf_10_train_data cf_10_test_x, cf_10_test_y = cf_10_test_data cf_10_train_y = tf.one_hot(cf_10_train_y.reshape((-1, )), 10) cf_10_test_y = tf.one_hot(cf_10_test_y.reshape((-1, )), 10)

In [ ]:

  Copied!     
 
print(cf_10_train_x.shape, cf_10_train_y.shape)
print(cf_10_test_x.shape, cf_10_test_y.shape)
print(cf_10_train_x.shape, cf_10_train_y.shape) print(cf_10_test_x.shape, cf_10_test_y.shape)

(50000, 32, 32, 3) (50000, 10)
(10000, 32, 32, 3) (10000, 10)

4.5 Building a CNN Model¶

In [ ]:

  Copied!     
 
cf_10_model = tf.keras.models.Sequential()

cf_10_model.add(tf.keras.layers.Input(shape = (32, 32, 3))) # input shape for our images


# Convolutional layer with 16 filters each of size (3, 3) and a relu activation function
cf_10_model.add(tf.keras.layers.Conv2D(16, (3, 3), activation='relu'))
cf_10_model.add(tf.keras.layers.MaxPooling2D())

# Convolutional layer with 32 filters each of size (3, 3) and a relu activation function

cf_10_model.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu'))
cf_10_model.add(tf.keras.layers.MaxPooling2D())

# Convert the data from 3 dimentions to 1 dimention
cf_10_model.add(tf.keras.layers.Flatten())
# Hidden Dense Layer
cf_10_model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu))

# Output Layer 
# 10 outputs, one for each class as the model will output the probability of each individual class
cf_10_model.add(tf.keras.layers.Dense(10, activation = tf.nn.softmax))

cf_10_model.summary()
cf_10_model = tf.keras.models.Sequential() cf_10_model.add(tf.keras.layers.Input(shape = (32, 32, 3))) # input shape for our images # Convolutional layer with 16 filters each of size (3, 3) and a relu activation function cf_10_model.add(tf.keras.layers.Conv2D(16, (3, 3), activation='relu')) cf_10_model.add(tf.keras.layers.MaxPooling2D()) # Convolutional layer with 32 filters each of size (3, 3) and a relu activation function cf_10_model.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu')) cf_10_model.add(tf.keras.layers.MaxPooling2D()) # Convert the data from 3 dimentions to 1 dimention cf_10_model.add(tf.keras.layers.Flatten()) # Hidden Dense Layer cf_10_model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu)) # Output Layer # 10 outputs, one for each class as the model will output the probability of each individual class cf_10_model.add(tf.keras.layers.Dense(10, activation = tf.nn.softmax)) cf_10_model.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 30, 30, 16)        448       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 15, 15, 16)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 13, 13, 32)        4640      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 6, 6, 32)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense_4 (Dense)             (None, 64)                73792     
                                                                 
 dense_5 (Dense)             (None, 10)                650       
                                                                 
=================================================================
Total params: 79,530
Trainable params: 79,530
Non-trainable params: 0
_________________________________________________________________

In [ ]:

  Copied!     
 
tf.keras.utils.plot_model(cf_10_model)
tf.keras.utils.plot_model(cf_10_model)

Out[ ]:

4.6 Compiling the Model¶

We use categorical crossentropy to calculate loss for many classes, Adam for optimization and use Accuracy as a metric so we can understand how well the model is performing.

In [ ]:

  Copied!     
 
cf_10_model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(0.001), metrics = ['Acc'])
cf_10_model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(0.001), metrics = ['Acc'])

4.7 Training the Model¶

In [ ]:

  Copied!     
 
history = cf_10_model.fit(cf_10_train_x, cf_10_train_y, epochs = 20, batch_size = 1024,
                validation_data = (cf_10_test_x, cf_10_test_y))
history = cf_10_model.fit(cf_10_train_x, cf_10_train_y, epochs = 20, batch_size = 1024, validation_data = (cf_10_test_x, cf_10_test_y))

Epoch 1/20
49/49 [==============================] - 4s 26ms/step - loss: 13.5017 - Acc: 0.1346 - val_loss: 2.6569 - val_Acc: 0.1639
Epoch 2/20
49/49 [==============================] - 1s 15ms/step - loss: 2.2346 - Acc: 0.2025 - val_loss: 2.0436 - val_Acc: 0.2622
Epoch 3/20
49/49 [==============================] - 1s 15ms/step - loss: 1.9384 - Acc: 0.3068 - val_loss: 1.8320 - val_Acc: 0.3497
Epoch 4/20
49/49 [==============================] - 1s 16ms/step - loss: 1.7267 - Acc: 0.3842 - val_loss: 1.7202 - val_Acc: 0.3964
Epoch 5/20
49/49 [==============================] - 1s 15ms/step - loss: 1.5805 - Acc: 0.4358 - val_loss: 1.5615 - val_Acc: 0.4397
Epoch 6/20
49/49 [==============================] - 1s 15ms/step - loss: 1.4846 - Acc: 0.4724 - val_loss: 1.4933 - val_Acc: 0.4700
Epoch 7/20
49/49 [==============================] - 1s 16ms/step - loss: 1.4071 - Acc: 0.5024 - val_loss: 1.4392 - val_Acc: 0.4926
Epoch 8/20
49/49 [==============================] - 1s 16ms/step - loss: 1.3645 - Acc: 0.5201 - val_loss: 1.4387 - val_Acc: 0.4934
Epoch 9/20
49/49 [==============================] - 1s 16ms/step - loss: 1.3254 - Acc: 0.5341 - val_loss: 1.3899 - val_Acc: 0.5163
Epoch 10/20
49/49 [==============================] - 1s 16ms/step - loss: 1.2808 - Acc: 0.5506 - val_loss: 1.3951 - val_Acc: 0.5074
Epoch 11/20
49/49 [==============================] - 1s 14ms/step - loss: 1.2438 - Acc: 0.5625 - val_loss: 1.3350 - val_Acc: 0.5327
Epoch 12/20
49/49 [==============================] - 1s 15ms/step - loss: 1.2046 - Acc: 0.5775 - val_loss: 1.3381 - val_Acc: 0.5369
Epoch 13/20
49/49 [==============================] - 1s 15ms/step - loss: 1.1844 - Acc: 0.5837 - val_loss: 1.3103 - val_Acc: 0.5491
Epoch 14/20
49/49 [==============================] - 1s 15ms/step - loss: 1.1450 - Acc: 0.5998 - val_loss: 1.3172 - val_Acc: 0.5510
Epoch 15/20
49/49 [==============================] - 1s 15ms/step - loss: 1.1243 - Acc: 0.6051 - val_loss: 1.2908 - val_Acc: 0.5581
Epoch 16/20
49/49 [==============================] - 1s 15ms/step - loss: 1.0974 - Acc: 0.6171 - val_loss: 1.2791 - val_Acc: 0.5667
Epoch 17/20
49/49 [==============================] - 1s 15ms/step - loss: 1.0740 - Acc: 0.6233 - val_loss: 1.2763 - val_Acc: 0.5671
Epoch 18/20
49/49 [==============================] - 1s 15ms/step - loss: 1.0558 - Acc: 0.6315 - val_loss: 1.2750 - val_Acc: 0.5677
Epoch 19/20
49/49 [==============================] - 1s 15ms/step - loss: 1.0383 - Acc: 0.6387 - val_loss: 1.2554 - val_Acc: 0.5732
Epoch 20/20
49/49 [==============================] - 1s 15ms/step - loss: 1.0196 - Acc: 0.6433 - val_loss: 1.2680 - val_Acc: 0.5788

4.8 Evaluating the Model¶

In [ ]:

  Copied!     
 
history = history.history
history = history.history

In [ ]:

  Copied!     
 
print(history.keys())
print(history.keys())

dict_keys(['loss', 'Acc', 'val_loss', 'val_Acc'])

In [ ]:

  Copied!     
 
plt.plot(range(1, 21), history['loss'])
plt.plot(range(1, 21), history['val_loss'])

plt.title('Crossentropy Loss While Training')
plt.xlabel('Epoch Number')
plt.ylabel('Crossentropy Loss')
plt.legend(['Training Loss', 'Testing Loss'])
plt.show()
plt.plot(range(1, 21), history['loss']) plt.plot(range(1, 21), history['val_loss']) plt.title('Crossentropy Loss While Training') plt.xlabel('Epoch Number') plt.ylabel('Crossentropy Loss') plt.legend(['Training Loss', 'Testing Loss']) plt.show()

In [ ]:

  Copied!     
 
plt.plot(range(1, 21), history['Acc'])
plt.plot(range(1, 21), history['val_Acc'])

plt.title('Accuracy While Training')
plt.xlabel('Epoch Number')
plt.ylabel('Accuracy')
plt.legend(['Training Accuracy', 'Testing Accuracy'])

plt.show()
plt.plot(range(1, 21), history['Acc']) plt.plot(range(1, 21), history['val_Acc']) plt.title('Accuracy While Training') plt.xlabel('Epoch Number') plt.ylabel('Accuracy') plt.legend(['Training Accuracy', 'Testing Accuracy']) plt.show()

5. Transfer Learning¶

Images can be varied and complex, large image datasets with 1000+ classes can take hours or days to train a model with. However, the learned numeric values and structure of convolutional and pooling layers can be very similar even with different datasets. For this reason we use a technique known as transfer learning which is where we take the convolutional and pooling layers of a large model trained by professionals, add new dense layers and apply them to our dataset.

This can often give us much higher accuracy with fewer epochs.

5.1 Downloading a Model¶

For this dataset we download the RegNetX160 model and tell it to expect an input size of (32, 32, 3). We also set it's trainable attribute to false as we do not want to re-train it, only to use the model.

To fully utilize this potential this is a link to a list with names and information regarding each model in the keras library. If you are looking for larger, more complex or different problem types (such as text, video, audio, etc) look at models in the Tensorflow Hub site with a more exhaustive library of Neural Network models.

In [ ]:

  Copied!     
 
base_model = tf.keras.applications.RegNetX160(input_shape=(32, 32, 3), 
                                               include_top=False,
                                               weights='imagenet')


base_model.trainable = False
base_model = tf.keras.applications.RegNetX160(input_shape=(32, 32, 3), include_top=False, weights='imagenet') base_model.trainable = False

In [ ]:

  Copied!     
 
# We can plot the structure of this model
tf.keras.utils.plot_model(base_model)
# We can plot the structure of this model tf.keras.utils.plot_model(base_model)

Out[ ]:

5.2 Creating a New Model using the Pretrained Model¶

In [ ]:

  Copied!     
 
new_cf_10_model = tf.keras.models.Sequential()
new_cf_10_model.add(tf.keras.layers.Input(shape = (32, 32, 3)))

# Adding the pre-trained model
new_cf_10_model.add(base_model)

new_cf_10_model.add(tf.keras.layers.Flatten())

new_cf_10_model.add(tf.keras.layers.Dense(512, activation = tf.nn.relu))
new_cf_10_model.add(tf.keras.layers.Dense(256, activation = tf.nn.relu))
new_cf_10_model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
new_cf_10_model.add(tf.keras.layers.Dense(10, activation = tf.nn.softmax))

new_cf_10_model.summary()
new_cf_10_model = tf.keras.models.Sequential() new_cf_10_model.add(tf.keras.layers.Input(shape = (32, 32, 3))) # Adding the pre-trained model new_cf_10_model.add(base_model) new_cf_10_model.add(tf.keras.layers.Flatten()) new_cf_10_model.add(tf.keras.layers.Dense(512, activation = tf.nn.relu)) new_cf_10_model.add(tf.keras.layers.Dense(256, activation = tf.nn.relu)) new_cf_10_model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu)) new_cf_10_model.add(tf.keras.layers.Dense(10, activation = tf.nn.softmax)) new_cf_10_model.summary()

Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 regnetx160 (Functional)     (None, 1, 1, 2048)        52340704  
                                                                 
 flatten_1 (Flatten)         (None, 2048)              0         
                                                                 
 dense_6 (Dense)             (None, 512)               1049088   
                                                                 
 dense_7 (Dense)             (None, 256)               131328    
                                                                 
 dense_8 (Dense)             (None, 128)               32896     
                                                                 
 dense_9 (Dense)             (None, 10)                1290      
                                                                 
=================================================================
Total params: 53,555,306
Trainable params: 1,214,602
Non-trainable params: 52,340,704
_________________________________________________________________

In [ ]:

  Copied!     
 
tf.keras.utils.plot_model(new_cf_10_model)
tf.keras.utils.plot_model(new_cf_10_model)

Out[ ]:

5.3 Compile the Model¶

In [ ]:

  Copied!     
 
new_cf_10_model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(0.001), metrics = ['Acc'])
new_cf_10_model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(0.001), metrics = ['Acc'])

5.4 Training the Model¶

In [ ]:

  Copied!     
 
history = new_cf_10_model.fit(cf_10_train_x, cf_10_train_y, epochs = 5, 
                              batch_size = 4096,
                              validation_data = (cf_10_test_x, cf_10_test_y))
history = new_cf_10_model.fit(cf_10_train_x, cf_10_train_y, epochs = 5, batch_size = 4096, validation_data = (cf_10_test_x, cf_10_test_y))

Epoch 1/5
13/13 [==============================] - 68s 4s/step - loss: 2.0653 - Acc: 0.3782 - val_loss: 1.1724 - val_Acc: 0.5880
Epoch 2/5
13/13 [==============================] - 31s 2s/step - loss: 1.0604 - Acc: 0.6350 - val_loss: 0.9637 - val_Acc: 0.6685
Epoch 3/5
13/13 [==============================] - 31s 2s/step - loss: 0.8973 - Acc: 0.6910 - val_loss: 0.8570 - val_Acc: 0.7048
Epoch 4/5
13/13 [==============================] - 31s 2s/step - loss: 0.8084 - Acc: 0.7184 - val_loss: 0.8088 - val_Acc: 0.7181
Epoch 5/5
13/13 [==============================] - 31s 2s/step - loss: 0.7579 - Acc: 0.7360 - val_loss: 0.7665 - val_Acc: 0.7323

5.5 Evaluation¶

In [ ]:

  Copied!     
 
history = history.history
history = history.history

In [ ]:

  Copied!     
 
plt.plot(range(1, 6), history['loss'])
plt.plot(range(1, 6), history['val_loss'])

plt.title('Crossentropy Loss While Training')
plt.xlabel('Epoch Number')
plt.ylabel('Crossentropy Loss')
plt.legend(['Training Loss', 'Testing Loss'])
plt.show()
plt.plot(range(1, 6), history['loss']) plt.plot(range(1, 6), history['val_loss']) plt.title('Crossentropy Loss While Training') plt.xlabel('Epoch Number') plt.ylabel('Crossentropy Loss') plt.legend(['Training Loss', 'Testing Loss']) plt.show()

In [ ]:

  Copied!     
 
plt.plot(range(1, 6), history['Acc'])
plt.plot(range(1, 6), history['val_Acc'])

plt.title('Accuracy While Training')
plt.xlabel('Epoch Number')
plt.ylabel('Accuracy')
plt.legend(['Training Accuracy', 'Testing Accuracy'])

plt.show()
plt.plot(range(1, 6), history['Acc']) plt.plot(range(1, 6), history['val_Acc']) plt.title('Accuracy While Training') plt.xlabel('Epoch Number') plt.ylabel('Accuracy') plt.legend(['Training Accuracy', 'Testing Accuracy']) plt.show()

6. Try It Yourself¶

Below are some more datasets I have loaded for you to try and train models on. See how far you can reach!

In [ ]:

  Copied!     
 
import pandas as pd # used to read the csv dataset file
import pandas as pd # used to read the csv dataset file

6.1 California Housing (Easy)¶

Train a regression model on this dataset.

In [ ]:

  Copied!     
 
ch_train_data = pd.read_csv("/content/sample_data/california_housing_train.csv")

ch_train_data_x = ch_train_data.drop("median_house_value", axis = 1).to_numpy()
ch_train_data_y = ch_train_data["median_house_value"].to_numpy()

ch_train_data
ch_train_data = pd.read_csv("/content/sample_data/california_housing_train.csv") ch_train_data_x = ch_train_data.drop("median_house_value", axis = 1).to_numpy() ch_train_data_y = ch_train_data["median_house_value"].to_numpy() ch_train_data

Out[ ]:

	longitude	latitude	housing_median_age	total_rooms	total_bedrooms	population	households	median_income	median_house_value
0	-114.31	34.19	15.0	5612.0	1283.0	1015.0	472.0	1.4936	66900.0
1	-114.47	34.40	19.0	7650.0	1901.0	1129.0	463.0	1.8200	80100.0
2	-114.56	33.69	17.0	720.0	174.0	333.0	117.0	1.6509	85700.0
3	-114.57	33.64	14.0	1501.0	337.0	515.0	226.0	3.1917	73400.0
4	-114.57	33.57	20.0	1454.0	326.0	624.0	262.0	1.9250	65500.0
...	...	...	...	...	...	...	...	...	...
16995	-124.26	40.58	52.0	2217.0	394.0	907.0	369.0	2.3571	111400.0
16996	-124.27	40.69	36.0	2349.0	528.0	1194.0	465.0	2.5179	79000.0
16997	-124.30	41.84	17.0	2677.0	531.0	1244.0	456.0	3.0313	103600.0
16998	-124.30	41.80	19.0	2672.0	552.0	1298.0	478.0	1.9797	85800.0
16999	-124.35	40.54	52.0	1820.0	300.0	806.0	270.0	3.0147	94600.0

17000 rows × 9 columns

In [ ]:

  Copied!     
 
ch_test_data = pd.read_csv("/content/sample_data/california_housing_test.csv")

ch_test_data_x = ch_test_data.drop("median_house_value", axis = 1).to_numpy()
ch_test_data_y = ch_test_data["median_house_value"].to_numpy()
ch_test_data = pd.read_csv("/content/sample_data/california_housing_test.csv") ch_test_data_x = ch_test_data.drop("median_house_value", axis = 1).to_numpy() ch_test_data_y = ch_test_data["median_house_value"].to_numpy()

In [ ]:

  Copied!     
 
print(ch_train_data_x.shape, ch_train_data_y.shape)
print(ch_train_data_x.shape, ch_train_data_y.shape)

(17000, 8) (17000,)

In [ ]:

  Copied!     
 
## Create your model here
## Create your model here

In [ ]:

  Copied!     
 
## Compile your model here
## Compile your model here

In [ ]:

  Copied!     
 
## Train your model here
## Train your model here

6.2 Pre-trained Cifar 100 (Medium)¶

We now use a larger version of the cifar data set with 100 different classes. In this section try to get a good accuracy score using a pretrained model. Here is the link to the dataset site : Cifar 100

You can use the same base model as we used before, but I encourage you to try another one and see how it works. You can even try to find models from Tensorflow Hub.

In [ ]:

  Copied!     
 
cf_100_train_data, cf_100_test_data = tf.keras.datasets.cifar100.load_data()

cf_100_train_x, cf_100_train_y = cf_100_train_data

cf_100_test_x, cf_100_test_y = cf_100_test_data

cf_100_train_y = tf.one_hot(cf_100_train_y.reshape((-1, )), 100)
cf_100_test_y = tf.one_hot(cf_100_test_y.reshape((-1, )), 100)
cf_100_train_data, cf_100_test_data = tf.keras.datasets.cifar100.load_data() cf_100_train_x, cf_100_train_y = cf_100_train_data cf_100_test_x, cf_100_test_y = cf_100_test_data cf_100_train_y = tf.one_hot(cf_100_train_y.reshape((-1, )), 100) cf_100_test_y = tf.one_hot(cf_100_test_y.reshape((-1, )), 100)

In [ ]:

  Copied!     
 
print(cf_100_train_x.shape, cf_100_train_y.shape)
print(cf_100_test_x.shape, cf_100_test_y.shape)
print(cf_100_train_x.shape, cf_100_train_y.shape) print(cf_100_test_x.shape, cf_100_test_y.shape)

(50000, 32, 32, 3) (50000, 100)
(10000, 32, 32, 3) (10000, 100)

In [ ]:

  Copied!     
 
## Import a model here
## Import a model here

In [ ]:

  Copied!     
 
## Create your model here
## Create your model here

In [ ]:

  Copied!     
 
## Compile your model here
## Compile your model here

In [ ]:

  Copied!     
 
## Train your model here
## Train your model here

6.3 From Scratch Cifar 100 (Hard)¶

Here you are expected to write a CNN from scratch to train on the cifar 100 dataset. Note that this model will need to be larger than the one used on the cifar 10 dataset as this is more complex.

In [ ]:

  Copied!     
 
cf_100_train_data, cf_100_test_data = tf.keras.datasets.cifar100.load_data()

cf_100_train_x, cf_100_train_y = cf_100_train_data

cf_100_test_x, cf_100_test_y = cf_100_test_data

cf_100_train_y = tf.one_hot(cf_100_train_y.reshape((-1, )), 100)
cf_100_test_y = tf.one_hot(cf_100_test_y.reshape((-1, )), 100)
cf_100_train_data, cf_100_test_data = tf.keras.datasets.cifar100.load_data() cf_100_train_x, cf_100_train_y = cf_100_train_data cf_100_test_x, cf_100_test_y = cf_100_test_data cf_100_train_y = tf.one_hot(cf_100_train_y.reshape((-1, )), 100) cf_100_test_y = tf.one_hot(cf_100_test_y.reshape((-1, )), 100)

In [ ]:

  Copied!     
 
print(cf_100_train_x.shape, cf_100_train_y.shape)
print(cf_100_test_x.shape, cf_100_test_y.shape)
print(cf_100_train_x.shape, cf_100_train_y.shape) print(cf_100_test_x.shape, cf_100_test_y.shape)

(50000, 32, 32, 3) (50000, 100)
(10000, 32, 32, 3) (10000, 100)

In [ ]:

  Copied!     
 
## Create your model here
## Create your model here

In [ ]:

  Copied!     
 
## Compile your model here
## Compile your model here

In [ ]:

  Copied!     
 
## Train your model here
## Train your model here

Hope You Enjoyed Learning Neural Networks¶

NN meme 3jpg.png