Deep Learning and Neural Network with kero 1. PART 1: preparing data 2. PART 2: train model and plot its progress

*kero version 0.6.2*

Hi! Neural network has been incorporated into python package kero, in version 0.6.2. Let’s continue with our attempts to implement neural network. Last time, we collected all the essential formulas here. Now, we will

- manually initiate the following simple 4-layered neural network,
- simulate some training data,
- and plot the progress of this training.

The plan in the immediate future is to use kero for image classification.

*Note: this is still at an experimental phase. User interface will not be very friendly yet 😦*

### Create some data points

We begin by creating some data points using **create_some_data_points.py **from here. I used the setting as shown below. The variable *input_dimension=3* because the Input Layer has 3 neurons. I created 24 data points and saved them into *some_data_points.txt*.

no_of_data_points = 24 input_dimension = 3 mean_value = 0.5 standard_dev = 0.1

The first 3 data saved are shown below (the full data consists of 24 lines like these).

0.42339179422571943 0.6040855122193824 0.5728640160551863 0.4290660956760066 0.5640408919216678 0.5586055127768119 0.5556853535965329 0.40871367426190885 0.5131828451081797 ...

### Manually Initiate a Neural Network

In **testNNupdater2aux.py**:

- PART 1 manually sets the weights and biases of the neural network and create a NeuralNetwork object.
- PART 2 loads the data points from the previous section into the correct format. Let us call the collection of input
*input_set={x1,x2,…x24}*. - PART 3 needs a little bit attention. Using the neural network manually initiated, compute the predicted output, which will be called Y_set={y1,y2,…,y24}. Each yk is the predicted output of each xk. And then we
**arbitrarily**make the true output, called*Y0_set={y0_1,y0_2,…,y0_24}*.

Yes, PART 3 is arbitrary. We are just making a random simulation. The **objective**: given *input_set* and *Y0_set*, then we know that each *xk* has a true output *y0_k*. However, our initial neural network predicts that *xk* will output *yk, *which is not correct. We want to train our neural network model so that, as the neural network learns the pattern, each *xk* will produce output that will be closer and closer to *yk, *hopefully, eventually it will yield the right output.

You can set *print_data=True* to print the data we have just simulated.

Great! Now we have the training data, let us proceed with the training in PART 2.

**testNNupdater2aux.py**

import kero.multib.NeuralNetwork as nn import kero.utils.utils as ut import numpy as np def prep(print_data=False): # ----------------- PART 1 ----------------- # create some neural network with manually initiated weights and biases #---------------------------------------- # weights : the collection of weights in the neural network # weights is a list [w_l], where w_l is the collection of weights between # the (l-1)-th and l-th layer # for l=2,3,...,L where l=1 is the input layer, l=2 the first hidden layer # and l=L is the output layer # w_l is a matrix (list of list) # so that w_l[i][j] is the weight between neuron j at layer l-1 and # neuron i at layer l # biases : the collection of biases in the neural network # biases is a list [b_l], where b_l is the collection of biases in the l-th layer # for l=2,3,...,L # arbitary choices arb0 = [0.1,0.1,0.1] arb1 = [0.1,0.2,0.3] arb2 = [0.4,0.5,0.6] arb3 = [-0.1,-0.1,-0.1] # just to show negative weight is okay # Weights and biases # Input layer - Hidden layer 1 # (layer 1 - layer 2) # ------------------------------------- # An Example: w_2[0][1] == 0.2. This means the weight between # between neuron 2 of the input layer and neuron 1 of hidden layer 1 is 0.2 # Note that w_2 is 3x3 matrix. Input layer and hidden layer 1 both have 3 neurons w_2 = [arb1, arb2, arb3] b_2 = [0, 0, 0] # Hidden layer 1 - Hidden layer 2 # (layer 2 - layer 3) # -------------------------------------- # w_3 is a 2x3 matrix. # Hidden layer 1 (layer 2) have 3 neurons # Hidden layer 2 (layer 3) have 2 neurons w_3 = [arb0, arb0] b_3 = [0,0] # Hidden layer 2 - Output layer # (layer 3 - layer 4) w_4 = [[0.1,0.1],[0.1,0.1]] b_4 = [0,0.1] net1=nn.NeuralNetwork() weights = [w_2,w_3,w_4] biases = [b_2,b_3,b_4] bulk={ "weights" : weights, "biases" : biases } net1.initiate_neural_network(bulk) net1.learning_rate=1e-3 # ----------------- PART 2 ----------------- # load some input data points as input_set txt = open("some_data_points.txt","r") temp = txt.read().split("\n") txt.close() input_set = [] for i in range(len(temp)): temp2 = temp[i].split(" ") temp3 = [float(x) for x in temp2 if x is not ''] # print(i," : ", temp[i].split(" ")) # print(i," : ", temp3) if len(temp3)>0: input_set.append(temp3) # for x in input_set: # for y in x: # print(y,end='\t\t') # print("") # convert the input set to numpy matrix, as required by our own convention input_set = [np.transpose(np.matrix(x)) for x in input_set] # ----------------- PART 3 ----------------- # Create some simulation of true/observed output Y0_set and NN computed output Y_set corresponding to each data point from input_set # nnexample = nn.NetworkUpdater() AF = nn.activationFunction(func = "Sigmoid") Y_set = [] Y0_set = [] collection_of_fed_forward_a_l = [] collection_of_fed_forward_z_l = [] # ------------------------------------------- # this is a collection of a_l_set and z_l_set over all data points # z_l_set is the collection of z values over all layers, l=2,3,...L # and a_l_set is the corresponding activated values for a_1 in input_set: # Recall: a_l_set and z_l_set each is a list of numpy matrices a_l_set, z_l_set = nnexample.feed_forward(weights, biases, a_1, AF, verbose=False, matrix_formatting="%6.2f") Y = a_l_set[-1] # numpy matrix Y_set.append(Y) Y0_set.append(np.random.normal(0,1)*Y) collection_of_fed_forward_a_l.append(a_l_set) collection_of_fed_forward_z_l.append(z_l_set) if print_data: count = 1 for a_1, a_L, Y0 in zip(input_set, Y_set, Y0_set): print("input ", count, ":") ut.print_numpy_matrix(a_1,formatting="%6.2f",no_of_space=10) print("a_L or Y: ") ut.print_numpy_matrix(a_L,formatting="%6.2f",no_of_space=10) print("Y0: ") ut.print_numpy_matrix(Y0,formatting="%6.2f",no_of_space=10) count = count + 1 print("---------------------------------") out={ "input_set":input_set, "Y_set":Y_set, "Y0_set": Y0_set, "collection_of_fed_forward_a_l" : collection_of_fed_forward_a_l, "collection_of_fed_forward_z_l" : collection_of_fed_forward_z_l, "weights" : weights, "biases" : biases, "NeuralNetwork" : net1 } return out print("---aux---") out = prep(print_data=False)