Deep Learning and Neural Network with kero PART 1

home > Machine Learning

Deep Learning and Neural Network with kero
1. PART 1: preparing data
2. PART 2: train model and plot its progress

kero version 0.6.2

Hi! Neural network has been incorporated into python package kero, in version 0.6.2. Let’s continue with our attempts to implement neural network. Last time, we collected all the essential formulas here. Now, we will

  1. manually initiate the following simple 4-layered neural network,
  2. simulate some training data,
  3. and plot the progress of this training.

The plan in the immediate future is to use kero for image classification.

Note: this is still at an experimental phase. User interface will not be very friendly yet 😦

nnimg

Create some data points

We begin by creating some data points using create_some_data_points.py from here. I used the setting as shown below. The variable input_dimension=3 because the Input Layer has 3 neurons. I created 24 data points and saved them into some_data_points.txt.

no_of_data_points = 24
input_dimension = 3
mean_value = 0.5
standard_dev = 0.1

The first 3 data saved are shown below (the full data consists of 24 lines like these).

0.42339179422571943 0.6040855122193824 0.5728640160551863 
0.4290660956760066 0.5640408919216678 0.5586055127768119 
0.5556853535965329 0.40871367426190885 0.5131828451081797 
...

Manually Initiate a Neural Network

In testNNupdater2aux.py:

  1. PART 1 manually sets the weights and biases of the neural network and create a NeuralNetwork object.
  2. PART 2 loads the data points from the previous section into the correct format. Let us call the collection of input input_set={x1,x2,…x24}.
  3. PART 3 needs a little bit attention. Using the neural network manually initiated, compute the predicted output, which will be called Y_set={y1,y2,…,y24}. Each yk is the predicted output of each xk. And then we arbitrarily make the true output, called Y0_set={y0_1,y0_2,…,y0_24}.

Yes, PART 3 is arbitrary. We are just making a random simulation. The objective: given input_set and Y0_set, then we know that each xk has a true output y0_k. However, our initial neural network predicts that xk will output yk, which is not correct. We want to train our neural network model so that, as the neural network learns the pattern, each xk will produce output that will be closer and closer to yk, hopefully, eventually it will yield the right output.

You can set print_data=True to print the data we have just simulated.

Great! Now we have the training data, let us proceed with the training in PART 2.

testNNupdater2aux.py

import kero.multib.NeuralNetwork as nn
import kero.utils.utils as ut
import numpy as np


def prep(print_data=False):
	# ----------------- PART 1 ----------------- 
	# create some neural network with manually initiated weights and biases

	#----------------------------------------
	# weights : the collection of weights in the neural network
	#   weights is a list [w_l], where w_l is the collection of weights between 
	#     the (l-1)-th and l-th layer
	#     for l=2,3,...,L where l=1 is the input layer, l=2 the first hidden layer
	#     and l=L is the output layer
	#   w_l is a matrix (list of list)
	#     so that w_l[i][j] is the weight between neuron j at layer l-1 and 
	#     neuron i at layer l 
	# biases : the collection of biases in the neural network
	#   biases is a list [b_l], where b_l is the collection of biases in the l-th layer
	#     for l=2,3,...,L

	# arbitary choices
	arb0 = [0.1,0.1,0.1]
	arb1 = [0.1,0.2,0.3]
	arb2 = [0.4,0.5,0.6]
	arb3 = [-0.1,-0.1,-0.1] # just to show negative weight is okay

	# Weights and biases
	# Input layer - Hidden layer 1
	# (layer 1 - layer 2)
	# -------------------------------------
	# An Example: w_2[0][1] == 0.2. This means the weight between 
	#   between neuron 2 of the input layer and neuron 1 of hidden layer 1 is 0.2 
	# Note that w_2 is 3x3 matrix. Input layer and hidden layer 1 both have 3 neurons
	w_2 = [arb1, arb2, arb3] 
	b_2 = [0, 0, 0]
	# Hidden layer 1 - Hidden layer 2
	# (layer 2 - layer 3)
	# --------------------------------------
	# w_3 is a 2x3 matrix.
	#   Hidden layer 1 (layer 2) have 3 neurons 
	#   Hidden layer 2 (layer 3) have 2 neurons
	w_3 = [arb0, arb0]
	b_3 = [0,0]
	# Hidden layer 2 - Output layer
	# (layer 3 - layer 4)
	w_4 = [[0.1,0.1],[0.1,0.1]]
	b_4 = [0,0.1]

	net1=nn.NeuralNetwork()
	weights = [w_2,w_3,w_4]
	biases = [b_2,b_3,b_4]
	bulk={
		"weights" : weights,
		"biases" : biases
	}
	net1.initiate_neural_network(bulk)
	net1.learning_rate=1e-3
	# ----------------- PART 2 ----------------- 
	# load some input data points as input_set

	txt = open("some_data_points.txt","r")
	temp = txt.read().split("\n")
	txt.close()
	input_set = []
	for i in range(len(temp)):
		temp2 = temp[i].split(" ")
		temp3 = [float(x) for x in temp2 if x is not '']
		# print(i," : ", temp[i].split(" "))
		# print(i," : ", temp3)
		if len(temp3)>0:
			input_set.append(temp3)
	# for x in input_set:
	# 	for y in x:
	# 		print(y,end='\t\t')
	# 	print("")
	# convert the input set to numpy matrix, as required by our own convention
	input_set = [np.transpose(np.matrix(x)) for x in input_set]

	# ----------------- PART 3 ----------------- 
	# Create some simulation of true/observed output Y0_set and NN computed output Y_set corresponding to each data point from input_set
	# 
	nnexample = nn.NetworkUpdater()
	AF = nn.activationFunction(func = "Sigmoid")
	Y_set = []
	Y0_set = []

	collection_of_fed_forward_a_l = []
	collection_of_fed_forward_z_l = []
	# -------------------------------------------
	# this is a collection of a_l_set and z_l_set over all data points
	#   z_l_set is the collection of z values over all layers, l=2,3,...L
	#   and a_l_set is the corresponding activated values


	for a_1 in input_set:
		# Recall: a_l_set and z_l_set each is a list of numpy matrices 
		a_l_set, z_l_set = nnexample.feed_forward(weights, biases, a_1, AF, verbose=False, matrix_formatting="%6.2f")
		Y = a_l_set[-1] # numpy matrix
		Y_set.append(Y)
		Y0_set.append(np.random.normal(0,1)*Y)
		collection_of_fed_forward_a_l.append(a_l_set)
		collection_of_fed_forward_z_l.append(z_l_set)


	if print_data:
		count = 1
		for a_1, a_L, Y0 in zip(input_set, Y_set, Y0_set):
			print("input ", count, ":")
			ut.print_numpy_matrix(a_1,formatting="%6.2f",no_of_space=10)
			print("a_L or Y: ")
			ut.print_numpy_matrix(a_L,formatting="%6.2f",no_of_space=10)
			print("Y0: ")
			ut.print_numpy_matrix(Y0,formatting="%6.2f",no_of_space=10)
			count = count + 1
			print("---------------------------------")

	out={
		"input_set":input_set, 
		"Y_set":Y_set,
		"Y0_set": Y0_set,
		"collection_of_fed_forward_a_l" : collection_of_fed_forward_a_l,
		"collection_of_fed_forward_z_l" : collection_of_fed_forward_z_l, 
		"weights" : weights, 
		"biases" : biases,
		"NeuralNetwork" : net1
	}
	return out
print("---aux---")
out = prep(print_data=False)