Unleash Machine Learning: Build Artificial Neuron in Python

STRUCTURE and FUNCTION of biological neuron

A neuron is a cell that processes and transmits information through electrical and chemical signals.

Soma Greek for body

Dendrite - Greek for tree

Nucleus

Axon

Axon hillock

Myelin sheath

Schwann cell

Nodes of Ranvier

Synapse

Chemical synapse

input X

output Y

weight W

sum inputs

activation function g: step, relu, logistic

Step by step calculation of the output given input and weights and the activation function

What logical function is it?

Put the results in a table.

Draw the results.

Observe that the outputs are linearly separable.

Student should solve this exercise on his/her own.

You can simplify the representation of a neuron if you transform the threshold into a bias input with the value 1 and a trainable bias weight.

One neuron is different from another neuron if it has different weights.

With different weights a neuron behaves differently and can approximate a different function.

Neurons draw decision boundaries...literaly.

The weight vector W = [w1 w2] is actually perpendicular to the decision boundary.

Perceptron learning algorithm:

1. initialize W randomly

2. pick a sample pattern p and compute the output y

3. if y != target change W

4. repeat 2,3

Do we want to design it using OOP pricinciples? Detailing all core objects and relationships between them for example:

• a neuron class, a synapse class, a signal class, an interface for different activation functions, a layer class, the entire network as a collection on layers

OR

• Do we want a terse, brain twisting, mathematical, matrix formulation that does everything in under 10 lines of code?

I have decided to pick an in-between form that I believe to be simple, clean and clear.

We will have 2 classes:

• a class represing the entire artificial neuron network
• a class represing a single layer of neurons

Layer:

self.id = id

self.n_neurons = layer_size

self.bias_val = 1

self.input = [0] * self.n_neurons

self.output = [0] * (self.n_neurons + use_bias)

self.output[0] = self.bias_val

self.error = [0] * self.n_neurons

self.weight = make_matrix(prev_layer_size + use_bias, self.n_neurons)

def predict(self, input):

"""

Return the network prediction for this input.

"""

self.set_input(input)

self.forward_propagate()

return self.get_output()

def train(self, inputs, targets, n_epochs):

• self.set_input(inputs[i])
• self.forward_propagate()
• self.update_error_output(targets[i])
• 
  
• self.backward_propagate()
• self.update_weights()

def backward_propagate(self):

"""

Backprop. Propagate the error from the output layer backwards to the input layer.

"""

run for xor - why doesnt it work with just 2 layers?

def hyperbolic_tangent(x):

• return math.tanh(x)

def deriv_hyperbolic_tangent(x):

• th = math.tanh(x)
• return 1 - th * th

Refactor the activation function to make the code clear.

good_range = 1.0 / math.sqrt(prev_layer_size + 1)

self.weight[i][j] = between(-good_range, good_range)

Intuition using visual decision boundary explaining why xor can't be solved with 1 output neuron but requires 2 extra hidden neurons.

Power:

• translate english to chinese is a function
• output driving commands given pixels or radar information is a function
• suggest what the customer should buy given previous purchasees and navigation history is a function

Idea1: you can create a step function

Idea2: you can aproximate any function using steps

Use a strange function:

def f(x):

• return 0.3*x*x + 2*np.sin(x) - 2*x + 0.5

Use our ANN as a function approximator to match the given function.

from sklearn.datasets import load_iris

iris = load_iris()

iris.feature_names

iris.target_names

• each row: sample, example, observation, record, instance
• each column: feature, predictor, attribute, independent variable, input, regressor, covariant
• target: respose, outcome, label, dependent variable

xsc = MinMaxScaler(feature_range=(0, 1), copy=True)

xsc.fit(X)

ylb = LabelBinarizer()

X_train, X_test, y_train, y_test = train_test_split(xsc.transform(X), y)

nn.train(X_train_l, y_train_l, 5000)

preds = np.array([nn.predict(record) for record in X_test_l])

confusion_matrix(y_test, ypred)

classification_report(y_test, ypred)

X = digits.data

y = digits.target

Extend dataset

print('addind more images by moving the original images up, left, right, down by 1 pixel')

nn.train(X_train, Y_train, 10)

visualize the confusion matrix

# see bad predictions

failed = y_pred != y_test

X2_test = X_test[failed]

y2_pred = y_pred[failed]

y2_test = y_test[failed]

print_digits(X2_test, y2_test, y2_pred)

def serialize(nn, fname):

with open(fname, 'wb') as f:

• # use protocol 2, default 0 has problems
• pickle.dump(nn, f, 2)

def deserialize(fname):

with open(fname, 'rb') as f:

• nn = pickle.load(f)
• return nn

# The MNIST database of handwritten digits, available from this page,

# has a training set of 60,000 examples, and a test set of 10,000 examples.

# 28 x 28 pixels

# load a nn model ex: /models/nn_mnist_iter1600000.pickle

model_fname = 'nn_mnist_iter1600000.pickle'

nn = ann_util.deserialize('models/' + model_fname)

https://aiception.com/ deep learning platform that offers AI as a service

http://www.aicrafters.com/ educational Artificial Intelligence courses