Perceptron: Whats, whys, and hows

If one day we will tell our great-grandchildren the story of the artificial intelligence, it should perhaps begin with a perceptron.

Biological Introduction

A neuron is an electrically excitable cell that receives, processes, and transmits information through electrical and chemical signals.

Very simplified diagram of a neuron

The signal propagates from neuron to neuron by means of an axon, which connects them. The connection takes place between the axon terminals of the emitting neuron and the dendrites of the receiving neuron, in a structure called synapse.

When a neuron is at rest, i.e. it is not receiving any signal, it is subject to a resting electric potential , which on average is around -70 millivolt and it is related to the difference between electric charge inside and outside the neuron. 

When a cell sends an electric impulse , an electric potential related to it propagates through the axon and: it is called action potential .

The action potential propagates through the axon

Eventually, the action potential reaches the neuron, where the electric tension rapidly changes. If its value gets bigger than a certain threshold, the so-called firing process is triggered, consisting in the emission of a signal from the neuron itself.

There is no partial firing, that is, when the electric potential reaches the threshold, the neuron fires an electric signal whose intensity is independent of the received one.

Illustration of the electrical chemical signal crossing a synapse. ( Andrii Vodolazhskyi via Shutterstock )

About the intensity of the received electrical signal, we should mention two points:

1. The electric potential variation, to which the neuron is subject when it receives one or more signals, is given by the sum of the received action potentials.

2. The intensity of the signal received from neuron N3 is, in general, not the same of the signal sent from that neuron: while propagating through the axon, the intensity changes for a factor which depends on the thickness of the myelin sheath covering the axon. The thicker it is, the smaller is the dispersion of the signal.

A mathematical model for the biological neuron

We can formalize this process in a very simple mathematical model. We are going to assign a variable to each term you found in bold in the previous paragraph:

1. The electric impulse the neuron receives from N different cells is described by an N-dimensional vector x , the input vector .
2. The resting electric potential is the threshold . If the sum of the received action potentials gets to the threshold, the neuron fires. I will indicate the threshold value with the letter b. Sometimes, the threshold is called bias .
3. The thicker is the myelin sheath of the axon carrying the electric impulse, the higher is the received intensity. In fact, the myelin is an insulator which facilitates the conducting of the electrical impulse, reducing dispersion along in its path to the new neuron. The multiplicative factor, representing the thickness of the myelin sheath which covers the axon connecting two neurons, is expressed in our model by the parameter w, called weight . If we consider the model of a neuron receiving at each time-step N impulses through N different axons, we get a vector of N weights, w .
4. The sum of the weighted inputs and the bias is sometimes referred to as induced local field , and we will indicate it with the letter v:
   v = w ∙ x + b
5. The output of the neuron, which expresses the intensity of the fired signal, is indicated with y. It is a function (the activation function ) of v.

The Perceptron’s architecture

To recap: a neuron receives an N-dimensional input x , which is weighted with the weight vector w . If the induced local field, v = w ∙ x + b, is equal or bigger than zero, then the neuron fires a signal with fixed intensity, let’s say 1. Otherwise, it does not, that means that the intensity of the firing is 0.

We can formalize that with a simple formula, using as activation function the Heavyside step function ϴ:

I put w and b on the right of a semicolon to distinguish them from x : in fact, the former are parameters, the latter is the input vector.

Let’s implement this function with Python:

To be continued — Deep Learning Pills

This was the first of a series of articles about deep learning, which I call Deep Learning Pills .

In the next one, we will see what kind of problems a perceptron can solve, how it learns to do that and what are its limitations. I will publish it very soon, so stay tuned!

Also, feel free to get in touch with me on Linkedin and Quora .

If you liked this article, I hope you will consider clapping or commenting :)

See you very soon,

Frank