AutoEncoders in Keras

Photo by Aditya Chinchure on  Unsplash

Motivation

Many of the recent deep learning models rely on extracting complex features from input data. The goal is to transform the input from its raw format, to another representation calculated by the neural network. This representation contains hidden features that describe the unique characteristics of a given input.

Consider a dataset of people faces, where each input is an image of a person. The representation of an image in its raw format is too complex to be used by machines. Instead, why not make the neural network automatically calculate the following for each: eye type, nose type, distance between eyes, nose position, etc. Well, this sounds interesting… Using these features, we could easily compare two faces , find similar faces , generate new faces , and many other interesting things.

This concept is called Encoding since we are generating an encoded version of a given input. In this article we will learn how to calculate encodings using AutoEncoders .

AutoEncoders

An AutoEncoder is a strange neural network, because both its input and output are the same. So, it is a network that tries to learn itself! This is crazy I know but you will see why this is useful.

Suppose we have the following neural network:

• An input layer with 100 neurons
• A Hidden layer with 3 neurons
• An output layer with 100 neurons (same as the input layer)

Now what happens if we fit the neural network to predict the same value as in the input — Learn itself basically? Does not this mean that the network learned how to reconstruct the input from only 3 dimensions (number of neurons in the hidden layer) ? This also mean that now each input could be represented in 3 dimensions instead of 100. In addition, these 3 dimensions or features are enough to reconstruct the input again. Well this is very interesting. It is like when compressing files. We reduce the file size, but we can uncompress it again and get the same data. In fact, it is not exactly the same data in AutoEncoders since they are lossy, but you got the point.

Objective

We will use the famous MNIST digits dataset to demonstrate the idea. The goal is to generate a 2D encoding from a given 28*28 image. So, we are implementing a dimensionality reduction algorithm using AutoEncoders! Cool right? Let us start…

Time to Code

First, we import the dataset:

from keras.datasets import mnist
(data, labels), (_, _) = mnist.load_data()

Need to reshape and rescale:

data = data.reshape(-1, 28*28) / 255.

Time to define the network. We need three layers:

• an input layer with size 28*28
• a hidden layer with size 2
• an output layer with size 28*28

from keras import models, layers
input_layer = layers.Input(shape=(28*28,))
encoding_layer = layers.Dense(2)(input_layer)
decoding_layer = layers.Dense(28*28) (encoding_layer)
autoencoder = models.Model(input_layer, decoding_layer)

Let us compile and train… We will fit the model using a binary cross entropy loss between the pixel values:

autoencoder.compile('adam', loss='binary_crossentropy')
autoencoder.fit(x = data, y = data, epochs=5)

Did you notice the trick? X = data and y = data as well.

After fitting the model, the network is supposed to learn how to calculate the hidden encodings. But we still have to extract the layer responsible for this. In the following, we define a new model where we remove the final layer since we do not need it anymore:

encoder = models.Model(input_layer, encoding_layer)

Now instead of predicting the final output, we are predicting only the hidden representation. Look how we use it:

encodings = encoder.predict(data)

That is it! Now your encodings variable is an (n, m) array where n is the number of examples and m is the number of dimensions. The first column is the first feature and the second column is the second one. But what are those features? Actually, we do not know. We only know that they are representative of each input value.

Let us plot them and see what we get.

Beautiful! See how the neural network learned the hidden features. Clearly it learned the different characteristics for each digit and how they are distributed in a 2D space. Now we could use these features for visualizations, clustering or any other purpose…

Final Thoughts

In this article we learned about AutoEncoders and how to apply them for dimensionality reduction. AutoEncoders are very powerful and used in many modern neural network architectures. In future posts, you will learn about more complex Encoder/Decoder network. If you like this article, please support us by sharing it with your network.