Step by Step Implementation: 3D Convolutional Neural Network in Keras

Learn how to implement your very own 3D CNN

In this article, we will be briefly explaining what a 3d CNN is, and how it is different from a generic 2d CNN. Then we will teach you step by step how to implement your own 3D Convolutional Neural Network using Keras.

1] What is a 3D Convolutional Neural Network?

A 3d CNN remains regardless of what we say a CNN that is very much similar to 2d CNN. Except that it differs in these following points (non-exhaustive listing):

3d Convolution Layers

Originally a 2d Convolution Layer is an entry per entry multiplication between the input and the different filters, where filters and inputs are 2d matrices. (fig.1)

fig.1 (copyrighted: own)

In a 3d Convolution Layer, the same operations are used. We do these operations on multiple pairs of 2d matrices. (fig.2)

fig.2 (copyrighted: own)

Padding options and slides step options work the same way.

3d MaxPool Layers

2d Maxpool Layers (2x2 filter) is about taking the maximum element of a small 2x2 square that we delimitate from the input. (fig.3)

fig.3(copyrighted: own)

Now in a 3d Maxpool (2x2x2), we look for the maximum element in a width 2 cube. This cube represents the space delimited by the 2x2x2 zone from the input. (fig.4)

fig.4(copyrighted: own)

Note that the number of operations (compared to 2d CNN layers) is multiplied by the size of the filters used (regardless of the layer being Maxpool or Convolution) and also multiplied by the size of the input itself.

2] 3d existing Datasets

So how does a data point for a 3d CNN look like?

One way to picture it is by using the following image (fig.5):

fig.5 (copyrighted: own)

Other existing datasets that you can use for your CNN are:

• RGB-D devices: Google Tango , Microsoft Kinect , etc.
• Lidar
• 3D reconstruction from multiple images

3] Preprocessing and Implementations

You can try for yourself the code on this dataset from Kaggle that we are using.

The required libraries to import are as follows:

To begin with, since the dataset is a bit specific, we use the following to helper functions to process them before giving them to the network.

Plus, the dataset is stored as h5 file, so to extract the actual data points, we are required to read from h5 file, and use the to_categorical function to transform it into vectors. In this step, we also prepare for cross-validation.

Finally, the model and the syntax for 3d CNN are as follows: (the architecture was picked without much refining since that is not the point of this article)

Note that the numbers of parameters will be a lot higher for the same number of layers compared to 2d CNN.

For your information, after a small sample training, we got the following accuracies and losses. (fig.6)

fig.6 (copyrighted: own)

4] But then a 3d? What for?

There happens to have many applications for a 3d CNN that are for instance:

• IRM data processing and therefore the inference
• self-driving
• Distance estimation

Alright, that’s pretty much all. I hope you will try this technology out!