A collection of various deep learning architectures, models, and tips

Deep Learning Models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

• Perceptron [ TensorFlow 1 ] [ PyTorch ]
• Logistic Regression [ TensorFlow 1 ] [ PyTorch ]
• Softmax Regression (Multinomial Logistic Regression) [ TensorFlow 1 ] [ PyTorch ]

Multilayer Perceptrons

• Multilayer Perceptron [ TensorFlow 1 ] [ PyTorch ]
• Multilayer Perceptron with Dropout [ TensorFlow 1 ] [ PyTorch ]
• Multilayer Perceptron with Batch Normalization [ TensorFlow 1 ] [ PyTorch ]
• Multilayer Perceptron with Backpropagation from Scratch [ TensorFlow 1 ] [ PyTorch ]

Convolutional Neural Networks

Basic

• Convolutional Neural Network [ TensorFlow 1 ] [ PyTorch ]
• Convolutional Neural Network with He Initialization [ PyTorch ]

Concepts

• Replacing Fully-Connnected by Equivalent Convolutional Layers [ PyTorch ]

Fully Convolutional

• Fully Convolutional Neural Network [ PyTorch ]

AlexNet

• AlexNet on CIFAR-10 [ PyTorch ]

VGG

• Convolutional Neural Network VGG-16 [ TensorFlow 1 ] [ PyTorch ]
• VGG-16 Gender Classifier Trained on CelebA [ PyTorch ]
• Convolutional Neural Network VGG-19 [ PyTorch ]

ResNet

• ResNet and Residual Blocks [ PyTorch ]
• ResNet-18 Digit Classifier Trained on MNIST [ PyTorch ]
• ResNet-18 Gender Classifier Trained on CelebA [ PyTorch ]
• ResNet-34 Digit Classifier Trained on MNIST [ PyTorch ]
• ResNet-34 Gender Classifier Trained on CelebA [ PyTorch ]
• ResNet-50 Digit Classifier Trained on MNIST [ PyTorch ]
• ResNet-50 Gender Classifier Trained on CelebA [ PyTorch ]
• ResNet-101 Gender Classifier Trained on CelebA [ PyTorch ]
• ResNet-152 Gender Classifier Trained on CelebA [ PyTorch ]

Network in Network

• Network in Network CIFAR-10 Classifier [ PyTorch ]

Metric Learning

• Siamese Network with Multilayer Perceptrons [ TensorFlow 1 ]

Autoencoders

Fully-connected Autoencoders

• Autoencoder [ TensorFlow 1 ] [ PyTorch ]

Convolutional Autoencoders

• Convolutional Autoencoder with Deconvolutions / Transposed Convolutions[ TensorFlow 1 ] [ PyTorch ]
• Convolutional Autoencoder with Deconvolutions (without pooling operations) [ PyTorch ]
• Convolutional Autoencoder with Nearest-neighbor Interpolation [ TensorFlow 1 ] [ PyTorch ]
• Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA [ PyTorch ]
• Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw [ PyTorch ]

Variational Autoencoders

• Variational Autoencoder [ PyTorch ]
• Convolutional Variational Autoencoder [ PyTorch ]

Conditional Variational Autoencoders

• Conditional Variational Autoencoder (with labels in reconstruction loss) [ PyTorch ]
• Conditional Variational Autoencoder (without labels in reconstruction loss) [ PyTorch ]
• Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss) [ PyTorch ]
• Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss) [ PyTorch ]

Generative Adversarial Networks (GANs)

• Fully Connected GAN on MNIST [ TensorFlow 1 ] [ PyTorch ]
• Convolutional GAN on MNIST [ TensorFlow 1 ] [ PyTorch ]
• Convolutional GAN on MNIST with Label Smoothing [ PyTorch ]

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

• A simple single-layer RNN (IMDB) [ PyTorch ]
• A simple single-layer RNN with packed sequences to ignore padding characters (IMDB) [ PyTorch ]
• RNN with LSTM cells (IMDB) [ PyTorch ]
• RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors [ PyTorch ]
• RNN with LSTM cells and Own Dataset in CSV Format (IMDB) [ PyTorch ]
• RNN with GRU cells (IMDB) [ PyTorch ]
• Multilayer bi-directional RNN (IMDB) [ PyTorch ]

Many-to-Many / Sequence-to-Sequence

• A simple character RNN to generate new text (Charles Dickens) [ PyTorch ]

Ordinal Regression

• Ordinal Regression CNN -- CORAL w. ResNet34 on AFAD-Lite [ PyTorch ]
• Ordinal Regression CNN -- Niu et al. 2016 w. ResNet34 on AFAD-Lite [ PyTorch ]
• Ordinal Regression CNN -- Beckham and Pal 2016 w. ResNet34 on AFAD-Lite [ PyTorch ]

Tips and Tricks

• Cyclical Learning Rate [ PyTorch ]

PyTorch Workflows and Mechanics

Custom Datasets

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5 [ PyTorch ]
• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA [ PyTorch ]
• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw [ PyTorch ]
• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset [ PyTorch ]

Training and Preprocessing

• Dataloading with Pinned Memory [ PyTorch ]
• Standardizing Images [ PyTorch ]
• Image Transformation Examples [ PyTorch ]
• Char-RNN with Own Text File [ PyTorch ]
• Sentiment Classification RNN with Own CSV File [ PyTorch ]

Parallel Computing

• Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA [ PyTorch ]

Other

• Sequential API and hooks [ PyTorch ]
• Weight Sharing Within a Layer [ PyTorch ]
• Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib [ PyTorch ]

Autograd

• Getting Gradients of an Intermediate Variable in PyTorch [ PyTorch ]

TensorFlow Workflows and Mechanics

Custom Datasets

• Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives [ TensorFlow 1 ]
• Storing an Image Dataset for Minibatch Training using HDF5 [ TensorFlow 1 ]
• Using Input Pipelines to Read Data from TFRecords Files [ TensorFlow 1 ]
• Using Queue Runners to Feed Images Directly from Disk [ TensorFlow 1 ]
• Using TensorFlow's Dataset API [ TensorFlow 1 ]

Training and Preprocessing

• Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives [ TensorFlow 1 ]