README.md

Awesome CIFAR Zoo

This repository contains the pytorch code for multiple CNN architectures and improve methods based on the following papers, hope the implementation and results will helpful for your research!!

• Architecure
  • (lenet) LeNet-5, convolutional neural networks
  • (alexnet) ImageNet Classification with Deep Convolutional Neural Networks
  • (vgg) Very Deep Convolutional Networks for Large-Scale Image Recognition
  • (resnet) Deep Residual Learning for Image Recognition
  • (preresnet) Identity Mappings in Deep Residual Networks
  • (resnext) Aggregated Residual Transformations for Deep Neural Networks
  • (densenet) Densely Connected Convolutional Networks
  • (senet) Squeeze-and-Excitation Networks
• Regularization
  • (shake-shake) Shake-Shake regularization
  • (cutout) Improved Regularization of Convolutional Neural Networks with Cutout
  • (mixup) mixup: Beyond Empirical Risk Minimization
• Learning Rate Scheduler
  • (cos_lr) SGDR: Stochastic Gradient Descent with Warm Restarts
  • (htd_lr) Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification

Requirements and Usage

Requirements

• Python >= 3.5
• PyTorch >= 0.4
• TensorFlow/Tensorboard (if you want to use the tensorboard for visualization)
• Other dependencies (pyyaml, easydict, tensorboardX)

pip install -r requirements.txt

Usage

simply run the cmd for the training:

## 1 GPU for lenet
CUDA_VISIBLE_DEVICES=0 python -u train.py --work-path ./experiments/cifar10/lenet

## resume from ckpt
CUDA_VISIBLE_DEVICES=0 python -u train.py --work-path ./experiments/cifar10/lenet --resume

## 2 GPUs for resnet1202
CUDA_VISIBLE_DEVICES=0,1 python -u train.py --work-path ./experiments/cifar10/preresnet1202

## 4 GPUs for densenet190bc
CUDA_VISIBLE_DEVICES=0,1,2,3 python -u train.py --work-path ./experiments/cifar10/densenet190bc

We use yaml file config.yaml to save the parameters, check any files in ./experimets for more details.
You can see the training curve via tensorboard, tensorboard --logdir path-to-event --port your-port.
The training log will be dumped via logging, check log.txt in your work path.

Results on CIFAR

Vanilla architecures

With additional regularization

PS: the default data augmentation methods are RandomCrop + RandomHorizontalFlip + Normalize,
and the √ means which additional method be used. ?

architecure epoch cutout mixup C10 test acc (%) preresnet20 250

91.88 preresnet20 250 √

92.57 preresnet20 250

√ 92.71 preresnet20 250 √ √ 92.66 preresnet110 250

94.24 preresnet110 250 √

94.67 preresnet110 250

√ 94.94 preresnet110 250 √ √ 95.66 se_resnext29_16x64d 300

96.15 se_resnext29_16x64d 300 √

96.60 se_resnext29_16x64d 300

√ 96.86 se_resnext29_16x64d 300 √ √ 97.03 shake_resnet26_2x64d 1800

96.94 shake_resnet26_2x64d 1800 √

97.20 shake_resnet26_2x64d 1800

√ 97.42 shake_resnet26_2x64d 1800 √ √ 97.71

PS: shake_resnet26_2x64d achieved 97.71% test accuracy with cutout and mixup!!
It's cool, right?

With different LR scheduler

architecure epoch step decay cosine htd(-6,3) C10 test acc (%) preresnet20 250 √

91.88 preresnet20 250

√

92.13 preresnet20 250

√ 92.44 preresnet110 250 √

94.24 preresnet110 250

√

94.48 preresnet110 250

√ 94.82

Acknowledgments

Provided codes were adapted from

• kuangliu/pytorch-cifar
• bearpaw/pytorch-classification
• timgaripov/swa
• xgastaldi/shake-shake
• uoguelph-mlrg/Cutout
• BIGBALLON/cifar-10-cnn

Feel free to contact me if you have any suggestions or questions, issues are welcome,
create a PR if you find any bugs or you want to contribute. ?