Skip to content

Latest commit

 

History

History
155 lines (110 loc) · 6.09 KB

README.md

File metadata and controls

155 lines (110 loc) · 6.09 KB
Raw

Kaggle CIFAR-10

Code for CIFAR-10 competition. http://www.kaggle.com/c/cifar-10

Summary

Description
Model Very Deep Convolutional Networks with 3x3 kernel [1]
Data Augmentation cropping, horizontal reflection [2] and scaling. see lib/data_augmentation.lua
Preprocessing Global Contrast Normalization (GCN) and ZCA whitening. see lib/preprocessing.lua
Training Time 20 hours on GTX760.
Prediction Time 2.5 hours on GTX760.
Result 0.93320 (single model). 0.94150 (average 6 models)

Neural Network Configurations

Layer type Parameters
input size: 24x24, channel: 3
convolution kernel: 3x3, channel: 64, padding: 1
relu
convolution kernel: 3x3, channel: 64, padding: 1
relu
max pooling kernel: 2x2, stride: 2
dropout rate: 0.25
convolution kernel: 3x3, channel: 128, padding: 1
relu
convolution kernel: 3x3, channel: 128, padding: 1
relu
max pooling kernel: 2x2, stride: 2
dropout rate: 0.25
convolution kernel: 3x3, channel: 256, padding: 1
relu
convolution kernel: 3x3, channel: 256, padding: 1
relu
convolution kernel: 3x3, channel: 256, padding: 1
relu
convolution kernel: 3x3, channel: 256, padding: 1
relu
max pooling kernel: 2x2, stride: 2
dropout rate: 0.25
linear channel: 1024
relu
dropout rate: 0.5
linear channel: 1024
relu
dropout rate: 0.5
linear channel: 10
softmax

Developer Environment

  • Ubuntu 14.04
  • 15GB RAM (This codebase can run on g2.2xlarge!)
  • CUDA (GTX760 or more higher GPU)
  • Torch7 latest
  • cuda-convnet2.torch

Installation

(This document is outdated. See: Getting started with Torch)

Install CUDA (on Ubuntu 14.04):

apt-get install nvidia-331
apt-get install nvidia-cuda-toolkit

Install Torch7 (see Torch (easy) install):

curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash

Install(or upgrade) dependency packages:

luarocks install torch
luarocks install nn
luarocks install cutorch
luarocks install cunn
luarocks install https://raw.githubusercontent.com/soumith/cuda-convnet2.torch/master/ccn2-scm-1.rockspec

Checking CUDA environment

th cuda_test.lua

Please check your Torch7/CUDA environment when this code fails.

Convert dataset

Place the data files into a subfolder ./data.

ls ./data
test  train  trainLabels.csv
  • th convert_data.lua

Local testing

th validate.lua

dataset:

train test
1-40000 40001-50000

Generating the submission.txt

th train.lua
th predict.lua

MISC

Model Averaging

Training with different seed parameter for each nodes.

(same model, same data, different initial weights, different training order)

th train.lua -seed 11
th train.lua -seed 12
...
th train.lua -seed 16

Mount the models directory for each nodes. for example, ec2/node1, ec2/node2, .., ec2/node6.

Edit the path of model file in predict_averaging.lua.

Run the prediction command.

th predict_averaging.lua

Network In Network

./nin_model.lua is an implementation of Network In Network [3]. This model gives score of 0.92400.

My NIN implementation is 2-layer NIN. Its differ from mavenlin's implementation. I tried to implement the mavenlin's 3-layer NIN. However, I did not get good result.

My implementation of 3-layer NIN is here.

Bug

global_contrast_normalization in ./lib/preprocessing.lua is incorrect implementation (This function is just z-score). but I was using this implementation in the competition.

Figure

data augmentation + preprocessing

data-augmentation-preprocessing

References

  • [1] Karen Simonyan, Andrew Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition", link
  • [2] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks", link
  • [3] Min Lin, Qiang Chen, Shuicheng Yan, "Network In Network", link
  • [4] R. Collobert, K. Kavukcuoglu, C. Farabet, "Torch7: A Matlab-like Environment for Machine Learning"