Knowledge Distillation for UNet

An implementation of Knowledge distillation for segmentation, to train a small (student) UNet from a larger (teacher) UNet thereby reducing the size of the network while achieving performance similar to the heavier model.

Results:

Dataset: Carvana Image Masking Challenge

Models trained without knowledge distillation

Models trained with knowledge distillation

References

• Distilling the Knowledge in a Neural Network - Geoffrey Hinton, Oriol Vinyals, Jeff Dean

• Structured Knowledge Distillation for Dense Prediction - Yifan Liu, Changyong Shun, Jingdong Wang, Chunhua Shen

• On Compressing U-net Using Knowledge Distillation - Karttikeya Mangalam, Mathieu Salzamann