Deep Learning Papers on Medical Image Analysis

Background

To the best of our knowledge, this is the first list of deep learning papers on medical applications. There are couple of lists for deep learning papers in general, or computer vision, for example Awesome Deep Learning Papers. In this list, I try to classify the papers based on their deep learning techniques and learning methodology. I believe this list could be a good starting point for DL researchers on Medical Applications.

Criteria

1. A list of top deep learning papers published since 2015.
2. Papers are collected from peer-reviewed journals and high reputed conferences. However, it may have recent papers on arXiv.
3. A meta-data is required along with the paper, i.e. Deep Learning technique, Imaging Modality, Area of Interest, Clinical Database (DB).

List of Journals / Conferences (J/C):

• Medical Image Analysis (MedIA)
• IEEE Transaction on Medical Imaging (IEEE-TMI)
• IEEE Transaction on Biomedical Engineering (IEEE-TBME)
• IEEE Journal of Biomedical and Health Informatics (IEEE-JBHI)
• International Journal on Computer Assisted Radiology and Surgery (IJCARS)
• International Conference on Information Processing in Medical Imaging (IPMI)
• International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)
• International Conference on Information Processing in Computer-Assisted Interventions (IPCAI)
• IEEE International Symposium on Biomedical Imaging (ISBI)

Shortcuts

Deep Learning Techniques:

• NN: Neural Networks
• MLP: Multilayer Perceptron
• RBM: Restricted Boltzmann Machine
• SAE: Stacked Auto-Encoders
• CAE: Convolutional Auto-Encoders
• CNN: Convolutional Neural Networks
• RNN: Recurrent Neural Networks
• LSTM: Long Short Term Memory
• M-CNN: Multi-Scale/View/Stream CNN
• MIL-CNN: Multi-instance Learning CNN
• FCN: Fully Convolutional Networks

Imaging Modality:

• US: Ultrasound
• MR/MRI: Magnetic Resonance Imaging
• PET: Positron Emission Tomography
• MG: Mammography
• CT: Computed Tompgraphy
• H&E: Hematoxylin & Eosin Histology Images
• RGB: Optical Images

Table of Contents

Deep Learning Techniques

• AutoEncoders/ Stacked AutoEncoders
• Convolutional Neural Networks
• Recurrent Neural Networks
• Generative Adversarial Networks

Medical Applications

• Annotation
• Classification
• Detection/ Localization
• Segmentation
• Registration
• Regression
• Image Reconstruction and Post-Processing
• Other tasks

---

Deep Learning Techniques

Auto-Encoders/ Stacked Auto-Encoders

Convolutional Neural Networks

• AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
• Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images

Recurrent Neural Networks

Generative Adversarial Networks

• Adversarial Deep Structured Nets for Mass Segmentation from Mammograms

Medical Applications

Annotation

Technique	Modality	Area	Paper Title	DB	J/C	Year
NN	H&E	N/A	Deep learning of feature representation with multiple instance learning for medical image analysis [pdf]		ICASSP	2014
M-CNN	H&E	Breast	AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images [pdf]	AMIDA	IEEE-TMI	2016
FCN	H&E	N/A	Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation pdf		MICCAI	2017

Classification

Technique	Modality	Area	Paper Title	DB	J/C	Year
M-CNN	CT	Lung	Multi-scale Convolutional Neural Networks for Lung Nodule Classification [pdf]	LIDC-IDRI	IPMI	2015
3D-CNN	MRI	Brain	Predicting Alzheimer's disease: a neuroimaging study with 3D convolutional neural networks [pdf]	ADNI	arXiv	2015
CNN+RNN	RGB	Eye	Automatic Feature Learning to Grade Nuclear Cataracts Based on Deep Learning [pdf]		IEEE-TBME	2015
CNN	X-ray	Knee	Quantifying Radiographic Knee Osteoarthritis Severity using Deep Convolutional Neural Networks [pdf]	O.E.1	arXiv	2016
CNN	H&E	Thyroid	A Deep Semantic Mobile Application for Thyroid Cytopathology [pdf]		SPIE	2016
3D-CNN, 3D-CAE	MRI	Brain	Alzheimer's Disease Diagnostics by a Deeply Supervised Adaptable 3D Convolutional Network [pdf]	ADNI	arXiv	2016
M-CNN	RGB	Skin	Multi-resolution-tract CNN with hybrid pretrained and skin-lesion trained layers [pdf]	Dermofit	MLMI	2016
CNN	RGB	Skin, Eye	Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes [pdf]	EDRA, DRD	arXiv	2016
M-CNN	CT	Lung	Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks [pdf]	LIDC-IDRI, ANODE09, DLCST	IEEE-TMI	2016
3D-CNN	CT	Lung	DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification [pdf]	LIDC-IDRI, LUNA16	IEEE-WACV	2018
3D-CNN	MRI	Brain	3D Deep Learning for Multi-modal Imaging-Guided Survival Time Prediction of Brain Tumor Patients [pdf]		MICCAI	2016
SAE	US, CT	Breast, Lung	Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans [pdf]	LIDC-IDRI	Nature	2016
CAE	MG	Breast	Unsupervised deep learning applied to breast density segmentation and mammographic risk scoring [pdf]		IEEE-TMI	2016
MIL-CNN	MG	Breast	Deep multi-instance networks with sparse label assignment for whole mammogram classification [pdf]	INbreast	MICCAI	2017
GCN	MRI	Brain	Spectral Graph Convolutions for Population-based Disease Prediction [pdf]	ADNI, ABIDE	arXiv	2017
CNN	RGB	Skin	Dermatologist-level classification of skin cancer with deep neural networks		Nature	2017
FCN + CNN	MRI	Liver-Liver Tumor	SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D convolutional neural networks [pdf]		ISBI	2017

Detection / Localization

Technique	Modality	Area	Paper Title	DB	J/C	Year
MLP	CT	Head-Neck	3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data [pdf]		MICCAI	2015
CNN	US	Fetal	Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks [pdf]		IEEE-JBHI	2015
2.5D-CNN	MRI	Femur	Automated anatomical landmark detection ondistal femur surface using convolutional neural network [pdf]	OAI	ISBI	2015
LSTM	US	Fetal	Automatic Fetal Ultrasound Standard Plane Detection Using Knowledge Transferred Recurrent Neural Networks [pdf]		MICCAI	2015
CNN	X-ray, MRI	Hand	Regressing Heatmaps for Multiple Landmark Localization using CNNs [pdf]	DHADS	MICCAI	2016
CNN	MRI, US, CT	-	An artificial agent for anatomical landmark detection in medical images [pdf]	SATCOM	MICCAI	2016
FCN	US	Fetal	Real-time Standard Scan Plane Detection and Localisation in Fetal Ultrasound using Fully Convolutional Neural Networks [pdf]		MICCAI	2016
CNN+LSTM	MRI	Heart	Recognizing end-diastole and end-systole frames via deep temporal regression network [pdf]		MICCAI	2016
M-CNN	MRI	Heart	Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation Neural Networks [pdf]		IEEE-TMI	2016
CNN	PET/CT	Heart	Automated detection of pulmonary nodules in PET/CT images: Ensemble false-positive reduction using a convolutional neural network technique Neural Networks [pdf]		MP	2016
3D-CNN	MRI	Brain	Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks [pdf]		IEEE-TMI	2016
CNN	X-ray, MG	-	Self-Transfer Learning for Fully Weakly Supervised Lesion Localization [pdf]	NIH,China, DDSM,MIAS	MICCAI	2016
CNN	RGB	Eye	Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images [pdf]	DRD, MESSIDOR	MICCAI	2016
GAN	-	-	Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery		IPMI	2017
FCN	X-ray	Cardiac	CathNets: Detection and Single-View Depth Prediction of Catheter Electrodes		MIAR	2016
3D-CNN	CT	Lung	DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification [pdf]	LIDC-IDRI, LUNA16	IEEE-WACV	2018
3D-CNN	CT	Lung	DeepEM: Deep 3D ConvNets with EM for weakly supervised pulmonary nodule detection [pdf]	LIDC-IDRI, LUNA16	MICCAI	2018

Segmentation

Technique	Modality	Area	Paper Title	DB	J/C	Year
U-Net	-	-	U-net: Convolutional networks for biomedical image segmentation		MICCAI	2015
FCN	MRI	Head-Neck	Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation [pdf]		arXiv	2016
U-Net	CT	Head-Neck	AnatomyNet: Deep learning for fast and fully automated whole‐volume segmentation of head and neck anatomy [pdf]		Medical Physics	2018
FCN	CT	Liver-Liver Tumor	Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields [pdf]		MICCAI	2016
3D-CNN	MRI	Spine	Model-Based Segmentation of Vertebral Bodies from MR Images with 3D CNNs		MICCAI	2016
FCN	CT	Liver-Liver Tumor	Automatic Liver and Tumor Segmentation of CT and MRI Volumes using Cascaded Fully Convolutional Neural Networks [pdf]		arXiv	2017
FCN	MRI	Liver-Liver Tumor	SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D convolutional neural networks [pdf]		ISBI	2017
3D-CNN	Diffusion MRI	Brain	q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI [pdf] (Section II.B.2)		IEEE-TMI	2016
GAN	MG	Breast Mass	Adversarial Deep Structured Nets for Mass Segmentation from Mammograms [pdf]	INbreast, DDSM-BCRP	ISBI	2018
3D-CNN	CT	Liver	3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes pdf		MICCAI	2017
3D-CNN	MRI	Brain	Unsupervised domain adaptation in brain lesion segmentation with adversarial networks pdf		IPMI	2017
FCN	FUNDUS	Retina	A Fully Convolutional Neural Network based Structured Prediction Approach Towards the Retinal Vessel Segmentation pdf		ISBI	2017

Registration

Technique	Modality	Area	Paper Title	DB	J/C	Year
3D-CNN	CT	Spine	An Artificial Agent for Robust Image Registration [pdf]			2016

Regression

Technique	Modality	Area	Paper Title	DB	J/C	Year
2.5D-CNN	MRI		Automated anatomical landmark detection ondistal femur surface using convolutional neural network [pdf]	OAI	ISBI	2015
3D-CNN	Diffusion MRI	Brain	q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI [pdf] (Section II.B.1)	[HCP] and other	IEEE-TMI	2016

Image Reconstruction and Post Processing

Technique	Modality	Area	Paper Title	DB	J/C	Year
CNN	CS-MRI		A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction pdf		IEEE-TMI	2017
GAN	CS-MRI		Deep Generative Adversarial Networks for Compressed Sensing Automates MRI pdf		NIPS	2017

Image synthesis for data augmentation

Technique	Modality	Area	Paper Title	DB	J/C	Year
GAN	RGB (Microscopy)	Red Blood Cells	Red blood cell image generation for data augmentation using Conditional Generative Adversarial Networks [pdf]		arXiv	2019
GAN	MRI	Brain	Learning Data Augmentation for Brain Tumor Segmentation with Coarse-to-Fine Generative Adversarial Networks [pdf]		arXiv	2018
GAN	MRI	Brain	Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks [pdf]		arXiv	2018
GAN	CT, MRI	Brain	GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks [pdf]		arXiv	2018
GAN	CT	Liver	GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification [pdf]		arXiv	2018

Other tasks

References