1. Neural Networks - Build, train, and apply fully connected deep neural networks; implement efficient (vectorized) neural networks; identify key parameters in a neural network’s architecture.

2. Improving NNs (tuning, regularization, and optimization) - Open the black box to understand the processes that drive performance and generate good results systematically. Train and develop test sets and analyze bias/variance for building deep learning applications; use standard neural network techniques such as initialization, L2 and dropout regularization, hyperparameter tuning, batch normalization, and gradient checking; implement and apply a variety of optimization algorithms, such as mini-batch gradient descent, Momentum, RMSprop and Adam, and check for their convergence; implement a neural network in TensorFlow.

3. Structuring projects - Diagnose errors in a machine learning system; prioritize strategies for reducing errors; understand complex ML settings, such as mismatched training/test sets, and comparing to and/or surpassing human-level performance; and apply end-to-end learning, transfer learning, and multi-task learning. Become a technical leader who can set the direction for an AI team.

4. CNNs - Understand how computer vision has evolved and become familiar with its applications such as autonomous driving, face recognition, and reading radiology images. Build a convolutional neural network, including recent variations such as residual networks; apply convolutional networks to visual detection and recognition tasks; and use neural style transfer to generate art and apply these algorithms to a variety of image, video, and other 2D or 3D data.

5. Seuence models - Become familiar with sequence models and their applications such as speech recognition, music synthesis, chatbots, machine translation, and natural language processing (NLP). Build and train Recurrent Neural Networks (RNNs) and commonly-used variants such as GRUs and LSTMs; apply RNNs to Character-level Language Modeling; gain experience with natural language processing and Word Embeddings; and use HuggingFace tokenizers and transformer models to solve different NLP tasks such as NER and Question Answering.

References

Implementing a Neural Network from Scratch in Python – An Introduction (Denny Britz, 2015)

Demystifying Deep Convolutional Neural Networks (Adam Harley)

CS231n: Convolutional Neural Networks for Visual Recognition (Stanford University)

Introduction to gradients and automatic differentiation (TensorFlow Documentation)

The Sequential model (TensorFlow Documentation)

The Functional API (TensorFlow Documentation)

Deep Residual Learning for Image Recognition (He, Zhang, Ren & Sun, 2015)

deep-learning-models/resnet50.py/ (GitHub: fchollet)

Classic CNN Architectures

Gradient-based learning applied to document recognition. Section 2-3 (Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 1998)

ImageNet Classification with Deep Convolutional Neural Networks (Krizhevsky, A., Sutskever, I. and Hinton, G. E., 2012)

Very Deep Convolutional Networks for Large-Scale Image Recognition. ( K. Simonyan and A. Zisserman 2015)

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (Howard, Zhu, Chen, Kalenichenko, Wang, Weyand, Andreetto, ​& Adam, 2017)

MobileNetV2: Inverted Residuals and Linear Bottlenecks (Sandler, Howard, Zhu, Zhmoginov &Chen, 2018)

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (Tan & Le, 2019)

You Only Look Once: Unified, Real-Time Object Detection (Redmon, Divvala, Girshick & Farhadi, 2015)

YOLO9000: Better, Faster, Stronger (Redmon & Farhadi, 2016)

YAD2K (GitHub: allanzelener)

YOLO: Real-Time Object Detection

Fully Convolutional Architectures for Multi-Class Segmentation in Chest Radiographs (Novikov, Lenis, Major, Hladůvka, Wimmer & Bühler, 2017)

Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks (Dong, Yang, Liu, Mo & Guo, 2017)

U-Net: Convolutional Networks for Biomedical Image Segmentation (Ronneberger, Fischer & Brox, 2015)

FaceNet: A Unified Embedding for Face Recognition and Clustering (Schroff, Kalenichenko & Philbin, 2015)

DeepFace: Closing the Gap to Human-Level Performance in Face Verification (Taigman, Yang, Ranzato & Wolf)

facenet (GitHub: davidsandberg)

How to Develop a Face Recognition System Using FaceNet in Keras (Jason Brownlee, 2019)

keras-facenet/notebook/tf_to_keras.ipynb (GitHub: nyoki-mtl)

A Neural Algorithm of Artistic Style (Gatys, Ecker & Bethge, 2015)

Convolutional neural networks for artistic style transfer

TensorFlow Implementation of "A Neural Algorithm of Artistic Style"

Very Deep Convolutional Networks For Large-Scale Image Recognition (Simonyan & Zisserman, 2015)

Pretrained models (MatConvNet)

Minimal character-level language model with a Vanilla Recurrent Neural Network, in Python/numpy (GitHub: karpathy)

The Unreasonable Effectiveness of Recurrent Neural Networks (Andrej Karpathy blog, 2015)

GloVe: Global Vectors for Word Representation (Pennington, Socher & Manning, 2014)

Attention Is All You Need (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser​ & Polosukhin, 2017)