From 5df9531d5ccb689f53a2ea25a3fa6fa140b160f2 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Tue, 30 Oct 2018 01:24:30 -0400 Subject: [PATCH 01/12] Documentation listing currently supported layers for Keras, Caffe and TensorFlow --- tutorials/supported_layers.md | 176 ++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 tutorials/supported_layers.md diff --git a/tutorials/supported_layers.md b/tutorials/supported_layers.md new file mode 100644 index 000000000..10c8098a8 --- /dev/null +++ b/tutorials/supported_layers.md @@ -0,0 +1,176 @@ +# Layers supported in Caffe, Keras, and Tensorflow +Below is are table showing which layers are supported by Caffe, Keras, and Tensorflow: + +### Core Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Activation | √ | √ | √ | +| ActivityRegularization | √ | √ | √ | +| Dense | × | √ | √ | +| Dropout | √ | √ | √ | +| Flatten | √ | √ | √ | +| Lambda | × | √ | √ | +| Masking | √ | √ | √ | +| Permute | × | √ | √ | +| Repeat Vector | × | √ | √ | +| Reshape | √ | √ | √ | +| Spatial Dropout 1D | × | √ | √ | +| Spatial Dropout 2D | × | √ | √ | +| Spatial Dropout 3D | × | √ | √ | + +#### Convolutional Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Conv1D | √ | √ | √ | +| Conv2D | √ | √ | √ | +| DepthwiseConv2D | × | × | √ | +| SeperableConv1D | × | √ | √ | +| SeperableConv2D | × | √ | √ | +| Conv2DTranspose | √ | √ | √ | +| Conv3D | √ | √ | √ | +| Conv3DTranspose | √ | √ | √ | +| Cropping1D | √ | √ | √ | +| Cropping2D | √ | √ | √ | +| Cropping3D | √ | √ | √ | +| Upsampling 1D | √ | √ | √ | +| Upsampling 2D | √ | √ | √ | +| Upsampling 3D | √ | √ | √ | +| ZeroPadding 1D | × | √ | √ | +| ZeroPadding 2D | × | √ | √ | +| ZeroPadding 3D | × | √ | √ | +| Im2Col | √ | × | × | +| Spatial Pyramid Pooling | √ | × | × | +* Upsampling in Caffe can be done by using methods shown here: https://gist.github.com/tnarihi/54744612d35776f53278 + +### Pooling Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| MaxPooling1D | √ | √ | √ | +| MaxPooling2D | √ | √ | √ | +| MaxPooling3D | √ | √ | √ | +| AveragePooling1D | √ | √ | √ | +| AveragePooling2D | √ | √ | √ | +| AveragePooling3D | √ | √ | √ | +| GlobalMaxPooling1D | × | √ | √ | +| GlobalAveragePooling1D | × | √ | √ | +| GlobalMaxPooling2D | × | √ | √ | +| GlobalAveragePooling2D | × | √ | √ | +| GlobalMaxPooling3D | × | √ | √ | +| GlobalAveragePooling3D | × | √ | √ | +| Stochastic Pooling | √ | × | × | + +### Locally-connected Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| LocallyConnected1D | × | √ | √ | +| LocallyConnected2D | × | √ | √ | + +### Recurrent Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| RNN | √ | √ | √ | +| SimpleRNN | × | √ | √ | +| GRU | × | √ | √ | +| LSTM | √ | √ | √ | +| ConvLSTM2D | × | √ | √ | +| SimpleRNNCell | × | √ | √ | +| GRUCell | × | √ | √ | +| LSTMCell | × | √ | √ | +| CuDDNGRU | × | √ | √ | +| CuDDNLSTM | × | √ | √ | +| StackedRNNCell | × | × | √ | + +### Embedding Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Embedding | √ | √ | √ | + +### Merge Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Add | × | √ | √ | +| Subtract | × | √ | √ | +| Multiply | × | √ | √ | +| Average | × | √ | √ | +| Minium | × | × | √ | +| Maximum | × | √ | √ | +| Concatenate | √ | √ | √ | +| Dot | × | √ | √ | + +### Activations Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| ReLu | √ | √ | √ | +| LeakyReLu | √ | √ | √ | +| PReLU | √ | √ | √ | +| ELU | √ | √ | √ | +| ThresholdedReLU | √ | √ | √ | +| Softmax | √ | √ | √ | +| Argmax | √ | × | × | +| Sigmoid | √ | √ | √ | +| TanH | √ | √ | √ | +| Absolute Value | √ | × | × | +| Power | √ | √ | × | +| Exp | √ | √ | × | +| Linear | × | √ | √ | +| Log | √ | √ | × | +| BNLL | √ | × | × | +| Bias | √ | × | × | +| Scale | √ | × | × | + +### Utility Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Argmax | √ | × | × | +| Slicing | √ | × | × | +| Eltwise | √ | × | × | +| Parameter | √ | × | × | +| Reduction | √ | × | × | +| Silence | √ | × | × | + +### Loss Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Multinomial Logistic Loss | √ | × | × | +| Infogain Loss | √ | × | × | +| Softmax with Loss | √ | × | √ | +| Sum-of-Squares/Euclidean | √ | × | × | +| Hinge / Margin | √ | √ | √ | +| Sigmoid Cross-Entropy Loss| √ | × | √ | +| Accuracy / Top-k layer | √ | × | × | +| Contrastive Loss | √ | × | × | + +### Normalization Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| BatchNormalization | √ | √ | √ | +| LRN | √ | × | × | +| MVN | √ | × | × | + +### Noise Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| GaussianNoise | × | √ | √ | +| GaussianDropout | √ | √ | √ | +| AlphaDropout | √ | √ | √ | + +### Layer Wrappers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| TimeDistributed | × | √ | √ | +| Bidirectional | × | √ | √ | + +### Custom Layers +| Caffe | Keras | Tensorflow | +| :-----------: | :----------: | :---------: | +| √ | √ | Use Keras API for custom layers | + +## Additional Notes: +* Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to here: https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md. +* Documentation for writing your own Keras layers is found here: https://keras.io/layers/writing-your-own-keras-layers/ +* Caffe 3D layers can be used in Caffe 2.0 + +## Documentation for Caffe, Keras, and Tensorflow layers +* Documentation for all Keras Layers is found here: https://keras.io/layers/about-keras-layers/ +* Documentation for all Caffe Layers is found here: http://caffe.berkeleyvision.org/tutorial/layers.html +* Documentation for all Tensorflow Layers is found here: https://www.tensorflow.org/api_docs/python/tf/layers From 5a1fdbfc988fce317b8e55e96976b4a225364fb1 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Tue, 30 Oct 2018 10:53:37 -0400 Subject: [PATCH 02/12] Moving supported_layers to docs/source --- docs/source/supported_layers.md | 161 ++++++++++++++++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 docs/source/supported_layers.md diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md new file mode 100644 index 000000000..e05635247 --- /dev/null +++ b/docs/source/supported_layers.md @@ -0,0 +1,161 @@ +# Layers supported in Caffe, Keras, and Tensorflow +Below is are table showing which layers are supported by Caffe, Keras, and Tensorflow: + ### Core Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Activation | √ | √ | √ | +| ActivityRegularization | √ | √ | √ | +| Dense | × | √ | √ | +| Dropout | √ | √ | √ | +| Flatten | √ | √ | √ | +| Lambda | × | √ | √ | +| Masking | √ | √ | √ | +| Permute | × | √ | √ | +| Repeat Vector | × | √ | √ | +| Reshape | √ | √ | √ | +| Spatial Dropout 1D | × | √ | √ | +| Spatial Dropout 2D | × | √ | √ | +| Spatial Dropout 3D | × | √ | √ | + #### Convolutional Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Conv1D | √ | √ | √ | +| Conv2D | √ | √ | √ | +| DepthwiseConv2D | × | × | √ | +| SeperableConv1D | × | √ | √ | +| SeperableConv2D | × | √ | √ | +| Conv2DTranspose | √ | √ | √ | +| Conv3D | √ | √ | √ | +| Conv3DTranspose | √ | √ | √ | +| Cropping1D | √ | √ | √ | +| Cropping2D | √ | √ | √ | +| Cropping3D | √ | √ | √ | +| Upsampling 1D | √ | √ | √ | +| Upsampling 2D | √ | √ | √ | +| Upsampling 3D | √ | √ | √ | +| ZeroPadding 1D | × | √ | √ | +| ZeroPadding 2D | × | √ | √ | +| ZeroPadding 3D | × | √ | √ | +| Im2Col | √ | × | × | +| Spatial Pyramid Pooling | √ | × | × | +* Upsampling in Caffe can be done by using methods shown here: https://gist.github.com/tnarihi/54744612d35776f53278 + ### Pooling Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| MaxPooling1D | √ | √ | √ | +| MaxPooling2D | √ | √ | √ | +| MaxPooling3D | √ | √ | √ | +| AveragePooling1D | √ | √ | √ | +| AveragePooling2D | √ | √ | √ | +| AveragePooling3D | √ | √ | √ | +| GlobalMaxPooling1D | × | √ | √ | +| GlobalAveragePooling1D | × | √ | √ | +| GlobalMaxPooling2D | × | √ | √ | +| GlobalAveragePooling2D | × | √ | √ | +| GlobalMaxPooling3D | × | √ | √ | +| GlobalAveragePooling3D | × | √ | √ | +| Stochastic Pooling | √ | × | × | + +### Locally-connected Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| LocallyConnected1D | × | √ | √ | +| LocallyConnected2D | × | √ | √ | + ### Recurrent Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| RNN | √ | √ | √ | +| SimpleRNN | × | √ | √ | +| GRU | × | √ | √ | +| LSTM | √ | √ | √ | +| ConvLSTM2D | × | √ | √ | +| SimpleRNNCell | × | √ | √ | +| GRUCell | × | √ | √ | +| LSTMCell | × | √ | √ | +| CuDDNGRU | × | √ | √ | +| CuDDNLSTM | × | √ | √ | +| StackedRNNCell | × | × | √ | + ### Embedding Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Embedding | √ | √ | √ | + ### Merge Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Add | × | √ | √ | +| Subtract | × | √ | √ | +| Multiply | × | √ | √ | +| Average | × | √ | √ | +| Minium | × | × | √ | +| Maximum | × | √ | √ | +| Concatenate | √ | √ | √ | +| Dot | × | √ | √ | + ### Activations Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| ReLu | √ | √ | √ | +| LeakyReLu | √ | √ | √ | +| PReLU | √ | √ | √ | +| ELU | √ | √ | √ | +| ThresholdedReLU | √ | √ | √ | +| Softmax | √ | √ | √ | +| Argmax | √ | × | × | +| Sigmoid | √ | √ | √ | +| TanH | √ | √ | √ | +| Absolute Value | √ | × | × | +| Power | √ | √ | × | +| Exp | √ | √ | × | +| Linear | × | √ | √ | +| Log | √ | √ | × | +| BNLL | √ | × | × | +| Bias | √ | × | × | +| Scale | √ | × | × | + ### Utility Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Argmax | √ | × | × | +| Slicing | √ | × | × | +| Eltwise | √ | × | × | +| Parameter | √ | × | × | +| Reduction | √ | × | × | +| Silence | √ | × | × | + ### Loss Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| Multinomial Logistic Loss | √ | × | × | +| Infogain Loss | √ | × | × | +| Softmax with Loss | √ | × | √ | +| Sum-of-Squares/Euclidean | √ | × | × | +| Hinge / Margin | √ | √ | √ | +| Sigmoid Cross-Entropy Loss| √ | × | √ | +| Accuracy / Top-k layer | √ | × | × | +| Contrastive Loss | √ | × | × | + ### Normalization Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| BatchNormalization | √ | √ | √ | +| LRN | √ | × | × | +| MVN | √ | × | × | + ### Noise Layers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| GaussianNoise | × | √ | √ | +| GaussianDropout | √ | √ | √ | +| AlphaDropout | √ | √ | √ | + ### Layer Wrappers +| Layer | Caffe | Keras | Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | +| TimeDistributed | × | √ | √ | +| Bidirectional | × | √ | √ | + ### Custom Layers +| Caffe | Keras | Tensorflow | +| :-----------: | :----------: | :---------: | +| √ | √ | Use Keras API for custom layers | + ## Additional Notes: +* Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to here: https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md. +* Documentation for writing your own Keras layers is found here: https://keras.io/layers/writing-your-own-keras-layers/ +* Caffe 3D layers can be used in Caffe 2.0 + ## Documentation for Caffe, Keras, and Tensorflow layers +* Documentation for all Keras Layers is found here: https://keras.io/layers/about-keras-layers/ +* Documentation for all Caffe Layers is found here: http://caffe.berkeleyvision.org/tutorial/layers.html +* Documentation for all Tensorflow Layers is found here: https://www.tensorflow.org/api_docs/python/tf/layers From 35791e41c170cb4bec08a46cec6fcf4862fb4e47 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Tue, 30 Oct 2018 10:54:33 -0400 Subject: [PATCH 03/12] Moved to proper location --- tutorials/supported_layers.md | 176 ---------------------------------- 1 file changed, 176 deletions(-) delete mode 100644 tutorials/supported_layers.md diff --git a/tutorials/supported_layers.md b/tutorials/supported_layers.md deleted file mode 100644 index 10c8098a8..000000000 --- a/tutorials/supported_layers.md +++ /dev/null @@ -1,176 +0,0 @@ -# Layers supported in Caffe, Keras, and Tensorflow -Below is are table showing which layers are supported by Caffe, Keras, and Tensorflow: - -### Core Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Activation | √ | √ | √ | -| ActivityRegularization | √ | √ | √ | -| Dense | × | √ | √ | -| Dropout | √ | √ | √ | -| Flatten | √ | √ | √ | -| Lambda | × | √ | √ | -| Masking | √ | √ | √ | -| Permute | × | √ | √ | -| Repeat Vector | × | √ | √ | -| Reshape | √ | √ | √ | -| Spatial Dropout 1D | × | √ | √ | -| Spatial Dropout 2D | × | √ | √ | -| Spatial Dropout 3D | × | √ | √ | - -#### Convolutional Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Conv1D | √ | √ | √ | -| Conv2D | √ | √ | √ | -| DepthwiseConv2D | × | × | √ | -| SeperableConv1D | × | √ | √ | -| SeperableConv2D | × | √ | √ | -| Conv2DTranspose | √ | √ | √ | -| Conv3D | √ | √ | √ | -| Conv3DTranspose | √ | √ | √ | -| Cropping1D | √ | √ | √ | -| Cropping2D | √ | √ | √ | -| Cropping3D | √ | √ | √ | -| Upsampling 1D | √ | √ | √ | -| Upsampling 2D | √ | √ | √ | -| Upsampling 3D | √ | √ | √ | -| ZeroPadding 1D | × | √ | √ | -| ZeroPadding 2D | × | √ | √ | -| ZeroPadding 3D | × | √ | √ | -| Im2Col | √ | × | × | -| Spatial Pyramid Pooling | √ | × | × | -* Upsampling in Caffe can be done by using methods shown here: https://gist.github.com/tnarihi/54744612d35776f53278 - -### Pooling Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| MaxPooling1D | √ | √ | √ | -| MaxPooling2D | √ | √ | √ | -| MaxPooling3D | √ | √ | √ | -| AveragePooling1D | √ | √ | √ | -| AveragePooling2D | √ | √ | √ | -| AveragePooling3D | √ | √ | √ | -| GlobalMaxPooling1D | × | √ | √ | -| GlobalAveragePooling1D | × | √ | √ | -| GlobalMaxPooling2D | × | √ | √ | -| GlobalAveragePooling2D | × | √ | √ | -| GlobalMaxPooling3D | × | √ | √ | -| GlobalAveragePooling3D | × | √ | √ | -| Stochastic Pooling | √ | × | × | - -### Locally-connected Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| LocallyConnected1D | × | √ | √ | -| LocallyConnected2D | × | √ | √ | - -### Recurrent Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| RNN | √ | √ | √ | -| SimpleRNN | × | √ | √ | -| GRU | × | √ | √ | -| LSTM | √ | √ | √ | -| ConvLSTM2D | × | √ | √ | -| SimpleRNNCell | × | √ | √ | -| GRUCell | × | √ | √ | -| LSTMCell | × | √ | √ | -| CuDDNGRU | × | √ | √ | -| CuDDNLSTM | × | √ | √ | -| StackedRNNCell | × | × | √ | - -### Embedding Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Embedding | √ | √ | √ | - -### Merge Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Add | × | √ | √ | -| Subtract | × | √ | √ | -| Multiply | × | √ | √ | -| Average | × | √ | √ | -| Minium | × | × | √ | -| Maximum | × | √ | √ | -| Concatenate | √ | √ | √ | -| Dot | × | √ | √ | - -### Activations Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| ReLu | √ | √ | √ | -| LeakyReLu | √ | √ | √ | -| PReLU | √ | √ | √ | -| ELU | √ | √ | √ | -| ThresholdedReLU | √ | √ | √ | -| Softmax | √ | √ | √ | -| Argmax | √ | × | × | -| Sigmoid | √ | √ | √ | -| TanH | √ | √ | √ | -| Absolute Value | √ | × | × | -| Power | √ | √ | × | -| Exp | √ | √ | × | -| Linear | × | √ | √ | -| Log | √ | √ | × | -| BNLL | √ | × | × | -| Bias | √ | × | × | -| Scale | √ | × | × | - -### Utility Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Argmax | √ | × | × | -| Slicing | √ | × | × | -| Eltwise | √ | × | × | -| Parameter | √ | × | × | -| Reduction | √ | × | × | -| Silence | √ | × | × | - -### Loss Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Multinomial Logistic Loss | √ | × | × | -| Infogain Loss | √ | × | × | -| Softmax with Loss | √ | × | √ | -| Sum-of-Squares/Euclidean | √ | × | × | -| Hinge / Margin | √ | √ | √ | -| Sigmoid Cross-Entropy Loss| √ | × | √ | -| Accuracy / Top-k layer | √ | × | × | -| Contrastive Loss | √ | × | × | - -### Normalization Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| BatchNormalization | √ | √ | √ | -| LRN | √ | × | × | -| MVN | √ | × | × | - -### Noise Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| GaussianNoise | × | √ | √ | -| GaussianDropout | √ | √ | √ | -| AlphaDropout | √ | √ | √ | - -### Layer Wrappers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| TimeDistributed | × | √ | √ | -| Bidirectional | × | √ | √ | - -### Custom Layers -| Caffe | Keras | Tensorflow | -| :-----------: | :----------: | :---------: | -| √ | √ | Use Keras API for custom layers | - -## Additional Notes: -* Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to here: https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md. -* Documentation for writing your own Keras layers is found here: https://keras.io/layers/writing-your-own-keras-layers/ -* Caffe 3D layers can be used in Caffe 2.0 - -## Documentation for Caffe, Keras, and Tensorflow layers -* Documentation for all Keras Layers is found here: https://keras.io/layers/about-keras-layers/ -* Documentation for all Caffe Layers is found here: http://caffe.berkeleyvision.org/tutorial/layers.html -* Documentation for all Tensorflow Layers is found here: https://www.tensorflow.org/api_docs/python/tf/layers From 66b9b6aef09c0e37949ddd34d7dd029d0ac167b7 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Tue, 30 Oct 2018 11:53:00 -0400 Subject: [PATCH 04/12] Fixed errors --- docs/source/supported_layers.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index e05635247..5c8707264 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -1,5 +1,5 @@ # Layers supported in Caffe, Keras, and Tensorflow -Below is are table showing which layers are supported by Caffe, Keras, and Tensorflow: +Below are tables showing which layers are supported by Caffe, Keras, and Tensorflow: ### Core Layers | Layer | Caffe | Keras | Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | @@ -134,7 +134,6 @@ Below is are table showing which layers are supported by Caffe, Keras, and Tenso | Layer | Caffe | Keras | Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | | BatchNormalization | √ | √ | √ | -| LRN | √ | × | × | | MVN | √ | × | × | ### Noise Layers | Layer | Caffe | Keras | Tensorflow | @@ -148,14 +147,15 @@ Below is are table showing which layers are supported by Caffe, Keras, and Tenso | TimeDistributed | × | √ | √ | | Bidirectional | × | √ | √ | ### Custom Layers -| Caffe | Keras | Tensorflow | -| :-----------: | :----------: | :---------: | -| √ | √ | Use Keras API for custom layers | +| Layer | Caffe | Keras | Tensorflow | +| :-----------: | :----------: | :---------: | :---------: | +| Custom Layers | √ | √ | Use Keras API for custom layers | +| LRN | √ | √ | √ | + ## Additional Notes: -* Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to here: https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md. -* Documentation for writing your own Keras layers is found here: https://keras.io/layers/writing-your-own-keras-layers/ -* Caffe 3D layers can be used in Caffe 2.0 +* Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to [here](https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md.) +* Documentation for writing your own Keras layers is found [here](https://keras.io/layers/writing-your-own-keras-layers/) ## Documentation for Caffe, Keras, and Tensorflow layers -* Documentation for all Keras Layers is found here: https://keras.io/layers/about-keras-layers/ -* Documentation for all Caffe Layers is found here: http://caffe.berkeleyvision.org/tutorial/layers.html -* Documentation for all Tensorflow Layers is found here: https://www.tensorflow.org/api_docs/python/tf/layers +* Documentation for all Keras Layers is found [here](https://keras.io/layers/about-keras-layers/) +* Documentation for all Caffe Layers is found [here](http://caffe.berkeleyvision.org/tutorial/layers.html) +* Documentation for all Tensorflow Layers is found [here](https://www.tensorflow.org/api_docs/python/tf/layers) From 6a9e4e3725d0c4d2e392624f5ae33ec01d07fb04 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Wed, 31 Oct 2018 17:08:11 -0400 Subject: [PATCH 05/12] Update supported_layers.md --- docs/source/supported_layers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index 5c8707264..80278c645 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -38,7 +38,7 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf | ZeroPadding 3D | × | √ | √ | | Im2Col | √ | × | × | | Spatial Pyramid Pooling | √ | × | × | -* Upsampling in Caffe can be done by using methods shown here: https://gist.github.com/tnarihi/54744612d35776f53278 +* Upsampling in Caffe can be done by using methods shown [here](https://gist.github.com/tnarihi/54744612d35776f53278) ### Pooling Layers | Layer | Caffe | Keras | Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | @@ -149,7 +149,7 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf ### Custom Layers | Layer | Caffe | Keras | Tensorflow | | :-----------: | :----------: | :---------: | :---------: | -| Custom Layers | √ | √ | Use Keras API for custom layers | +| | √ | √ | Use Keras API for custom layers | | LRN | √ | √ | √ | ## Additional Notes: From 438f104ba40e31f24fd4db850c66db0a38bfcef7 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Sun, 4 Nov 2018 17:04:19 -0500 Subject: [PATCH 06/12] Update supported_layers.md --- docs/source/supported_layers.md | 273 ++++++++++++++++---------------- 1 file changed, 138 insertions(+), 135 deletions(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index 80278c645..c91e01c84 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -1,156 +1,159 @@ # Layers supported in Caffe, Keras, and Tensorflow + +The columns Frabrik Caffe, Fabrik Keras, and Fabrik Tensorflow show which layers are currently supported by Fabrik in those libraries. + Below are tables showing which layers are supported by Caffe, Keras, and Tensorflow: ### Core Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Activation | √ | √ | √ | -| ActivityRegularization | √ | √ | √ | -| Dense | × | √ | √ | -| Dropout | √ | √ | √ | -| Flatten | √ | √ | √ | -| Lambda | × | √ | √ | -| Masking | √ | √ | √ | -| Permute | × | √ | √ | -| Repeat Vector | × | √ | √ | -| Reshape | √ | √ | √ | -| Spatial Dropout 1D | × | √ | √ | -| Spatial Dropout 2D | × | √ | √ | -| Spatial Dropout 3D | × | √ | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| Activation | √ | √ | √ | √× | √× | √× | +| ActivityRegularization | √ | √ | √ |√× | √× | √× | +| Dense | × | √ | √ |√× | √× | √× | +| Dropout | √ | √ | √ |√× | √× | √× | +| Flatten | √ | √ | √ |√× | √× | √× | +| Lambda | × | √ | √ |√× | √× | √× | +| Masking | √ | √ | √ |√× | √× | √× | +| Permute | × | √ | √ |√× | √× | √× | +| Repeat Vector | × | √ | √ |√× | √× | √× | +| Reshape | √ | √ | √ |√× | √× | √× | +| Spatial Dropout 1D | × | √ | √ |√× | √× | √× | +| Spatial Dropout 2D | × | √ | √ |√× | √× | √× | +| Spatial Dropout 3D | × | √ | √ |√× | √× | √× | #### Convolutional Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Conv1D | √ | √ | √ | -| Conv2D | √ | √ | √ | -| DepthwiseConv2D | × | × | √ | -| SeperableConv1D | × | √ | √ | -| SeperableConv2D | × | √ | √ | -| Conv2DTranspose | √ | √ | √ | -| Conv3D | √ | √ | √ | -| Conv3DTranspose | √ | √ | √ | -| Cropping1D | √ | √ | √ | -| Cropping2D | √ | √ | √ | -| Cropping3D | √ | √ | √ | -| Upsampling 1D | √ | √ | √ | -| Upsampling 2D | √ | √ | √ | -| Upsampling 3D | √ | √ | √ | -| ZeroPadding 1D | × | √ | √ | -| ZeroPadding 2D | × | √ | √ | -| ZeroPadding 3D | × | √ | √ | -| Im2Col | √ | × | × | -| Spatial Pyramid Pooling | √ | × | × | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| Conv1D | √ | √ | √ |√× | √× | √× | +| Conv2D | √ | √ | √ |√× | √× | √× | +| DepthwiseConv2D | × | × | √ |√× | √× | √× | +| SeperableConv1D | × | √ | √ |√× | √× | √× | +| SeperableConv2D | × | √ | √ |√× | √× | √× | +| Conv2DTranspose | √ | √ | √ |√× | √× | √× | +| Conv3D | √ | √ | √ |√× | √× | √× | +| Conv3DTranspose | √ | √ | √ |√× | √× | √× | +| Cropping1D | √ | √ | √ |√× | √× | √× | +| Cropping2D | √ | √ | √ |√× | √× | √× | +| Cropping3D | √ | √ | √ |√× | √× | √× | +| Upsampling 1D | √ | √ | √ |√× | √× | √× | +| Upsampling 2D | √ | √ | √ |√× | √× | √× | +| Upsampling 3D | √ | √ | √ |√× | √× | √× | +| ZeroPadding 1D | × | √ | √ |√× | √× | √× | +| ZeroPadding 2D | × | √ | √ |√× | √× | √× | +| ZeroPadding 3D | × | √ | √ |√× | √× | √× | +| Im2Col | √ | × | × |√× | √× | √× | +| Spatial Pyramid Pooling | √ | × | × |√× | √× | √× | * Upsampling in Caffe can be done by using methods shown [here](https://gist.github.com/tnarihi/54744612d35776f53278) ### Pooling Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| MaxPooling1D | √ | √ | √ | -| MaxPooling2D | √ | √ | √ | -| MaxPooling3D | √ | √ | √ | -| AveragePooling1D | √ | √ | √ | -| AveragePooling2D | √ | √ | √ | -| AveragePooling3D | √ | √ | √ | -| GlobalMaxPooling1D | × | √ | √ | -| GlobalAveragePooling1D | × | √ | √ | -| GlobalMaxPooling2D | × | √ | √ | -| GlobalAveragePooling2D | × | √ | √ | -| GlobalMaxPooling3D | × | √ | √ | -| GlobalAveragePooling3D | × | √ | √ | -| Stochastic Pooling | √ | × | × | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| MaxPooling1D | √ | √ | √ |√× | √× | √× | +| MaxPooling2D | √ | √ | √ |√× | √× | √× | +| MaxPooling3D | √ | √ | √ |√× | √× | √× | +| AveragePooling1D | √ | √ | √ |√× | √× | √× | +| AveragePooling2D | √ | √ | √ |√× | √× | √× | +| AveragePooling3D | √ | √ | √ |√× | √× | √× | +| GlobalMaxPooling1D | × | √ | √ |√× | √× | √× | +| GlobalAveragePooling1D | × | √ | √ |√× | √× | √× | +| GlobalMaxPooling2D | × | √ | √ |√× | √× | √× | +| GlobalAveragePooling2D | × | √ | √ |√× | √× | √× | +| GlobalMaxPooling3D | × | √ | √ |√× | √× | √× | +| GlobalAveragePooling3D | × | √ | √ |√× | √× | √× | +| Stochastic Pooling | √ | × | × |√× | √× | √× | ### Locally-connected Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| LocallyConnected1D | × | √ | √ | -| LocallyConnected2D | × | √ | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| LocallyConnected1D | × | √ | √ |√× | √× | √× | +| LocallyConnected2D | × | √ | √ |√× | √× | √× | ### Recurrent Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| RNN | √ | √ | √ | -| SimpleRNN | × | √ | √ | -| GRU | × | √ | √ | -| LSTM | √ | √ | √ | -| ConvLSTM2D | × | √ | √ | -| SimpleRNNCell | × | √ | √ | -| GRUCell | × | √ | √ | -| LSTMCell | × | √ | √ | -| CuDDNGRU | × | √ | √ | -| CuDDNLSTM | × | √ | √ | -| StackedRNNCell | × | × | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| RNN | √ | √ | √ |√× | √× | √× | +| SimpleRNN | × | √ | √ |√× | √× | √× | +| GRU | × | √ | √ |√× | √× | √× | +| LSTM | √ | √ | √ |√× | √× | √× | +| ConvLSTM2D | × | √ | √ |√× | √× | √× | +| SimpleRNNCell | × | √ | √ |√× | √× | √× | +| GRUCell | × | √ | √ |√× | √× | √× | +| LSTMCell | × | √ | √ |√× | √× | √× | +| CuDDNGRU | × | √ | √ |√× | √× | √× | +| CuDDNLSTM | × | √ | √ |√× | √× | √× | +| StackedRNNCell | × | × | √ |√× | √× | √× | ### Embedding Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Embedding | √ | √ | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| Embedding | √ | √ | √ |√× | √× | √× | ### Merge Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Add | × | √ | √ | -| Subtract | × | √ | √ | -| Multiply | × | √ | √ | -| Average | × | √ | √ | -| Minium | × | × | √ | -| Maximum | × | √ | √ | -| Concatenate | √ | √ | √ | -| Dot | × | √ | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| Add | × | √ | √ |√× | √× | √× | +| Subtract | × | √ | √ |√× | √× | √× | +| Multiply | × | √ | √ |√× | √× | √× | +| Average | × | √ | √ |√× | √× | √× | +| Minium | × | × | √ |√× | √× | √× | +| Maximum | × | √ | √ |√× | √× | √× | +| Concatenate | √ | √ | √ |√× | √× | √× | +| Dot | × | √ | √ |√× | √× | √× | ### Activations Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| ReLu | √ | √ | √ | -| LeakyReLu | √ | √ | √ | -| PReLU | √ | √ | √ | -| ELU | √ | √ | √ | -| ThresholdedReLU | √ | √ | √ | -| Softmax | √ | √ | √ | -| Argmax | √ | × | × | -| Sigmoid | √ | √ | √ | -| TanH | √ | √ | √ | -| Absolute Value | √ | × | × | -| Power | √ | √ | × | -| Exp | √ | √ | × | -| Linear | × | √ | √ | -| Log | √ | √ | × | -| BNLL | √ | × | × | -| Bias | √ | × | × | -| Scale | √ | × | × | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| ReLu | √ | √ | √ |√× | √× | √× | +| LeakyReLu | √ | √ | √ |√× | √× | √× | +| PReLU | √ | √ | √ |√× | √× | √× | +| ELU | √ | √ | √ |√× | √× | √× | +| ThresholdedReLU | √ | √ | √ |√× | √× | √× | +| Softmax | √ | √ | √ |√× | √× | √× | +| Argmax | √ | × | × |√× | √× | √× | +| Sigmoid | √ | √ | √ |√× | √× | √× | +| TanH | √ | √ | √ |√× | √× | √× | +| Absolute Value | √ | × | × |√× | √× | √× | +| Power | √ | √ | × |√× | √× | √× | +| Exp | √ | √ | × |√× | √× | √× | +| Linear | × | √ | √ |√× | √× | √× | +| Log | √ | √ | × |√× | √× | √× | +| BNLL | √ | × | × |√× | √× | √× | +| Bias | √ | × | × |√× | √× | √× | +| Scale | √ | × | × |√× | √× | √× | ### Utility Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Argmax | √ | × | × | -| Slicing | √ | × | × | -| Eltwise | √ | × | × | -| Parameter | √ | × | × | -| Reduction | √ | × | × | -| Silence | √ | × | × | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| Argmax | √ | × | × |√× | √× | √× | +| Slicing | √ | × | × |√× | √× | √× | +| Eltwise | √ | × | × |√× | √× | √× | +| Parameter | √ | × | × |√× | √× | √× | +| Reduction | √ | × | × |√× | √× | √× | +| Silence | √ | × | × |√× | √× | √× | ### Loss Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| Multinomial Logistic Loss | √ | × | × | -| Infogain Loss | √ | × | × | -| Softmax with Loss | √ | × | √ | -| Sum-of-Squares/Euclidean | √ | × | × | -| Hinge / Margin | √ | √ | √ | -| Sigmoid Cross-Entropy Loss| √ | × | √ | -| Accuracy / Top-k layer | √ | × | × | -| Contrastive Loss | √ | × | × | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| Multinomial Logistic Loss | √ | × | × |√× | √× | √× | +| Infogain Loss | √ | × | × |√× | √× | √× | +| Softmax with Loss | √ | × | √ |√× | √× | √× | +| Sum-of-Squares/Euclidean | √ | × | × |√× | √× | √× | +| Hinge / Margin | √ | √ | √ |√× | √× | √× | +| Sigmoid Cross-Entropy Loss| √ | × | √ |√× | √× | √× | +| Accuracy / Top-k layer | √ | × | × |√× | √× | √× | +| Contrastive Loss | √ | × | × |√× | √× | √× | ### Normalization Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| BatchNormalization | √ | √ | √ | -| MVN | √ | × | × | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| BatchNormalization | √ | √ | √ |√× | √× | √× | +| MVN | √ | × | × |√× | √× | √× | ### Noise Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| GaussianNoise | × | √ | √ | -| GaussianDropout | √ | √ | √ | -| AlphaDropout | √ | √ | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| GaussianNoise | × | √ | √ |√× | √× | √× | +| GaussianDropout | √ | √ | √ |√× | √× | √× | +| AlphaDropout | √ | √ | √ |√× | √× | √× | ### Layer Wrappers -| Layer | Caffe | Keras | Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | -| TimeDistributed | × | √ | √ | -| Bidirectional | × | √ | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| TimeDistributed | × | √ | √ |√× | √× | √× | +| Bidirectional | × | √ | √ |√× | √× | √× | ### Custom Layers -| Layer | Caffe | Keras | Tensorflow | -| :-----------: | :----------: | :---------: | :---------: | -| | √ | √ | Use Keras API for custom layers | -| LRN | √ | √ | √ | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| :-----------: | :----------: | :---------: | :---------: | :-----------: | :--------------: | :-----------------: | +| | √ | √ | Use Keras API for custom layers |√× | √× | √× +| LRN | √ | √ | √ |√× | √× | √× | ## Additional Notes: * Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to [here](https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md.) From dc14f86ceb6fbaff0c82c52555d668b5299d8c3c Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Sun, 4 Nov 2018 22:39:12 -0500 Subject: [PATCH 07/12] Keras Updated --- docs/source/supported_layers.md | 246 ++++++++++++++++---------------- 1 file changed, 123 insertions(+), 123 deletions(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index c91e01c84..5478cf924 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -1,159 +1,159 @@ # Layers supported in Caffe, Keras, and Tensorflow -The columns Frabrik Caffe, Fabrik Keras, and Fabrik Tensorflow show which layers are currently supported by Fabrik in those libraries. +The columns Fabrik Caffe, Fabrik Keras, and Fabrik Tensorflow show which layers are currently supported by Fabrik in those libraries. Below are tables showing which layers are supported by Caffe, Keras, and Tensorflow: ### Core Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Activation | √ | √ | √ | √× | √× | √× | -| ActivityRegularization | √ | √ | √ |√× | √× | √× | -| Dense | × | √ | √ |√× | √× | √× | -| Dropout | √ | √ | √ |√× | √× | √× | -| Flatten | √ | √ | √ |√× | √× | √× | -| Lambda | × | √ | √ |√× | √× | √× | -| Masking | √ | √ | √ |√× | √× | √× | -| Permute | × | √ | √ |√× | √× | √× | -| Repeat Vector | × | √ | √ |√× | √× | √× | -| Reshape | √ | √ | √ |√× | √× | √× | -| Spatial Dropout 1D | × | √ | √ |√× | √× | √× | -| Spatial Dropout 2D | × | √ | √ |√× | √× | √× | -| Spatial Dropout 3D | × | √ | √ |√× | √× | √× | +| Activation | √ | √ | √ | √× | √ | √× | +| ActivityRegularization | √ | √ | √ |√× | √ | √× | +| Dense | × | √ | √ |√× | √ | √× | +| Dropout | √ | √ | √ |√× | √ | √× | +| Flatten | √ | √ | √ |√× | √ | √× | +| Lambda | × | √ | √ |√× | × | √× | +| Masking | √ | √ | √ |√× | √ | √× | +| Permute | × | √ | √ |√× | √ | √× | +| Repeat Vector | × | √ | √ |√× | √ | √× | +| Reshape | √ | √ | √ |√× | √ | √× | +| Spatial Dropout 1D | × | √ | √ |√× | × | √× | +| Spatial Dropout 2D | × | √ | √ |√× | × | √× | +| Spatial Dropout 3D | × | √ | √ |√× | × | √× | #### Convolutional Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Conv1D | √ | √ | √ |√× | √× | √× | -| Conv2D | √ | √ | √ |√× | √× | √× | -| DepthwiseConv2D | × | × | √ |√× | √× | √× | -| SeperableConv1D | × | √ | √ |√× | √× | √× | -| SeperableConv2D | × | √ | √ |√× | √× | √× | -| Conv2DTranspose | √ | √ | √ |√× | √× | √× | -| Conv3D | √ | √ | √ |√× | √× | √× | -| Conv3DTranspose | √ | √ | √ |√× | √× | √× | -| Cropping1D | √ | √ | √ |√× | √× | √× | -| Cropping2D | √ | √ | √ |√× | √× | √× | -| Cropping3D | √ | √ | √ |√× | √× | √× | -| Upsampling 1D | √ | √ | √ |√× | √× | √× | -| Upsampling 2D | √ | √ | √ |√× | √× | √× | -| Upsampling 3D | √ | √ | √ |√× | √× | √× | -| ZeroPadding 1D | × | √ | √ |√× | √× | √× | -| ZeroPadding 2D | × | √ | √ |√× | √× | √× | -| ZeroPadding 3D | × | √ | √ |√× | √× | √× | -| Im2Col | √ | × | × |√× | √× | √× | -| Spatial Pyramid Pooling | √ | × | × |√× | √× | √× | -* Upsampling in Caffe can be done by using methods shown [here](https://gist.github.com/tnarihi/54744612d35776f53278) +| Conv1D | √ | √ | √ |√× | √ | √× | +| Conv2D | √ | √ | √ |√× | √ | √× | +| DepthwiseConv2D | × | × | √ |√× | √ | √× | +| SeperableConv1D | × | √ | √ |√× | √ | √× | +| SeperableConv2D | × | √ | √ |√× | √ | √× | +| Conv2DTranspose | √ | √ | √ |√× | √ | √× | +| Conv3D | √ | √ | √ |√× | √ | √× | +| Conv3DTranspose | √ | √ | √ |√× | × | √× | +| Cropping1D | √ | √ | √ |√× | × | √× | +| Cropping2D | √ | √ | √ |√× | × | √× | +| Cropping3D | √ | √ | √ |√× | × | √× | +| Upsampling 1D | √ | √ | √ |√× | √ | √× | +| Upsampling 2D | √ | √ | √ |√× | √ | √× | +| Upsampling 3D | √ | √ | √ |√× | √ | √× | +| ZeroPadding 1D | × | √ | √ |√× | √ | √× | +| ZeroPadding 2D | × | √ | √ |√× | √ | √× | +| ZeroPadding 3D | × | √ | √ |√× | √ | √× | +| Im2Col | √ | × | × |√× | × | √× | +| Spatial Pyramid Pooling | √ | × | × |√× | × | √× | +* Upsampling in Caffe can be done by using methods shown [here](https://gist.github.com/tnarihi/54744612d35776f53278) ### Pooling Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| MaxPooling1D | √ | √ | √ |√× | √× | √× | -| MaxPooling2D | √ | √ | √ |√× | √× | √× | -| MaxPooling3D | √ | √ | √ |√× | √× | √× | -| AveragePooling1D | √ | √ | √ |√× | √× | √× | -| AveragePooling2D | √ | √ | √ |√× | √× | √× | -| AveragePooling3D | √ | √ | √ |√× | √× | √× | -| GlobalMaxPooling1D | × | √ | √ |√× | √× | √× | -| GlobalAveragePooling1D | × | √ | √ |√× | √× | √× | -| GlobalMaxPooling2D | × | √ | √ |√× | √× | √× | -| GlobalAveragePooling2D | × | √ | √ |√× | √× | √× | -| GlobalMaxPooling3D | × | √ | √ |√× | √× | √× | -| GlobalAveragePooling3D | × | √ | √ |√× | √× | √× | -| Stochastic Pooling | √ | × | × |√× | √× | √× | +| MaxPooling1D | √ | √ | √ |√× | √ | √× | +| MaxPooling2D | √ | √ | √ |√× | √ | √× | +| MaxPooling3D | √ | √ | √ |√× | √ | √× | +| AveragePooling1D | √ | √ | √ |√× | √ | √× | +| AveragePooling2D | √ | √ | √ |√× | √ | √× | +| AveragePooling3D | √ | √ | √ |√× | √ | √× | +| GlobalMaxPooling1D | × | √ | √ |√× | × | √× | +| GlobalAveragePooling1D | × | √ | √ |√× | × | √× | +| GlobalMaxPooling2D | × | √ | √ |√× | × | √× | +| GlobalAveragePooling2D | × | √ | √ |√× | × | √× | +| GlobalMaxPooling3D | × | √ | √ |√× | × | √× | +| GlobalAveragePooling3D | × | √ | √ |√× | × | √× | +| Stochastic Pooling | √ | × | × |√× | × | √× | ### Locally-connected Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| LocallyConnected1D | × | √ | √ |√× | √× | √× | -| LocallyConnected2D | × | √ | √ |√× | √× | √× | +| LocallyConnected1D | × | √ | √ |√× | √ | √× | +| LocallyConnected2D | × | √ | √ |√× | √ | √× | ### Recurrent Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| RNN | √ | √ | √ |√× | √× | √× | -| SimpleRNN | × | √ | √ |√× | √× | √× | -| GRU | × | √ | √ |√× | √× | √× | -| LSTM | √ | √ | √ |√× | √× | √× | -| ConvLSTM2D | × | √ | √ |√× | √× | √× | -| SimpleRNNCell | × | √ | √ |√× | √× | √× | -| GRUCell | × | √ | √ |√× | √× | √× | -| LSTMCell | × | √ | √ |√× | √× | √× | -| CuDDNGRU | × | √ | √ |√× | √× | √× | -| CuDDNLSTM | × | √ | √ |√× | √× | √× | -| StackedRNNCell | × | × | √ |√× | √× | √× | +| RNN | √ | √ | √ |√× | × | √× | +| SimpleRNN | × | √ | √ |√× | √ | √× | +| GRU | × | √ | √ |√× | √ | √× | +| LSTM | √ | √ | √ |√× | √ | √× | +| ConvLSTM2D | × | √ | √ |√× | × | √× | +| SimpleRNNCell | × | √ | √ |√× | × | √× | +| GRUCell | × | √ | √ |√× | × | √× | +| LSTMCell | × | √ | √ |√× | × | √× | +| CuDDNGRU | × | √ | √ |√× | × | √× | +| CuDDNLSTM | × | √ | √ |√× | × | √× | +| StackedRNNCell | × | × | √ |√× | × | √× | ### Embedding Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Embedding | √ | √ | √ |√× | √× | √× | +| Embedding | √ | √ | √ |√× | √ | √× | ### Merge Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Add | × | √ | √ |√× | √× | √× | -| Subtract | × | √ | √ |√× | √× | √× | -| Multiply | × | √ | √ |√× | √× | √× | -| Average | × | √ | √ |√× | √× | √× | -| Minium | × | × | √ |√× | √× | √× | -| Maximum | × | √ | √ |√× | √× | √× | -| Concatenate | √ | √ | √ |√× | √× | √× | -| Dot | × | √ | √ |√× | √× | √× | +| Add | × | √ | √ |√× | √ | √× | +| Subtract | × | √ | √ |√× | × | √× | +| Multiply | × | √ | √ |√× | √ | √× | +| Average | × | √ | √ |√× | √ | √× | +| Minium | × | × | √ |√× | × | √× | +| Maximum | × | √ | √ |√× | √ | √× | +| Concatenate | √ | √ | √ |√× | √ | √× | +| Dot | × | √ | √ |√× | √ | √× | ### Activations Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| ReLu | √ | √ | √ |√× | √× | √× | -| LeakyReLu | √ | √ | √ |√× | √× | √× | -| PReLU | √ | √ | √ |√× | √× | √× | -| ELU | √ | √ | √ |√× | √× | √× | -| ThresholdedReLU | √ | √ | √ |√× | √× | √× | -| Softmax | √ | √ | √ |√× | √× | √× | -| Argmax | √ | × | × |√× | √× | √× | -| Sigmoid | √ | √ | √ |√× | √× | √× | -| TanH | √ | √ | √ |√× | √× | √× | -| Absolute Value | √ | × | × |√× | √× | √× | -| Power | √ | √ | × |√× | √× | √× | -| Exp | √ | √ | × |√× | √× | √× | -| Linear | × | √ | √ |√× | √× | √× | -| Log | √ | √ | × |√× | √× | √× | -| BNLL | √ | × | × |√× | √× | √× | -| Bias | √ | × | × |√× | √× | √× | -| Scale | √ | × | × |√× | √× | √× | +| ReLu | √ | √ | √ |√× | √ | √× | +| LeakyReLu | √ | √ | √ |√× | √ | √× | +| PReLU | √ | √ | √ |√× | √ | √× | +| ELU | √ | √ | √ |√× | √ | √× | +| ThresholdedReLU | √ | √ | √ |√× | √ | √× | +| Softmax | √ | √ | √ |√× | √ | √× | +| Argmax | √ | × | × |√× | × | √× | +| Sigmoid | √ | √ | √ |√× | √ | √× | +| Hard Sigmoid | √ | √ | √ |√× | √ | √× | +| TanH | √ | √ | √ |√× | √ | √× | +| Absolute Value | √ | × | × |√× | × | √× | +| Power | √ | √ | × |√× | × | √× | +| Exp | √ | √ | × |√× | × | √× | +| Linear | × | √ | √ |√× | × | √× | +| Log | √ | √ | × |√× | × | √× | +| BNLL | √ | × | × |√× | × | √× | +| Bias | √ | × | × |√× | √ | √× | +| Scale | √ | × | × |√× | √ | √× | ### Utility Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Argmax | √ | × | × |√× | √× | √× | -| Slicing | √ | × | × |√× | √× | √× | -| Eltwise | √ | × | × |√× | √× | √× | -| Parameter | √ | × | × |√× | √× | √× | -| Reduction | √ | × | × |√× | √× | √× | -| Silence | √ | × | × |√× | √× | √× | +| Slicing | √ | × | × |√× | × | √× | +| Eltwise | √ | × | × |√× | √ | √× | +| Parameter | √ | × | × |√× | × | √× | +| Reduction | √ | × | × |√× | × | √× | +| Silence | √ | × | × |√× | × | √× | ### Loss Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Multinomial Logistic Loss | √ | × | × |√× | √× | √× | -| Infogain Loss | √ | × | × |√× | √× | √× | -| Softmax with Loss | √ | × | √ |√× | √× | √× | -| Sum-of-Squares/Euclidean | √ | × | × |√× | √× | √× | -| Hinge / Margin | √ | √ | √ |√× | √× | √× | -| Sigmoid Cross-Entropy Loss| √ | × | √ |√× | √× | √× | -| Accuracy / Top-k layer | √ | × | × |√× | √× | √× | -| Contrastive Loss | √ | × | × |√× | √× | √× | +| Multinomial Logistic Loss | √ | × | × |√× | × | √× | +| Infogain Loss | √ | × | × |√× | × | √× | +| Softmax with Loss | √ | × | √ |√× | × | √× | +| Sum-of-Squares/Euclidean | √ | × | × |√× | × | √× | +| Hinge / Margin | √ | √ | √ |√× | × | √× | +| Sigmoid Cross-Entropy Loss| √ | × | √ |√× | × | √× | +| Accuracy / Top-k layer | √ | × | × |√× | × | √× | +| Contrastive Loss | √ | × | × |√× | × | √× | ### Normalization Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| BatchNormalization | √ | √ | √ |√× | √× | √× | -| MVN | √ | × | × |√× | √× | √× | +| BatchNormalization | √ | √ | √ |√× | √ | √× | +| MVN | √ | × | × |√× | × | √× | ### Noise Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| GaussianNoise | × | √ | √ |√× | √× | √× | -| GaussianDropout | √ | √ | √ |√× | √× | √× | -| AlphaDropout | √ | √ | √ |√× | √× | √× | +| GaussianNoise | × | √ | √ |√× | √ | √× | +| GaussianDropout | √ | √ | √ |√× | √ | √× | +| AlphaDropout | √ | √ | √ |√× | √ | √× | ### Layer Wrappers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| TimeDistributed | × | √ | √ |√× | √× | √× | -| Bidirectional | × | √ | √ |√× | √× | √× | +| TimeDistributed | × | √ | √ |√× | √ | √× | +| Bidirectional | × | √ | √ |√× | √ | √× | ### Custom Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Frabrik Keras | Frabrik Tensorflow | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------: | :----------: | :---------: | :---------: | :-----------: | :--------------: | :-----------------: | -| | √ | √ | Use Keras API for custom layers |√× | √× | √× -| LRN | √ | √ | √ |√× | √× | √× | +| | √ | √ | Use Keras API for custom layers | NA | NA | NA| +| LRN | √ | √ | √ |√× | √ | √× | ## Additional Notes: * Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to [here](https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md.) From f92957506081333549ea121d1bbf51035c6d61bd Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Sun, 4 Nov 2018 22:57:49 -0500 Subject: [PATCH 08/12] Updated Tensorflow --- docs/source/supported_layers.md | 216 ++++++++++++++++---------------- 1 file changed, 110 insertions(+), 106 deletions(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index 5478cf924..4a6d49e71 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -6,58 +6,59 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf ### Core Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Activation | √ | √ | √ | √× | √ | √× | -| ActivityRegularization | √ | √ | √ |√× | √ | √× | -| Dense | × | √ | √ |√× | √ | √× | -| Dropout | √ | √ | √ |√× | √ | √× | -| Flatten | √ | √ | √ |√× | √ | √× | -| Lambda | × | √ | √ |√× | × | √× | -| Masking | √ | √ | √ |√× | √ | √× | -| Permute | × | √ | √ |√× | √ | √× | -| Repeat Vector | × | √ | √ |√× | √ | √× | -| Reshape | √ | √ | √ |√× | √ | √× | -| Spatial Dropout 1D | × | √ | √ |√× | × | √× | -| Spatial Dropout 2D | × | √ | √ |√× | × | √× | -| Spatial Dropout 3D | × | √ | √ |√× | × | √× | +| Activation | √ | √ | √ | √× | √ | √ | +| ActivityRegularization | √ | √ | √ |√× | √ | × | +| Dense | × | √ | √ |√× | √ | × | +| Dropout | √ | √ | √ |√× | √ | √ | +| Flatten | √ | √ | √ |√× | √ | × | +| Lambda | × | √ | √ |√× | × | × | +| Masking | √ | √ | √ |√× | √ | × | +| Permute | × | √ | √ |√× | √ | × | +| Repeat Vector | × | √ | √ |√× | √ | √ | +| Reshape | √ | √ | √ |√× | √ | × | +| Spatial Dropout 1D | × | √ | √ |√× | × | × | +| Spatial Dropout 2D | × | √ | √ |√× | × | × | +| Spatial Dropout 3D | × | √ | √ |√× | × | × | #### Convolutional Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Conv1D | √ | √ | √ |√× | √ | √× | -| Conv2D | √ | √ | √ |√× | √ | √× | -| DepthwiseConv2D | × | × | √ |√× | √ | √× | -| SeperableConv1D | × | √ | √ |√× | √ | √× | -| SeperableConv2D | × | √ | √ |√× | √ | √× | -| Conv2DTranspose | √ | √ | √ |√× | √ | √× | -| Conv3D | √ | √ | √ |√× | √ | √× | -| Conv3DTranspose | √ | √ | √ |√× | × | √× | -| Cropping1D | √ | √ | √ |√× | × | √× | -| Cropping2D | √ | √ | √ |√× | × | √× | -| Cropping3D | √ | √ | √ |√× | × | √× | -| Upsampling 1D | √ | √ | √ |√× | √ | √× | -| Upsampling 2D | √ | √ | √ |√× | √ | √× | -| Upsampling 3D | √ | √ | √ |√× | √ | √× | -| ZeroPadding 1D | × | √ | √ |√× | √ | √× | -| ZeroPadding 2D | × | √ | √ |√× | √ | √× | -| ZeroPadding 3D | × | √ | √ |√× | √ | √× | -| Im2Col | √ | × | × |√× | × | √× | -| Spatial Pyramid Pooling | √ | × | × |√× | × | √× | +| Conv1D | √ | √ | √ |√× | √ | √ | +| Conv2D | √ | √ | √ |√× | √ | √ | +| DepthwiseConv2D | × | × | √ |√× | √ | √ | +| SeperableConv1D | × | √ | √ |√× | √ | × | +| SeperableConv2D | × | √ | √ |√× | √ | × | +| Conv2DTranspose | √ | √ | √ |√× | √ | × | +| Conv3D | √ | √ | √ |√× | √ | √ | +| Conv3DTranspose | √ | √ | √ |√× | × | × | +| Deconvolution | × | √ | √ |√× | × | √ | +| Cropping1D | √ | √ | √ |√× | × | × | +| Cropping2D | √ | √ | √ |√× | × | × | +| Cropping3D | √ | √ | √ |√× | × | × | +| Upsampling 1D | √ | √ | √ |√× | √ | × | +| Upsampling 2D | √ | √ | √ |√× | √ | × | +| Upsampling 3D | √ | √ | √ |√× | √ | × | +| ZeroPadding 1D | × | √ | √ |√× | √ | √ | +| ZeroPadding 2D | × | √ | √ |√× | √ | √ | +| ZeroPadding 3D | × | √ | √ |√× | √ | √ | +| Im2Col | √ | × | × |√× | × | × | +| Spatial Pyramid Pooling | √ | × | × |√× | × | × | * Upsampling in Caffe can be done by using methods shown [here](https://gist.github.com/tnarihi/54744612d35776f53278) ### Pooling Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| MaxPooling1D | √ | √ | √ |√× | √ | √× | -| MaxPooling2D | √ | √ | √ |√× | √ | √× | -| MaxPooling3D | √ | √ | √ |√× | √ | √× | -| AveragePooling1D | √ | √ | √ |√× | √ | √× | -| AveragePooling2D | √ | √ | √ |√× | √ | √× | -| AveragePooling3D | √ | √ | √ |√× | √ | √× | -| GlobalMaxPooling1D | × | √ | √ |√× | × | √× | -| GlobalAveragePooling1D | × | √ | √ |√× | × | √× | -| GlobalMaxPooling2D | × | √ | √ |√× | × | √× | -| GlobalAveragePooling2D | × | √ | √ |√× | × | √× | -| GlobalMaxPooling3D | × | √ | √ |√× | × | √× | -| GlobalAveragePooling3D | × | √ | √ |√× | × | √× | -| Stochastic Pooling | √ | × | × |√× | × | √× | +| MaxPooling1D | √ | √ | √ |√× | √ | × | +| MaxPooling2D | √ | √ | √ |√× | √ | √ | +| MaxPooling3D | √ | √ | √ |√× | √ | √ | +| AveragePooling1D | √ | √ | √ |√× | √ | × | +| AveragePooling2D | √ | √ | √ |√× | √ | × | +| AveragePooling3D | √ | √ | √ |√× | √ | × | +| GlobalMaxPooling1D | × | √ | √ |√× | × | × | +| GlobalAveragePooling1D | × | √ | √ |√× | × | × | +| GlobalMaxPooling2D | × | √ | √ |√× | × | × | +| GlobalAveragePooling2D | × | √ | √ |√× | × | × | +| GlobalMaxPooling3D | × | √ | √ |√× | × | × | +| GlobalAveragePooling3D | × | √ | √ |√× | × | × | +| Stochastic Pooling | √ | × | × |√× | × | × | ### Locally-connected Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | @@ -65,95 +66,96 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf | LocallyConnected1D | × | √ | √ |√× | √ | √× | | LocallyConnected2D | × | √ | √ |√× | √ | √× | ### Recurrent Layers -| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| RNN | √ | √ | √ |√× | × | √× | -| SimpleRNN | × | √ | √ |√× | √ | √× | -| GRU | × | √ | √ |√× | √ | √× | -| LSTM | √ | √ | √ |√× | √ | √× | -| ConvLSTM2D | × | √ | √ |√× | × | √× | -| SimpleRNNCell | × | √ | √ |√× | × | √× | -| GRUCell | × | √ | √ |√× | × | √× | -| LSTMCell | × | √ | √ |√× | × | √× | -| CuDDNGRU | × | √ | √ |√× | × | √× | -| CuDDNLSTM | × | √ | √ |√× | × | √× | -| StackedRNNCell | × | × | √ |√× | × | √× | +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------:| :-----------------: | +| RNN | √ | √ | √ |√× | × | × | +| SimpleRNN | × | √ | √ |√× | √ | × | +| GRU | × | √ | √ |√× | √ | × | +| LSTM | √ | √ | √ |√× | √ | × | +| ConvLSTM2D | × | √ | √ |√× | × | × | +| SimpleRNNCell | × | √ | √ |√× | × | × | +| GRUCell | × | √ | √ |√× | × | × | +| LSTMCell | × | √ | √ |√× | × | × | +| CuDDNGRU | × | √ | √ |√× | × | × | +| CuDDNLSTM | × | √ | √ |√× | × | × | +| StackedRNNCell | × | × | √ |√× | × | × | ### Embedding Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Embedding | √ | √ | √ |√× | √ | √× | +| Embedding | √ | √ | √ |√× | √ | × | ### Merge Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Add | × | √ | √ |√× | √ | √× | -| Subtract | × | √ | √ |√× | × | √× | -| Multiply | × | √ | √ |√× | √ | √× | -| Average | × | √ | √ |√× | √ | √× | -| Minium | × | × | √ |√× | × | √× | -| Maximum | × | √ | √ |√× | √ | √× | -| Concatenate | √ | √ | √ |√× | √ | √× | -| Dot | × | √ | √ |√× | √ | √× | +| Add | × | √ | √ |√× | √ | √ | +| Subtract | × | √ | √ |√× | × | × | +| Multiply | × | √ | √ |√× | √ | √ | +| Average | × | √ | √ |√× | √ | √ | +| Minium | × | × | √ |√× | × | × | +| Maximum | × | √ | √ |√× | √ | × | +| Concatenate | √ | √ | √ |√× | √ | √ | +| Dot | × | √ | √ |√× | √ | √ | ### Activations Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| ReLu | √ | √ | √ |√× | √ | √× | -| LeakyReLu | √ | √ | √ |√× | √ | √× | -| PReLU | √ | √ | √ |√× | √ | √× | -| ELU | √ | √ | √ |√× | √ | √× | -| ThresholdedReLU | √ | √ | √ |√× | √ | √× | -| Softmax | √ | √ | √ |√× | √ | √× | -| Argmax | √ | × | × |√× | × | √× | -| Sigmoid | √ | √ | √ |√× | √ | √× | -| Hard Sigmoid | √ | √ | √ |√× | √ | √× | -| TanH | √ | √ | √ |√× | √ | √× | -| Absolute Value | √ | × | × |√× | × | √× | -| Power | √ | √ | × |√× | × | √× | -| Exp | √ | √ | × |√× | × | √× | -| Linear | × | √ | √ |√× | × | √× | -| Log | √ | √ | × |√× | × | √× | -| BNLL | √ | × | × |√× | × | √× | -| Bias | √ | × | × |√× | √ | √× | -| Scale | √ | × | × |√× | √ | √× | +| ReLu | √ | √ | √ |√× | √ | √ | +| LeakyReLu | √ | √ | √ |√× | √ | √ | +| PReLU | √ | √ | √ |√× | √ | × | +| ELU | √ | √ | √ |√× | √ | √ | +| ThresholdedReLU | √ | √ | √ |√× | √ | × | +| Softmax | √ | √ | √ |√× | √ | √ | +| Argmax | √ | × | × |√× | × | × | +| Sigmoid | √ | √ | √ |√× | √ | √ | +| Hard Sigmoid | √ | √ | √ |√× | √ | × | +| TanH | √ | √ | √ |√× | √ | × | +| SELU | × | √ | √ |√× | × | √ | +| Absolute Value | √ | × | × |√× | × | × | +| Power | √ | √ | × |√× | × | × | +| Exp | √ | √ | × |√× | × | × | +| Linear | × | √ | √ |√× | × | × | +| Log | √ | √ | × |√× | × | × | +| BNLL | √ | × | × |√× | × | × | +| Bias | √ | × | × |√× | √ | √ | +| Scale | √ | × | × |√× | √ | √ | ### Utility Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Slicing | √ | × | × |√× | × | √× | -| Eltwise | √ | × | × |√× | √ | √× | -| Parameter | √ | × | × |√× | × | √× | -| Reduction | √ | × | × |√× | × | √× | -| Silence | √ | × | × |√× | × | √× | +| Slicing | √ | × | × |√× | × | × | +| Eltwise | √ | × | × |√× | √ | √ | +| Parameter | √ | × | × |√× | × | √ | +| Reduction | √ | × | × |√× | × | × | +| Silence | √ | × | × |√× | × | × | ### Loss Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Multinomial Logistic Loss | √ | × | × |√× | × | √× | -| Infogain Loss | √ | × | × |√× | × | √× | -| Softmax with Loss | √ | × | √ |√× | × | √× | -| Sum-of-Squares/Euclidean | √ | × | × |√× | × | √× | -| Hinge / Margin | √ | √ | √ |√× | × | √× | -| Sigmoid Cross-Entropy Loss| √ | × | √ |√× | × | √× | -| Accuracy / Top-k layer | √ | × | × |√× | × | √× | -| Contrastive Loss | √ | × | × |√× | × | √× | +| Multinomial Logistic Loss | √ | × | × |√× | × | × | +| Infogain Loss | √ | × | × |√× | × | × | +| Softmax with Loss | √ | × | √ |√× | × | × | +| Sum-of-Squares/Euclidean | √ | × | × |√× | × | × | +| Hinge / Margin | √ | √ | √ |√× | × | × | +| Sigmoid Cross-Entropy Loss| √ | × | √ |√× | × | × | +| Accuracy / Top-k layer | √ | × | × |√× | × | × | +| Contrastive Loss | √ | × | × |√× | × | × | ### Normalization Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| BatchNormalization | √ | √ | √ |√× | √ | √× | -| MVN | √ | × | × |√× | × | √× | +| BatchNormalization | √ | √ | √ |√× | √ | √ | +| MVN | √ | × | × |√× | × | × | ### Noise Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| GaussianNoise | × | √ | √ |√× | √ | √× | -| GaussianDropout | √ | √ | √ |√× | √ | √× | -| AlphaDropout | √ | √ | √ |√× | √ | √× | +| GaussianNoise | × | √ | √ |√× | √ | × | +| GaussianDropout | √ | √ | √ |√× | √ | × | +| AlphaDropout | √ | √ | √ |√× | √ | √ | ### Layer Wrappers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| TimeDistributed | × | √ | √ |√× | √ | √× | -| Bidirectional | × | √ | √ |√× | √ | √× | +| TimeDistributed | × | √ | √ |√× | √ | × | +| Bidirectional | × | √ | √ |√× | √ | × | ### Custom Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------: | :----------: | :---------: | :---------: | :-----------: | :--------------: | :-----------------: | | | √ | √ | Use Keras API for custom layers | NA | NA | NA| -| LRN | √ | √ | √ |√× | √ | √× | +| LRN | √ | √ | √ |√× | √ | √ | ## Additional Notes: * Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to [here](https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md.) @@ -162,3 +164,5 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf * Documentation for all Keras Layers is found [here](https://keras.io/layers/about-keras-layers/) * Documentation for all Caffe Layers is found [here](http://caffe.berkeleyvision.org/tutorial/layers.html) * Documentation for all Tensorflow Layers is found [here](https://www.tensorflow.org/api_docs/python/tf/layers) + +#### * This documentation is subject to change as more layers are supported. From 16a1705b7072b7809989a38587b10d60c3d8ba6c Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Sun, 4 Nov 2018 23:15:18 -0500 Subject: [PATCH 09/12] Updated Caffe --- docs/source/supported_layers.md | 220 ++++++++++++++++---------------- 1 file changed, 111 insertions(+), 109 deletions(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index 4a6d49e71..745e49e28 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -1,161 +1,163 @@ # Layers supported in Caffe, Keras, and Tensorflow -The columns Fabrik Caffe, Fabrik Keras, and Fabrik Tensorflow show which layers are currently supported by Fabrik in those libraries. +###### The columns Caffe, Keras, and Tensorflow show which layers are supported in those libraries. +###### The columns Fabrik Caffe, Fabrik Keras, and Fabrik Tensorflow show which layers are currently supported by Fabrik in those libraries. + Below are tables showing which layers are supported by Caffe, Keras, and Tensorflow: ### Core Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Activation | √ | √ | √ | √× | √ | √ | -| ActivityRegularization | √ | √ | √ |√× | √ | × | -| Dense | × | √ | √ |√× | √ | × | -| Dropout | √ | √ | √ |√× | √ | √ | -| Flatten | √ | √ | √ |√× | √ | × | -| Lambda | × | √ | √ |√× | × | × | -| Masking | √ | √ | √ |√× | √ | × | -| Permute | × | √ | √ |√× | √ | × | -| Repeat Vector | × | √ | √ |√× | √ | √ | -| Reshape | √ | √ | √ |√× | √ | × | -| Spatial Dropout 1D | × | √ | √ |√× | × | × | -| Spatial Dropout 2D | × | √ | √ |√× | × | × | -| Spatial Dropout 3D | × | √ | √ |√× | × | × | +| Activation | √ | √ | √ | √ | √ | √ | +| ActivityRegularization | √ | √ | √ | × | √ | × | +| Dense | × | √ | √ | × | √ | × | +| Dropout | √ | √ | √ | √ | √ | √ | +| Flatten | √ | √ | √ | √ | √ | × | +| Lambda | × | √ | √ | × | × | × | +| Masking | √ | √ | √ | × | √ | × | +| Permute | × | √ | √ | × | √ | × | +| Repeat Vector | × | √ | √ | × | √ | √ | +| Reshape | √ | √ | √ | √ | √ | × | +| Spatial Dropout 1D | × | √ | √ | × | × | × | +| Spatial Dropout 2D | × | √ | √ | × | × | × | +| Spatial Dropout 3D | × | √ | √ | × | × | × | #### Convolutional Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Conv1D | √ | √ | √ |√× | √ | √ | -| Conv2D | √ | √ | √ |√× | √ | √ | -| DepthwiseConv2D | × | × | √ |√× | √ | √ | -| SeperableConv1D | × | √ | √ |√× | √ | × | -| SeperableConv2D | × | √ | √ |√× | √ | × | -| Conv2DTranspose | √ | √ | √ |√× | √ | × | -| Conv3D | √ | √ | √ |√× | √ | √ | -| Conv3DTranspose | √ | √ | √ |√× | × | × | -| Deconvolution | × | √ | √ |√× | × | √ | -| Cropping1D | √ | √ | √ |√× | × | × | -| Cropping2D | √ | √ | √ |√× | × | × | -| Cropping3D | √ | √ | √ |√× | × | × | -| Upsampling 1D | √ | √ | √ |√× | √ | × | -| Upsampling 2D | √ | √ | √ |√× | √ | × | -| Upsampling 3D | √ | √ | √ |√× | √ | × | -| ZeroPadding 1D | × | √ | √ |√× | √ | √ | -| ZeroPadding 2D | × | √ | √ |√× | √ | √ | -| ZeroPadding 3D | × | √ | √ |√× | √ | √ | -| Im2Col | √ | × | × |√× | × | × | -| Spatial Pyramid Pooling | √ | × | × |√× | × | × | +| Conv1D | √ | √ | √ | √ | √ | √ | +| Conv2D | √ | √ | √ | √ | √ | √ | +| DepthwiseConv2D | × | × | √ | × | √ | √ | +| SeperableConv1D | × | √ | √ | × | √ | × | +| SeperableConv2D | × | √ | √ | × | √ | × | +| Conv2DTranspose | √ | √ | √ | × | √ | × | +| Conv3D | √ | √ | √ | √ | √ | √ | +| Conv3DTranspose | √ | √ | √ | × | × | × | +| Deconvolution | × | √ | √ | √ | × | √ | +| Cropping1D | √ | √ | √ | √ | × | × | +| Cropping2D | √ | √ | √ | √ | × | × | +| Cropping3D | √ | √ | √ | √ | × | × | +| Upsampling 1D | √ | √ | √ | × | √ | × | +| Upsampling 2D | √ | √ | √ | × | √ | × | +| Upsampling 3D | √ | √ | √ | × | √ | × | +| ZeroPadding 1D | × | √ | √ | × | √ | √ | +| ZeroPadding 2D | × | √ | √ | × | √ | √ | +| ZeroPadding 3D | × | √ | √ | × | √ | √ | +| Im2Col | √ | × | × | × | × | × | +| Spatial Pyramid Pooling | √ | × | × | √ | × | × | * Upsampling in Caffe can be done by using methods shown [here](https://gist.github.com/tnarihi/54744612d35776f53278) ### Pooling Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| MaxPooling1D | √ | √ | √ |√× | √ | × | -| MaxPooling2D | √ | √ | √ |√× | √ | √ | -| MaxPooling3D | √ | √ | √ |√× | √ | √ | -| AveragePooling1D | √ | √ | √ |√× | √ | × | -| AveragePooling2D | √ | √ | √ |√× | √ | × | -| AveragePooling3D | √ | √ | √ |√× | √ | × | -| GlobalMaxPooling1D | × | √ | √ |√× | × | × | -| GlobalAveragePooling1D | × | √ | √ |√× | × | × | -| GlobalMaxPooling2D | × | √ | √ |√× | × | × | -| GlobalAveragePooling2D | × | √ | √ |√× | × | × | -| GlobalMaxPooling3D | × | √ | √ |√× | × | × | -| GlobalAveragePooling3D | × | √ | √ |√× | × | × | -| Stochastic Pooling | √ | × | × |√× | × | × | +| MaxPooling1D | √ | √ | √ | √ | √ | × | +| MaxPooling2D | √ | √ | √ | √ | √ | √ | +| MaxPooling3D | √ | √ | √ | √ | √ | √ | +| AveragePooling1D | √ | √ | √ | × | √ | × | +| AveragePooling2D | √ | √ | √ | × | √ | × | +| AveragePooling3D | √ | √ | √ | × | √ | × | +| GlobalMaxPooling1D | × | √ | √ | × | × | × | +| GlobalAveragePooling1D | × | √ | √ | × | × | × | +| GlobalMaxPooling2D | × | √ | √ | × | × | × | +| GlobalAveragePooling2D | × | √ | √ | × | × | × | +| GlobalMaxPooling3D | × | √ | √ | × | × | × | +| GlobalAveragePooling3D | × | √ | √ | × | × | × | +| Stochastic Pooling | √ | × | × | √ | × | × | ### Locally-connected Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| LocallyConnected1D | × | √ | √ |√× | √ | √× | -| LocallyConnected2D | × | √ | √ |√× | √ | √× | +| LocallyConnected1D | × | √ | √ | × | √ | √× | +| LocallyConnected2D | × | √ | √ | × | √ | √× | ### Recurrent Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------:| :-----------------: | -| RNN | √ | √ | √ |√× | × | × | -| SimpleRNN | × | √ | √ |√× | √ | × | -| GRU | × | √ | √ |√× | √ | × | -| LSTM | √ | √ | √ |√× | √ | × | -| ConvLSTM2D | × | √ | √ |√× | × | × | -| SimpleRNNCell | × | √ | √ |√× | × | × | -| GRUCell | × | √ | √ |√× | × | × | -| LSTMCell | × | √ | √ |√× | × | × | -| CuDDNGRU | × | √ | √ |√× | × | × | -| CuDDNLSTM | × | √ | √ |√× | × | × | -| StackedRNNCell | × | × | √ |√× | × | × | +| RNN | √ | √ | √ | √ | × | × | +| SimpleRNN | × | √ | √ | × | √ | × | +| GRU | × | √ | √ | × | √ | × | +| LSTM | √ | √ | √ | √ | √ | × | +| ConvLSTM2D | × | √ | √ | × | × | × | +| SimpleRNNCell | × | √ | √ | × | × | × | +| GRUCell | × | √ | √ | × | × | × | +| LSTMCell | × | √ | √ | × | × | × | +| CuDDNGRU | × | √ | √ | × | × | × | +| CuDDNLSTM | × | √ | √ | × | × | × | +| StackedRNNCell | × | × | √ | × | × | × | ### Embedding Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Embedding | √ | √ | √ |√× | √ | × | +| Embedding | √ | √ | √ | × | √ | × | ### Merge Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Add | × | √ | √ |√× | √ | √ | -| Subtract | × | √ | √ |√× | × | × | -| Multiply | × | √ | √ |√× | √ | √ | -| Average | × | √ | √ |√× | √ | √ | -| Minium | × | × | √ |√× | × | × | -| Maximum | × | √ | √ |√× | √ | × | -| Concatenate | √ | √ | √ |√× | √ | √ | -| Dot | × | √ | √ |√× | √ | √ | +| Add | × | √ | √ | √ | √ | √ | +| Subtract | × | √ | √ | × | × | × | +| Multiply | × | √ | √ | √ | √ | √ | +| Average | × | √ | √ | √ | √ | √ | +| Minium | × | × | √ | × | × | × | +| Maximum | × | √ | √ | √ | √ | × | +| Concatenate | √ | √ | √ | × | √ | √ | +| Dot | × | √ | √ | √ | √ | √ | ### Activations Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| ReLu | √ | √ | √ |√× | √ | √ | -| LeakyReLu | √ | √ | √ |√× | √ | √ | -| PReLU | √ | √ | √ |√× | √ | × | -| ELU | √ | √ | √ |√× | √ | √ | -| ThresholdedReLU | √ | √ | √ |√× | √ | × | -| Softmax | √ | √ | √ |√× | √ | √ | -| Argmax | √ | × | × |√× | × | × | -| Sigmoid | √ | √ | √ |√× | √ | √ | -| Hard Sigmoid | √ | √ | √ |√× | √ | × | -| TanH | √ | √ | √ |√× | √ | × | -| SELU | × | √ | √ |√× | × | √ | -| Absolute Value | √ | × | × |√× | × | × | -| Power | √ | √ | × |√× | × | × | -| Exp | √ | √ | × |√× | × | × | -| Linear | × | √ | √ |√× | × | × | -| Log | √ | √ | × |√× | × | × | -| BNLL | √ | × | × |√× | × | × | -| Bias | √ | × | × |√× | √ | √ | -| Scale | √ | × | × |√× | √ | √ | +| ReLu | √ | √ | √ | √ | √ | √ | +| LeakyReLu | √ | √ | √ | × | √ | √ | +| PReLU | √ | √ | √ | √ | √ | × | +| ELU | √ | √ | √ | √ | √ | √ | +| ThresholdedReLU | √ | √ | √ | √ | √ | × | +| Softmax | √ | √ | √ | × | √ | √ | +| Argmax | √ | × | × | √ | × | × | +| Sigmoid | √ | √ | √ | √ | √ | √ | +| Hard Sigmoid | √ | √ | √ | × | √ | × | +| TanH | √ | √ | √ | √ | √ | × | +| SELU | × | √ | √ | × | × | √ | +| Absolute Value | √ | × | × | √ | × | × | +| Power | √ | √ | × | √ | × | × | +| Exp | √ | √ | × | √ | × | × | +| Linear | × | √ | √ | × | × | × | +| Log | √ | √ | × | √ | × | × | +| BNLL | √ | × | × | √ | × | × | +| Bias | √ | × | × | √ | √ | √ | +| Scale | √ | × | × | √ | √ | √ | ### Utility Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Slicing | √ | × | × |√× | × | × | -| Eltwise | √ | × | × |√× | √ | √ | -| Parameter | √ | × | × |√× | × | √ | -| Reduction | √ | × | × |√× | × | × | -| Silence | √ | × | × |√× | × | × | +| Slicing | √ | × | × | √ | × | × | +| Eltwise | √ | × | × | √ | √ | √ | +| Parameter | √ | × | × | √ | × | √ | +| Reduction | √ | × | × | √ | × | × | +| Silence | √ | × | × | × | × | × | ### Loss Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Multinomial Logistic Loss | √ | × | × |√× | × | × | -| Infogain Loss | √ | × | × |√× | × | × | -| Softmax with Loss | √ | × | √ |√× | × | × | -| Sum-of-Squares/Euclidean | √ | × | × |√× | × | × | -| Hinge / Margin | √ | √ | √ |√× | × | × | -| Sigmoid Cross-Entropy Loss| √ | × | √ |√× | × | × | -| Accuracy / Top-k layer | √ | × | × |√× | × | × | -| Contrastive Loss | √ | × | × |√× | × | × | +| Multinomial Logistic Loss | √ | × | × | × | × | × | +| Infogain Loss | √ | × | × | √ | × | × | +| Softmax with Loss | √ | × | √ | √ | × | × | +| Sum-of-Squares/Euclidean | √ | × | × | × | × | × | +| Hinge / Margin | √ | √ | √ | √ | × | × | +| Sigmoid Cross-Entropy Loss| √ | × | √ | × | × | × | +| Accuracy / Top-k layer | √ | × | × | √ | × | × | +| Contrastive Loss | √ | × | × | √ | × | × | ### Normalization Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| BatchNormalization | √ | √ | √ |√× | √ | √ | -| MVN | √ | × | × |√× | × | × | +| BatchNormalization | √ | √ | √ | √ | √ | √ | +| MVN | √ | × | × | √ | × | × | ### Noise Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| GaussianNoise | × | √ | √ |√× | √ | × | -| GaussianDropout | √ | √ | √ |√× | √ | × | -| AlphaDropout | √ | √ | √ |√× | √ | √ | +| GaussianNoise | × | √ | √ | × | √ | × | +| GaussianDropout | √ | √ | √ | × | √ | × | +| AlphaDropout | √ | √ | √ | × | √ | √ | ### Layer Wrappers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| TimeDistributed | × | √ | √ |√× | √ | × | -| Bidirectional | × | √ | √ |√× | √ | × | +| TimeDistributed | × | √ | √ | × | √ | × | +| Bidirectional | × | √ | √ | × | √ | × | ### Custom Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------: | :----------: | :---------: | :---------: | :-----------: | :--------------: | :-----------------: | | | √ | √ | Use Keras API for custom layers | NA | NA | NA| -| LRN | √ | √ | √ |√× | √ | √ | +| LRN | √ | √ | √ | √ | √ | √ | ## Additional Notes: * Keras does not support the LRN layer used in Alexnet & many other models. To use the LRN layer refer to [here](https://github.com/Cloud-CV/Fabrik/blob/master/tutorials/keras_custom_layer_usage.md.) From fc822987b1a28cee5c721449703e264f49604f9e Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Sun, 4 Nov 2018 23:25:27 -0500 Subject: [PATCH 10/12] Added Data Layers --- docs/source/supported_layers.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index 745e49e28..c81a81d81 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -62,6 +62,19 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf | GlobalAveragePooling3D | × | √ | √ | × | × | × | | Stochastic Pooling | √ | × | × | √ | × | × | + ### Data Layers +| Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | +| Image Data | √ | √ | √ | √ | √ | √ | +| Data | √ | √ | √ | √ | √ | √ | +| HDF5 Data | √ | √ | √ | √ | √ | √ | +| HDF5 Output Data | √ | √ | √ | √ | √ | √ | +| Input | √ | √ | √ | √ | √ | √ | +| Window Data | √ | × | × | √ | × | × | +| Memory Data | √ | √ | √ | √ | √ | √ | +| Dummy Data | √ | × | × | √ | × | × | +| Python | √ | √ | √ | √ | √ | √ | + ### Locally-connected Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | From 9a6267a9ab9bc31814fddf593793b80a572fe264 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Mon, 5 Nov 2018 09:12:23 -0500 Subject: [PATCH 11/12] Fixed Errors Not complete --- docs/source/supported_layers.md | 60 ++++++++++++++++----------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index c81a81d81..a3cf38976 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -7,20 +7,20 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorflow: ### Core Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Activation | √ | √ | √ | √ | √ | √ | -| ActivityRegularization | √ | √ | √ | × | √ | × | -| Dense | × | √ | √ | × | √ | × | -| Dropout | √ | √ | √ | √ | √ | √ | -| Flatten | √ | √ | √ | √ | √ | × | -| Lambda | × | √ | √ | × | × | × | -| Masking | √ | √ | √ | × | √ | × | -| Permute | × | √ | √ | × | √ | × | -| Repeat Vector | × | √ | √ | × | √ | √ | -| Reshape | √ | √ | √ | √ | √ | × | -| Spatial Dropout 1D | × | √ | √ | × | × | × | -| Spatial Dropout 2D | × | √ | √ | × | × | × | -| Spatial Dropout 3D | × | √ | √ | × | × | × | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :---------------: | +| Activation | × | √ | √ | × | √ | √ | +| ActivityRegularization | × | √ | √ | × | √ | × | +| Inner Product | √ | √ (Dense | √ | √ | √ | × | +| Dropout | √ | √ | √ | √ | √ | √ | +| Flatten | √ | √ | √ | √ | √ | × | +| Lambda | × | √ | √ | × | × | × | +| Masking | √ | √ | √ | × | √ | × | +| Permute | × | √ | √ | × | √ | × | +| Repeat Vector | × | √ | √ | × | √ | √ | +| Reshape | √ | √ | √ | √ | √ | × | +| Spatial Dropout 1D | × | √ | √ | × | × | × | +| Spatial Dropout 2D | × | √ | √ | × | × | × | +| Spatial Dropout 3D | × | √ | √ | × | × | × | #### Convolutional Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | @@ -65,15 +65,15 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf ### Data Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Image Data | √ | √ | √ | √ | √ | √ | -| Data | √ | √ | √ | √ | √ | √ | -| HDF5 Data | √ | √ | √ | √ | √ | √ | -| HDF5 Output Data | √ | √ | √ | √ | √ | √ | +| Image Data | √ | √ | √ | √ | × | × | +| Data | √ | √ | √ | √ | × | × | +| HDF5 Data | √ | √ | √ | √ | × | × | +| HDF5 Output Data | √ | √ | √ | √ | × | × | | Input | √ | √ | √ | √ | √ | √ | | Window Data | √ | × | × | √ | × | × | -| Memory Data | √ | √ | √ | √ | √ | √ | +| Memory Data | √ | √ | √ | √ | × | × | | Dummy Data | √ | × | × | √ | × | × | -| Python | √ | √ | √ | √ | √ | √ | +| Python | √ | √ | √ | √ | × | × | ### Locally-connected Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | @@ -130,24 +130,24 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf | Log | √ | √ | × | √ | × | × | | BNLL | √ | × | × | √ | × | × | | Bias | √ | × | × | √ | √ | √ | -| Scale | √ | × | × | √ | √ | √ | +| Scale | √ | × | × | √ | × | × | ### Utility Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | -| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Slicing | √ | × | × | √ | × | × | -| Eltwise | √ | × | × | √ | √ | √ | -| Parameter | √ | × | × | √ | × | √ | -| Reduction | √ | × | × | √ | × | × | -| Silence | √ | × | × | × | × | × | +| :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :---------------: | +| Slicing | √ | × | × | √ | × | × | +| Eltwise | √ | × | × | √ | √ | √ | +| Parameter | √ | × | × | √ | × | √ | +| Reduction | √ | × | × | √ | × | × | +| Silence | √ | × | × | √ | × | × | ### Loss Layers | Layer | Caffe | Keras | Tensorflow | Fabrik Caffe | Fabrik Keras | Fabrik Tensorflow | | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :-----------------: | -| Multinomial Logistic Loss | √ | × | × | × | × | × | +| Multinomial Logistic Loss | √ | × | × | √ | × | × | | Infogain Loss | √ | × | × | √ | × | × | | Softmax with Loss | √ | × | √ | √ | × | × | -| Sum-of-Squares/Euclidean | √ | × | × | × | × | × | +| Sum-of-Squares/Euclidean | √ | × | × | √ | × | × | | Hinge / Margin | √ | √ | √ | √ | × | × | -| Sigmoid Cross-Entropy Loss| √ | × | √ | × | × | × | +| Sigmoid Cross-Entropy Loss| √ | × | √ | √ | × | × | | Accuracy / Top-k layer | √ | × | × | √ | × | × | | Contrastive Loss | √ | × | × | √ | × | × | ### Normalization Layers From a67bbe3e8b774002a0f27b284d98da9a251c30a8 Mon Sep 17 00:00:00 2001 From: Adam Zamlynny Date: Mon, 5 Nov 2018 09:12:39 -0500 Subject: [PATCH 12/12] Typo --- docs/source/supported_layers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/supported_layers.md b/docs/source/supported_layers.md index a3cf38976..c3189e601 100644 --- a/docs/source/supported_layers.md +++ b/docs/source/supported_layers.md @@ -10,7 +10,7 @@ Below are tables showing which layers are supported by Caffe, Keras, and Tensorf | :-----------------------: | :-----------: | :----------: | :---------: | :-----------: | :--------------: | :---------------: | | Activation | × | √ | √ | × | √ | √ | | ActivityRegularization | × | √ | √ | × | √ | × | -| Inner Product | √ | √ (Dense | √ | √ | √ | × | +| Inner Product | √ | √ (Dense) | √ | √ | √ | × | | Dropout | √ | √ | √ | √ | √ | √ | | Flatten | √ | √ | √ | √ | √ | × | | Lambda | × | √ | √ | × | × | × |