From e987fdd462c99e26cfd7aa2c59f006d91ea44b94 Mon Sep 17 00:00:00 2001 From: Lai Wei Date: Sun, 9 Dec 2018 20:50:32 -0800 Subject: [PATCH 1/5] update hybridize documentation --- docs/tutorials/gluon/hybrid.md | 90 ++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/docs/tutorials/gluon/hybrid.md b/docs/tutorials/gluon/hybrid.md index 6d64acdce275..d4f5843444aa 100644 --- a/docs/tutorials/gluon/hybrid.md +++ b/docs/tutorials/gluon/hybrid.md @@ -137,4 +137,94 @@ to gluon with `SymbolBlock`: net2 = gluon.SymbolBlock.imports('model-symbol.json', ['data'], 'model-0001.params') ``` +## Operators that does not work with hybridize + +While most APIs are the same in NDArray and Symbol, there are some differences. Writting `F.operator` and call `hybridize` may not work all the time. +Here we list all the APIs that do not work and provide you the work arounds. + +### Element-wise Operators + +The following arithmetic and comparison APIs are automatically broadcasted if the input NDArrays have different shapes. +However, that's not the case in Symbol API, it's not automatically broadcasted and you have to manually specify whether to use element-wise operator or broadcast operators. + + +| NDArray APIs | Description | +|---|---| +| [*NDArray.__add__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__add__) | x.__add__(y) <=> x+y <=> mx.nd.add(x, y) | +| [*NDArray.__sub__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__sub__) | x.__sub__(y) <=> x-y <=> mx.nd.subtract(x, y) | +| [*NDArray.__mul__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mul__) | x.__mul__(y) <=> x*y <=> mx.nd.multiply(x, y) | +| [*NDArray.__div__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__div__) | x.__div__(y) <=> x/y <=> mx.nd.divide(x, y) | +| [*NDArray.__mod__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mod__) | x.__mod__(y) <=> x%y <=> mx.nd.modulo(x, y) | +| [*NDArray.__lt__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__lt__) | x.__lt__(y) <=> x mx.nd.lesser(x, y) | +| [*NDArray.__le__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__le__) | x.__le__(y) <=> x<=y <=> mx.nd.less_equal(x, y) | +| [*NDArray.__gt__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__gt__) | x.__gt__(y) <=> x>y <=> mx.nd.greater(x, y) | +| [*NDArray.__ge__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ge__) | x.__ge__(y) <=> x>=y <=> mx.nd.greater_equal(x, y)| +| [*NDArray.__eq__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__eq__) | x.__eq__(y) <=> x==y <=> mx.nd.equal(x, y) | +| [*NDArray.__ne__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ne__) | x.__ne__(y) <=> x!=y <=> mx.nd.not_equal(x, y) | + +The current workaround is to use corecponding broadcast operators for arithmetic and comparison to avoid potential hybridization failure when input shapes are different. + +| Symbol APIs | Description | +|---|---| +|[*broadcast_add*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_add) | Returns element-wise sum of the input arrays with broadcasting. | +|[*broadcast_sub*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_sub) | Returns element-wise difference of the input arrays with broadcasting. | +|[*broadcast_mul*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_mul) | Returns element-wise product of the input arrays with broadcasting. | +|[*broadcast_div*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_div) | Returns element-wise division of the input arrays with broadcasting. | +|[*broadcast_mod*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_mod) | Returns element-wise modulo of the input arrays with broadcasting. | +|[*broadcast_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_equal) | Returns the result of element-wise *equal to* (==) comparison operation with broadcasting. | +|[*broadcast_not_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_not_equal) | Returns the result of element-wise *not equal to* (!=) comparison operation with broadcasting. | +|[*broadcast_greater*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_greater) | Returns the result of element-wise *greater than* (>) comparison operation with broadcasting. | +|[*broadcast_greater_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_greater_equal) | Returns the result of element-wise *greater than or equal to* (>=) comparison operation with broadcasting. | +|[*broadcast_lesser*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_lesser) | Returns the result of element-wise *lesser than* (<) comparison operation with broadcasting. | +|[*broadcast_lesser_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_lesser_equal) | Returns the result of element-wise *lesser than or equal to* (<=) comparison operation with broadcasting. | + +For example, if you wan to add a NDarray to your input x, use `broadcast_add` instead of `+`: + +```python +def hybrid_forward(self, F, x): + # avoid writting: return x + F.ones((1, 1)) + return F.broadcast_add(x, F.ones((1, 1))) +``` + +### Shape + +Gluon's imperative interface is very flexible and allows you to print shape of the NDArray. However, Symbol does not have shape attributes. As a result, you need to avoid printing shapes in `hybrid_forward`. +Otherwise, you will get the following error: +```bash +AttributeError: 'Symbol' object has no attribute 'shape' +``` + +### Slice +[] in NDArray is to get a slice from the array. [] in Symbol is to get an output from a grouped symbol. +For example, you will get different result for the following method before and after hybridization. + +```python +def hybrid_forward(self, F, x): + return x[0] +``` + +The current workaround is explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) operators in `hybrid_forwar`d. + + +### Not implemented operators + +Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure + +#### Array API + +mx.nd.array() is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols. + +#### In-Place Arithmetic Operators + +In place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`. +For example, avoid `x += y` and use `x = x + y`, other wise you will get `NotImplementedError`. + +| NDArray in-place arithmetic operators | Description | +|---|---| +|[*NDArray.__iadd__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__iadd__) | x.__iadd__(y) <=> x+=y | +|[*NDArray.__isub__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__isub__) | x.__isub__(y) <=> x-=y | +|[*NDArray.__imul__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imul__) | x.__imul__(y) <=> x*=y | +|[*NDArray.__idiv__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__idiv__) | x.__rdiv__(y) <=> x/=y | +|[*NDArray.__imod__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imod__) | x.__rmod__(y) <=> x%=y | + From ee7df70a0e64e51d233c9f043b927b64805908bd Mon Sep 17 00:00:00 2001 From: Lai Wei Date: Mon, 17 Dec 2018 16:10:27 -0800 Subject: [PATCH 2/5] address review comments --- docs/tutorials/gluon/hybrid.md | 38 +++++++++++++++++----------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/tutorials/gluon/hybrid.md b/docs/tutorials/gluon/hybrid.md index d4f5843444aa..1684b1cd59a0 100644 --- a/docs/tutorials/gluon/hybrid.md +++ b/docs/tutorials/gluon/hybrid.md @@ -139,7 +139,7 @@ net2 = gluon.SymbolBlock.imports('model-symbol.json', ['data'], 'model-0001.para ## Operators that does not work with hybridize -While most APIs are the same in NDArray and Symbol, there are some differences. Writting `F.operator` and call `hybridize` may not work all the time. +If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function, F will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all the time. Here we list all the APIs that do not work and provide you the work arounds. ### Element-wise Operators @@ -150,17 +150,17 @@ However, that's not the case in Symbol API, it's not automatically broadcasted a | NDArray APIs | Description | |---|---| -| [*NDArray.__add__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__add__) | x.__add__(y) <=> x+y <=> mx.nd.add(x, y) | -| [*NDArray.__sub__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__sub__) | x.__sub__(y) <=> x-y <=> mx.nd.subtract(x, y) | -| [*NDArray.__mul__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mul__) | x.__mul__(y) <=> x*y <=> mx.nd.multiply(x, y) | -| [*NDArray.__div__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__div__) | x.__div__(y) <=> x/y <=> mx.nd.divide(x, y) | -| [*NDArray.__mod__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mod__) | x.__mod__(y) <=> x%y <=> mx.nd.modulo(x, y) | -| [*NDArray.__lt__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__lt__) | x.__lt__(y) <=> x mx.nd.lesser(x, y) | -| [*NDArray.__le__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__le__) | x.__le__(y) <=> x<=y <=> mx.nd.less_equal(x, y) | -| [*NDArray.__gt__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__gt__) | x.__gt__(y) <=> x>y <=> mx.nd.greater(x, y) | -| [*NDArray.__ge__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ge__) | x.__ge__(y) <=> x>=y <=> mx.nd.greater_equal(x, y)| -| [*NDArray.__eq__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__eq__) | x.__eq__(y) <=> x==y <=> mx.nd.equal(x, y) | -| [*NDArray.__ne__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ne__) | x.__ne__(y) <=> x!=y <=> mx.nd.not_equal(x, y) | +| [*NDArray.\__add\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__add__) | x.\__add\__(y) <=> x+y <=> mx.nd.add(x, y) | +| [*NDArray.\__sub\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__sub__) | x.\__sub\__(y) <=> x-y <=> mx.nd.subtract(x, y) | +| [*NDArray.\__mul\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mul__) | x.\__mul\__(y) <=> x*y <=> mx.nd.multiply(x, y) | +| [*NDArray.\__div\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__div__) | x.\__div\__(y) <=> x/y <=> mx.nd.divide(x, y) | +| [*NDArray.\__mod\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mod__) | x.\__mod\__(y) <=> x%y <=> mx.nd.modulo(x, y) | +| [*NDArray.\__lt\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__lt__) | x.\__lt\__(y) <=> x mx.nd.lesser(x, y) | +| [*NDArray.\__le\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__le__) | x.\__le\__(y) <=> x<=y <=> mx.nd.less_equal(x, y) | +| [*NDArray.\__gt\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__gt__) | x.\__gt\__(y) <=> x>y <=> mx.nd.greater(x, y) | +| [*NDArray.\__ge\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ge__) | x.\__ge\__(y) <=> x>=y <=> mx.nd.greater_equal(x, y)| +| [*NDArray.\__eq\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__eq__) | x.\__eq\__(y) <=> x==y <=> mx.nd.equal(x, y) | +| [*NDArray.\__ne\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ne__) | x.\__ne\__(y) <=> x!=y <=> mx.nd.not_equal(x, y) | The current workaround is to use corecponding broadcast operators for arithmetic and comparison to avoid potential hybridization failure when input shapes are different. @@ -178,11 +178,11 @@ The current workaround is to use corecponding broadcast operators for arithmetic |[*broadcast_lesser*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_lesser) | Returns the result of element-wise *lesser than* (<) comparison operation with broadcasting. | |[*broadcast_lesser_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_lesser_equal) | Returns the result of element-wise *lesser than or equal to* (<=) comparison operation with broadcasting. | -For example, if you wan to add a NDarray to your input x, use `broadcast_add` instead of `+`: +For example, if you want to add a NDarray to your input x, use `broadcast_add` instead of `+`: ```python def hybrid_forward(self, F, x): - # avoid writting: return x + F.ones((1, 1)) + # avoid writing: return x + F.ones((1, 1)) return F.broadcast_add(x, F.ones((1, 1))) ``` @@ -195,7 +195,7 @@ AttributeError: 'Symbol' object has no attribute 'shape' ``` ### Slice -[] in NDArray is to get a slice from the array. [] in Symbol is to get an output from a grouped symbol. +`[]` in NDArray is used to get a slice from the array. However, `[]` in Symbol is used to get an output from a grouped symbol. For example, you will get different result for the following method before and after hybridization. ```python @@ -203,7 +203,7 @@ def hybrid_forward(self, F, x): return x[0] ``` -The current workaround is explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) operators in `hybrid_forwar`d. +The current workaround is to explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) operators in `hybrid_forward`. ### Not implemented operators @@ -212,12 +212,12 @@ Some of the often used operators in NDArray are not implemented in Symbol, and #### Array API -mx.nd.array() is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols. +`mx.nd.array()` is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols. #### In-Place Arithmetic Operators -In place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`. -For example, avoid `x += y` and use `x = x + y`, other wise you will get `NotImplementedError`. +In-place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`. +For example, avoid `x += y` and use `x = x + y`, otherwise you will get `NotImplementedError`. | NDArray in-place arithmetic operators | Description | |---|---| From ea444376579a4dff1a9468745f9caab9fe41c0a1 Mon Sep 17 00:00:00 2001 From: Lai Wei Date: Tue, 18 Dec 2018 10:00:08 -0800 Subject: [PATCH 3/5] improve doc --- docs/tutorials/gluon/hybrid.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/tutorials/gluon/hybrid.md b/docs/tutorials/gluon/hybrid.md index 1684b1cd59a0..6f22aada66e1 100644 --- a/docs/tutorials/gluon/hybrid.md +++ b/docs/tutorials/gluon/hybrid.md @@ -140,7 +140,7 @@ net2 = gluon.SymbolBlock.imports('model-symbol.json', ['data'], 'model-0001.para ## Operators that does not work with hybridize If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function, F will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all the time. -Here we list all the APIs that do not work and provide you the work arounds. +Here we list some frequently used NDArray APIs that can't be hybridized and provide you the work arounds. ### Element-wise Operators @@ -162,7 +162,7 @@ However, that's not the case in Symbol API, it's not automatically broadcasted a | [*NDArray.\__eq\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__eq__) | x.\__eq\__(y) <=> x==y <=> mx.nd.equal(x, y) | | [*NDArray.\__ne\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ne__) | x.\__ne\__(y) <=> x!=y <=> mx.nd.not_equal(x, y) | -The current workaround is to use corecponding broadcast operators for arithmetic and comparison to avoid potential hybridization failure when input shapes are different. +The current workaround is to use corresponding broadcast operators for arithmetic and comparison to avoid potential hybridization failure when input shapes are different. | Symbol APIs | Description | |---|---| @@ -203,7 +203,7 @@ def hybrid_forward(self, F, x): return x[0] ``` -The current workaround is to explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) operators in `hybrid_forward`. +The current workaround is to explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) or [`slice_axis`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice_axis) operators in `hybrid_forward`. ### Not implemented operators @@ -221,10 +221,10 @@ For example, avoid `x += y` and use `x = x + y`, otherwise you will get `NotImp | NDArray in-place arithmetic operators | Description | |---|---| -|[*NDArray.__iadd__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__iadd__) | x.__iadd__(y) <=> x+=y | -|[*NDArray.__isub__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__isub__) | x.__isub__(y) <=> x-=y | -|[*NDArray.__imul__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imul__) | x.__imul__(y) <=> x*=y | -|[*NDArray.__idiv__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__idiv__) | x.__rdiv__(y) <=> x/=y | -|[*NDArray.__imod__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imod__) | x.__rmod__(y) <=> x%=y | +|[*NDArray.\__iadd\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__iadd__) | x.__iadd__(y) <=> x+=y | +|[*NDArray.\__isub\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__isub__) | x.__isub__(y) <=> x-=y | +|[*NDArray.\__imul\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imul__) | x.__imul__(y) <=> x*=y | +|[*NDArray.\__idiv\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__idiv__) | x.__rdiv__(y) <=> x/=y | +|[*NDArray.\__imod\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imod__) | x.__rmod__(y) <=> x%=y | From 33cd247264afe3082f745bf615733f17c1b29246 Mon Sep 17 00:00:00 2001 From: Lai Wei Date: Wed, 19 Dec 2018 12:08:14 -0800 Subject: [PATCH 4/5] address comments --- docs/tutorials/gluon/hybrid.md | 55 +++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 20 deletions(-) diff --git a/docs/tutorials/gluon/hybrid.md b/docs/tutorials/gluon/hybrid.md index 6f22aada66e1..770d04c8f1fb 100644 --- a/docs/tutorials/gluon/hybrid.md +++ b/docs/tutorials/gluon/hybrid.md @@ -1,6 +1,9 @@ # Hybrid - Faster training and easy deployment -*Note: a newer version is available [here](http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html).* +*Related Contents:* +* [Fast, portable neural networks with Gluon HybridBlocks](https://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html) +* [A Hybrid of Imperative and Symbolic Programming +](http://en.diveintodeeplearning.org/chapter_computational-performance/hybridize.html) Deep learning frameworks can be roughly divided into two categories: declarative and imperative. With declarative frameworks (including Tensorflow, Theano, etc) @@ -137,15 +140,16 @@ to gluon with `SymbolBlock`: net2 = gluon.SymbolBlock.imports('model-symbol.json', ['data'], 'model-0001.params') ``` -## Operators that does not work with hybridize +## Operators that do not work with hybridize -If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function, F will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all the time. -Here we list some frequently used NDArray APIs that can't be hybridized and provide you the work arounds. +If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function. +`F` will be `mxnet.nd` before you hybridize and `mxnet.sym` after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all of the time. +Here we list some frequently used NDArray APIs that can't be hybridized and provide you the work arounds. ### Element-wise Operators -The following arithmetic and comparison APIs are automatically broadcasted if the input NDArrays have different shapes. -However, that's not the case in Symbol API, it's not automatically broadcasted and you have to manually specify whether to use element-wise operator or broadcast operators. +In NDArray APIs, the following arithmetic and comparison APIs are automatically broadcasted if the input NDArrays have different shapes. +However, that's not the case in Symbol API. It's not automatically broadcasted, and you have to manually specify to use another set of broadcast operators for Symbols expected to have different shapes. | NDArray APIs | Description | @@ -155,7 +159,7 @@ However, that's not the case in Symbol API, it's not automatically broadcasted a | [*NDArray.\__mul\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mul__) | x.\__mul\__(y) <=> x*y <=> mx.nd.multiply(x, y) | | [*NDArray.\__div\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__div__) | x.\__div\__(y) <=> x/y <=> mx.nd.divide(x, y) | | [*NDArray.\__mod\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mod__) | x.\__mod\__(y) <=> x%y <=> mx.nd.modulo(x, y) | -| [*NDArray.\__lt\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__lt__) | x.\__lt\__(y) <=> x mx.nd.lesser(x, y) | +| [*NDArray.\__lt\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__lt__) | x.\__lt\__(y) <=> x x mx.nd.lesser(x, y) | | [*NDArray.\__le\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__le__) | x.\__le\__(y) <=> x<=y <=> mx.nd.less_equal(x, y) | | [*NDArray.\__gt\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__gt__) | x.\__gt\__(y) <=> x>y <=> mx.nd.greater(x, y) | | [*NDArray.\__ge\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ge__) | x.\__ge\__(y) <=> x>=y <=> mx.nd.greater_equal(x, y)| @@ -186,9 +190,11 @@ def hybrid_forward(self, F, x): return F.broadcast_add(x, F.ones((1, 1))) ``` +If you used `+`, it would still work before hybridization, but will throw an error of shape missmtach after hybridization. + ### Shape -Gluon's imperative interface is very flexible and allows you to print shape of the NDArray. However, Symbol does not have shape attributes. As a result, you need to avoid printing shapes in `hybrid_forward`. +Gluon's imperative interface is very flexible and allows you to print the shape of the NDArray. However, Symbol does not have shape attributes. As a result, you need to avoid printing shapes in `hybrid_forward`. Otherwise, you will get the following error: ```bash AttributeError: 'Symbol' object has no attribute 'shape' @@ -196,7 +202,7 @@ AttributeError: 'Symbol' object has no attribute 'shape' ### Slice `[]` in NDArray is used to get a slice from the array. However, `[]` in Symbol is used to get an output from a grouped symbol. -For example, you will get different result for the following method before and after hybridization. +For example, you will get different results for the following method before and after hybridization. ```python def hybrid_forward(self, F, x): @@ -208,23 +214,32 @@ The current workaround is to explicitly call [`slice`](https://mxnet.incubator.a ### Not implemented operators -Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure +Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure + +#### NDArray.asnumpy +Symbol does not support `asnumpy` function, you need to avoid calling `asnumpy` in `hybrid_forward`. -#### Array API +#### Array creation APIs -`mx.nd.array()` is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols. +`mx.nd.array()` is used a lot, but Symbol does not have the `array` API. The current workaround is to use `F.ones`, `F.zeros`, or `F.full`, which exist in both the NDArray and Symbol APIs. #### In-Place Arithmetic Operators -In-place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`. -For example, avoid `x += y` and use `x = x + y`, otherwise you will get `NotImplementedError`. +In-place arithmetic operators may be used in Gluon imperative mode, however if you expect to hybridize, you should write these operations explicitly instead. +For example, avoid writing `x += y` and use `x = x + y`, otherwise you will get `NotImplementedError`. This applies to all the following operators: | NDArray in-place arithmetic operators | Description | |---|---| -|[*NDArray.\__iadd\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__iadd__) | x.__iadd__(y) <=> x+=y | -|[*NDArray.\__isub\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__isub__) | x.__isub__(y) <=> x-=y | -|[*NDArray.\__imul\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imul__) | x.__imul__(y) <=> x*=y | -|[*NDArray.\__idiv\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__idiv__) | x.__rdiv__(y) <=> x/=y | -|[*NDArray.\__imod\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imod__) | x.__rmod__(y) <=> x%=y | +|[*NDArray.\__iadd\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__iadd__) | x.\__iadd\__(y) <=> x+=y | +|[*NDArray.\__isub\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__isub__) | x.\__isub\__(y) <=> x-=y | +|[*NDArray.\__imul\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imul__) | x.\__imul\__(y) <=> x*=y | +|[*NDArray.\__idiv\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__idiv__) | x.\__rdiv\__(y) <=> x/=y | +|[*NDArray.\__imod\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imod__) | x.\__rmod\__(y) <=> x%=y | + + + +## Summary + +The recommended practice is to utilize the flexibility of imperative NDArray API during experimentation. Once you finalized your model, make necessary changes mentioned above so you can call `hybridize` function to improve performance. - + \ No newline at end of file From d2c8d0b9f9031360553cceee6280810c7ee3ebd8 Mon Sep 17 00:00:00 2001 From: Lai Wei Date: Fri, 21 Dec 2018 15:02:36 -0800 Subject: [PATCH 5/5] address comments --- docs/tutorials/gluon/hybrid.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tutorials/gluon/hybrid.md b/docs/tutorials/gluon/hybrid.md index 770d04c8f1fb..f11622bd6fd1 100644 --- a/docs/tutorials/gluon/hybrid.md +++ b/docs/tutorials/gluon/hybrid.md @@ -1,6 +1,6 @@ # Hybrid - Faster training and easy deployment -*Related Contents:* +*Related Content:* * [Fast, portable neural networks with Gluon HybridBlocks](https://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html) * [A Hybrid of Imperative and Symbolic Programming ](http://en.diveintodeeplearning.org/chapter_computational-performance/hybridize.html) @@ -214,10 +214,10 @@ The current workaround is to explicitly call [`slice`](https://mxnet.incubator.a ### Not implemented operators -Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure +Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure. #### NDArray.asnumpy -Symbol does not support `asnumpy` function, you need to avoid calling `asnumpy` in `hybrid_forward`. +Symbol does not support the `asnumpy` function. You need to avoid calling `asnumpy` in `hybrid_forward`. #### Array creation APIs