fix bug in nag optimizer #13683

solin319 · 2018-12-19T03:53:20Z

grad += wd * weight
mom[:] += grad
grad[:] += self.momentum * mom
weight[:] += -lr * grad

This will minus wd*weight twice, but in formula

state = momentum * state + grad + wd * weight 
weight = weight - (lr * (grad + momentum * state))

only minus once.

``` grad += wd * weight mom[:] += grad grad[:] += self.momentum * mom weight[:] += -lr * grad ``` This will minus wd*weight twice, but in`state = momentum * state + grad + wd * weight weight = weight - (lr * (grad + momentum * state)) ` only minus once.

fix bug in nag test

anirudhacharya · 2018-12-19T18:55:48Z

python/mxnet/optimizer/optimizer.py

@@ -974,8 +974,7 @@ def update(self, index, weight, grad, state):
        if state is not None:
            mom = state
            mom[:] *= self.momentum
-            grad += wd * weight
-            mom[:] += grad
+            mom[:] += grad + wd * weight
            grad[:] += self.momentum * mom
            weight[:] += -lr * grad


can you make this weight[:] -= lr * grad it is more clear this way

anirudhacharya · 2018-12-19T19:13:31Z

python/mxnet/optimizer/optimizer.py

@@ -974,8 +974,7 @@ def update(self, index, weight, grad, state):
        if state is not None:
            mom = state
            mom[:] *= self.momentum
-            grad += wd * weight
-            mom[:] += grad
+            mom[:] += grad + wd * weight


can you replace with mom[:] = (self.momentum * mom[:]) + grad + wd * weight
and delete line 976. It will be more readable

Roshrini · 2018-12-19T19:27:59Z

@mxnet-label-bot Add [pr-awaiting-response]

solin319 · 2018-12-20T02:14:55Z

We have rewrited nag followed by anirudhacharya's suggests.

solin319 · 2018-12-21T00:51:15Z

@szha @Roshrini

Roshrini · 2019-01-02T22:06:14Z

@anirudhacharya Can you take a look again?
@sandeep-krishnamurthy For review/merge

anirudhacharya · 2019-01-14T22:34:24Z

LGTM

@mxnet-label-bot update [pr-awaiting-merge]

szha · 2019-01-15T03:28:37Z

tests/python/unittest/test_optimizer.py

-              mom[:] *= self.momentum
-              grad += wd * weight
-              mom[:] += grad
+              mom[:] = self.momentum * mom[:] + grad + wd * weight


try doing all these with in-place operators.

szha · 2019-01-15T03:28:42Z

tests/python/unittest/test_optimizer.py

-                mom[:] *= self.momentum
-                grad32 += wd * weight32
-                mom[:] += grad32
+                mom[:] = self.momentum * mom[:] + grad32 + wd * weight32


try doing all these with in-place operators.

solin319 · 2019-01-15T04:33:38Z

Which operator in in-place operator? @szha
mx.nd.add ?

szha · 2019-01-15T04:42:11Z

Currently the rhs will result in allocating temporary space for the results of self.momentum * mom[:], self.momentum * mom[:] + grad32, and self.momentum * mom[:] + grad32 + wd * weight32. They were written in the previous way to avoid such memory spikes.

solin319 · 2019-01-15T06:21:03Z

Shall we change the code back to

mom[:] *= self.momentum
grad32 += wd * weight32
mom[:] += grad32

stu1130 · 2019-01-15T23:18:18Z

@mxnet-label-bot update [pr-work-in-progress]

solin319 · 2019-01-16T14:18:31Z

Code has changed with in-place operators.

* fix bug in nag optimizer ``` grad += wd * weight mom[:] += grad grad[:] += self.momentum * mom weight[:] += -lr * grad ``` This will minus wd*weight twice, but in`state = momentum * state + grad + wd * weight weight = weight - (lr * (grad + momentum * state)) ` only minus once. * fix bug in nag test fix bug in nag test * rewrite nag test * rewrite nag * fix nag with in-place operations * fix nag with in-place operations

fix bug in nag optimizer

1a065e8

``` grad += wd * weight mom[:] += grad grad[:] += self.momentum * mom weight[:] += -lr * grad ``` This will minus wd*weight twice, but in`state = momentum * state + grad + wd * weight weight = weight - (lr * (grad + momentum * state)) ` only minus once.

solin319 requested a review from szha as a code owner December 19, 2018 03:53

fix bug in nag test

641bf6c

fix bug in nag test

anirudhacharya reviewed Dec 19, 2018

View reviewed changes

marcoabreu added the pr-awaiting-response PR is reviewed and waiting for contributor to respond label Dec 19, 2018

solin319 added 2 commits December 20, 2018 10:09

rewrite nag test

ea63cb1

rewrite nag

7633a1d

Roshrini approved these changes Jan 2, 2019

View reviewed changes

marcoabreu added pr-awaiting-merge Review and CI is complete. Ready to Merge and removed pr-awaiting-response PR is reviewed and waiting for contributor to respond labels Jan 14, 2019

szha reviewed Jan 15, 2019

View reviewed changes

marcoabreu added pr-work-in-progress PR is still work in progress and removed pr-awaiting-merge Review and CI is complete. Ready to Merge labels Jan 15, 2019

fix nag with in-place operations

510dd33

solin319 requested a review from eric-haibin-lin as a code owner January 16, 2019 14:12

fix nag with in-place operations

0fa1466

szha merged commit 9314689 into apache:master Jan 16, 2019

anirudhacharya mentioned this pull request Feb 15, 2019

Use In-place operator to prevent memory spikes in optimizer updates #13960

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix bug in nag optimizer #13683

fix bug in nag optimizer #13683

solin319 commented Dec 19, 2018

anirudhacharya Dec 19, 2018

anirudhacharya Dec 19, 2018

Roshrini commented Dec 19, 2018

solin319 commented Dec 20, 2018

solin319 commented Dec 21, 2018

Roshrini commented Jan 2, 2019

anirudhacharya commented Jan 14, 2019

szha Jan 15, 2019

szha Jan 15, 2019

solin319 commented Jan 15, 2019 •

edited

Loading

szha commented Jan 15, 2019

solin319 commented Jan 15, 2019 •

edited

Loading

stu1130 commented Jan 15, 2019

solin319 commented Jan 16, 2019

fix bug in nag optimizer #13683

fix bug in nag optimizer #13683

Conversation

solin319 commented Dec 19, 2018

anirudhacharya Dec 19, 2018

Choose a reason for hiding this comment

anirudhacharya Dec 19, 2018

Choose a reason for hiding this comment

Roshrini commented Dec 19, 2018

solin319 commented Dec 20, 2018

solin319 commented Dec 21, 2018

Roshrini commented Jan 2, 2019

anirudhacharya commented Jan 14, 2019

szha Jan 15, 2019

Choose a reason for hiding this comment

szha Jan 15, 2019

Choose a reason for hiding this comment

solin319 commented Jan 15, 2019 • edited Loading

szha commented Jan 15, 2019

solin319 commented Jan 15, 2019 • edited Loading

stu1130 commented Jan 15, 2019

solin319 commented Jan 16, 2019

solin319 commented Jan 15, 2019 •

edited

Loading

solin319 commented Jan 15, 2019 •

edited

Loading