This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Adam Optimizer Memory Leak in Scala #14080
Comments
Hey, this is the MXNet Label Bot. |
@satyakrishnagorti Here's a documentation on how to manage memory in MXNet Scala : The first method in this document details using : |
Looks like the Adam Optimizer needs to be updated to use ResourceScope. Should probably take a look at the other optimizers at the same time. Thanks for reporting this @satyakrishnagorti. @mxnet-label-bot add [Scala, bug] |
7 tasks
nswamy
pushed a commit
that referenced
this issue
Mar 28, 2019
* Fixes for memory leak when reshaping executor * Fixed Adam Optimizer memory leak * Cleanup for PR * Added unit test for new ResourceScope method * Removing import that was added by overzealous ide * Add back in an import * Added flags for executor to know whether or not it owns NDArrays for disposal * Moving to ResourceScope.using implementation * Changes to make ResourceScope.using work with existing scope * Updating ResourceScope to work with existing scopes via usingIfScopeExists method * Fix clojure unit tests * Fixes to be compatibile with how clojure is using ResourceScope * Removing some unnecessary changes * Adding scope assertion in unit test
vdantu
pushed a commit
to vdantu/incubator-mxnet
that referenced
this issue
Mar 31, 2019
…e#14372) * Fixes for memory leak when reshaping executor * Fixed Adam Optimizer memory leak * Cleanup for PR * Added unit test for new ResourceScope method * Removing import that was added by overzealous ide * Add back in an import * Added flags for executor to know whether or not it owns NDArrays for disposal * Moving to ResourceScope.using implementation * Changes to make ResourceScope.using work with existing scope * Updating ResourceScope to work with existing scopes via usingIfScopeExists method * Fix clojure unit tests * Fixes to be compatibile with how clojure is using ResourceScope * Removing some unnecessary changes * Adding scope assertion in unit test
lanking520
pushed a commit
to lanking520/incubator-mxnet
that referenced
this issue
Apr 1, 2019
…e#14372) * Fixes for memory leak when reshaping executor * Fixed Adam Optimizer memory leak * Cleanup for PR * Added unit test for new ResourceScope method * Removing import that was added by overzealous ide * Add back in an import * Added flags for executor to know whether or not it owns NDArrays for disposal * Moving to ResourceScope.using implementation * Changes to make ResourceScope.using work with existing scope * Updating ResourceScope to work with existing scopes via usingIfScopeExists method * Fix clojure unit tests * Fixes to be compatibile with how clojure is using ResourceScope * Removing some unnecessary changes * Adding scope assertion in unit test
ZhennanQin
pushed a commit
to ZhennanQin/incubator-mxnet
that referenced
this issue
Apr 3, 2019
…e#14372) * Fixes for memory leak when reshaping executor * Fixed Adam Optimizer memory leak * Cleanup for PR * Added unit test for new ResourceScope method * Removing import that was added by overzealous ide * Add back in an import * Added flags for executor to know whether or not it owns NDArrays for disposal * Moving to ResourceScope.using implementation * Changes to make ResourceScope.using work with existing scope * Updating ResourceScope to work with existing scopes via usingIfScopeExists method * Fix clojure unit tests * Fixes to be compatibile with how clojure is using ResourceScope * Removing some unnecessary changes * Adding scope assertion in unit test
nswamy
pushed a commit
that referenced
this issue
Apr 5, 2019
* Fixes for memory leak when reshaping executor * Fixed Adam Optimizer memory leak * Cleanup for PR * Added unit test for new ResourceScope method * Removing import that was added by overzealous ide * Add back in an import * Added flags for executor to know whether or not it owns NDArrays for disposal * Moving to ResourceScope.using implementation * Changes to make ResourceScope.using work with existing scope * Updating ResourceScope to work with existing scopes via usingIfScopeExists method * Fix clojure unit tests * Fixes to be compatibile with how clojure is using ResourceScope * Removing some unnecessary changes * Adding scope assertion in unit test
lanking520
added a commit
that referenced
this issue
Apr 5, 2019
* Fixes for memory leak when reshaping executor * Fixed Adam Optimizer memory leak * Cleanup for PR * Added unit test for new ResourceScope method * Removing import that was added by overzealous ide * Add back in an import * Added flags for executor to know whether or not it owns NDArrays for disposal * Moving to ResourceScope.using implementation * Changes to make ResourceScope.using work with existing scope * Updating ResourceScope to work with existing scopes via usingIfScopeExists method * Fix clojure unit tests * Fixes to be compatibile with how clojure is using ResourceScope * Removing some unnecessary changes * Adding scope assertion in unit test
haohuanw
pushed a commit
to haohuanw/incubator-mxnet
that referenced
this issue
Jun 23, 2019
…e#14372) * Fixes for memory leak when reshaping executor * Fixed Adam Optimizer memory leak * Cleanup for PR * Added unit test for new ResourceScope method * Removing import that was added by overzealous ide * Add back in an import * Added flags for executor to know whether or not it owns NDArrays for disposal * Moving to ResourceScope.using implementation * Changes to make ResourceScope.using work with existing scope * Updating ResourceScope to work with existing scopes via usingIfScopeExists method * Fix clojure unit tests * Fixes to be compatibile with how clojure is using ResourceScope * Removing some unnecessary changes * Adding scope assertion in unit test
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
Memory leak issue while using Adam optimizer with MXNet Scala Bindings. Running the code below will keep consuming more and more memory till you run out.
Steps to Reproduce
Issue
The issue is (I think) some temporary NDArrays are not getting disposed in Adam optimizer when using
disposeDepsExcept
.The places exactly where the memory leak occurs is in 3 locations where the method
disposeDepsExcept
is used in Adam'supdate
method.Temporary Fix
Replace all the 3 lines that use
disposeDepsExcept
inupdate
method ofAdam.scala
by explicitly disposing the temporary NDArrays that were created as shown belowInstead of the 3 following lines in
Adam.scala
Replace it by:
The above changes fixes things for now, but for some reason
disposeDepsExcept
is not doing its job in this case.Environment info (Required)
Package used (Python/R/Scala/Julia): Scala
For Scala user, please provide:
Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio): gcc
MXNet commit hash: 96b4b6e
The text was updated successfully, but these errors were encountered: