Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

NDArray API NN Optimizer (Multi-* update category) issues #15643

Closed
ChaiBapchya opened this issue Jul 23, 2019 · 3 comments
Closed

NDArray API NN Optimizer (Multi-* update category) issues #15643

ChaiBapchya opened this issue Jul 23, 2019 · 3 comments

Comments

@ChaiBapchya
Copy link
Contributor

Neural Network Optimizer updates such as
multi_mp_sgd_mom_update, multi_mp_sgd_update, multi_sgd_mom_update, multi_sgd_update

They are present in Symbol API doc but not in NDArray API doc.
However, upon checking definition of 1 of the operators

>>> help(mx.nd.multi_sgd_mom_update)
Help on function multi_sgd_mom_update:

multi_sgd_mom_update(*data, **kwargs)
    Momentum update function for Stochastic Gradient Descent (SGD) optimizer.
    
    Momentum update has better convergence rates on neural networks. Mathematically it looks
    like below:
    
    .. math::
    
      v_1 = \alpha * \nabla J(W_0)\\
      v_t = \gamma v_{t-1} - \alpha * \nabla J(W_{t-1})\\
      W_t = W_{t-1} + v_t
    
    It updates the weights using::
    
      v = momentum * v - learning_rate * gradient
      weight += v
    
    Where the parameter ``momentum`` is the decay rate of momentum estimates at each epoch.
    
    
    
    Defined in src/operator/optimizer_op.cc:L372
    
    Parameters
    ----------
    data : NDArray[]
        Weights, gradients and momentum
    lrs : tuple of <float>, required
        Learning rates.
    wds : tuple of <float>, required
        Weight decay augments the objective function with a regularization term that penalizes large weights. The penalty scales with the square of the magnitude of each weight.
    momentum : float, optional, default=0
        The decay rate of momentum estimates at each epoch.
    rescale_grad : float, optional, default=1
        Rescale gradient to grad = rescale_grad*grad.
    clip_gradient : float, optional, default=-1
        Clip gradient to the range of [-clip_gradient, clip_gradient] If clip_gradient <= 0, gradient clipping is turned off. grad = max(min(grad, clip_gradient), -clip_gradient).
    num_weights : int, optional, default='1'
@mxnet-label-bot
Copy link
Contributor

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Doc

@ChaiBapchya
Copy link
Contributor Author

@mxnet-label-bot add [Operator, Doc]

@ChaiBapchya ChaiBapchya changed the title NDArray API NN Optimizer (Multi-* update category) absent in Doc NDArray API NN Optimizer (Multi-* update category) issues Jul 27, 2019
ChaiBapchya added a commit to ChaiBapchya/mxnet that referenced this issue Jul 27, 2019
sandeep-krishnamurthy pushed a commit that referenced this issue Jul 28, 2019
* optimizer for opperf benchmark

* Trigger notification

* missed function call

* added params

* minor typos

* Trigger notification

* resolve default params

* temp remove multi op

* take care of #15643

* numbering typo
@ChaiBapchya
Copy link
Contributor Author

Looked at the wrong branch in the doc.
Found here -
http://mxnet.incubator.apache.org/versions/master/api/python/ndarray/ndarray.html#mxnet.ndarray.multi_mp_sgd_mom_update

Important to navigate to the Master branch in the Versions dropdown of the Home page of the website.

anirudhacharya pushed a commit to anirudhacharya/mxnet that referenced this issue Aug 20, 2019
…5522)

* optimizer for opperf benchmark

* Trigger notification

* missed function call

* added params

* minor typos

* Trigger notification

* resolve default params

* temp remove multi op

* take care of apache#15643

* numbering typo
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants