[MXNET-1095] [WIP] Support higher order gradient for cos,sin,negative,mul

[sxjscience]

Description

For context, please refer #10002

Support higher order gradient for cos, sin, negative, mul. I'll add the test later. For now, you may use the following code to test the correctness:

import mxnet.ndarray as nd
from mxnet import autograd
from mxnet.test_utils import assert_almost_equal

x = nd.array([3.0])
x.attach_grad()
with autograd.record():
    y = nd.sin(x)
    y_grad = autograd.grad(y, x, create_graph=True, retain_graph=True)[0]
    z = y_grad * y_grad # Should be cos(x)^2
z.backward()
print(x.grad) # Should be - 2 * cos(x) * sin(x)
gt_grad = - 2 * nd.cos(x) * nd.sin(x)
assert_almost_equal(x.grad.asnumpy(), gt_grad.asnumpy())

[0.2794155]
<NDArray 1 @cpu(0)>

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• higher-order gradient of sin,cos,negative,mul, tests

Comments

• Need to support more OPs.

[sxjscience]

Need to benchmark the performance though. (I feel that the performance should be similar.)

[Roshrini]

@mxnet-label-bot [pr-work-in-progress, Operator]

[ankkhedia]

@sxjscience Thanks for the contribution!
Did you get a chance to benchmark theses changes?

[anirudhacharya]

@sxjscience pinging for update on this PR :)

[stu1130]

@sxjscience ping again
feel free to close it and reopen it once you have time

[sxjscience]

I'll continue working on it next week.

[roywei]

@sxjscience Thanks for the contribution, do you have any documentations to link? It will help othert contributors adding more operators. Thanks!

[sandeep-krishnamurthy]

@apeforest - FYI

[stu1130]

@sxjscience any plan to work on this? Thanks

[sxjscience]

I’m working on a paper these days. May find some time when the paper is submitted. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Jake Lee <[email protected]> Sent: Wednesday, January 16, 2019 6:56 AM To: apache/incubator-mxnet Cc: Xingjian SHI; Mention Subject: Re: [apache/incubator-mxnet] [MXNET-1095] [WIP] Support higher order gradient for cos,sin,negative,mul (#12821) @sxjscience<https://github.com/sxjscience> any plan to work on this? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#12821 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AE8D7rQWMq3OwgLo35mWD90CpdcLNwQXks5vDlyWgaJpZM4XbAw2>.

[pinaraws]

@sxjscience Any updates? Are you still planning to continue to work on this PR?

[sxjscience]

No. @apeforest will continue working on that.

…

________________________________ From: pinaraws <[email protected]> Sent: Tuesday, March 19, 2019 11:22:37 PM To: apache/incubator-mxnet Cc: Xingjian SHI; Mention Subject: Re: [apache/incubator-mxnet] [MXNET-1095] [WIP] Support higher order gradient for cos,sin,negative,mul (#12821) @sxjscience<https://github.com/sxjscience> Any updates? Are you still planning to continue to work on this PR? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#12821 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AE8D7pXnwrd7NhBBb694M3ehj9DmwdsJks5vYXE9gaJpZM4XbAw2>.

[piyushghai]

@sxjscience Can this PR be closed if the work is being followed up in another PR by @apeforest as commented above ?

[sxjscience]

Yes, it can be closed. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Piyush Ghai <[email protected]> Sent: Tuesday, April 9, 2019 8:25:49 AM To: apache/incubator-mxnet Cc: Xingjian SHI; Mention Subject: Re: [apache/incubator-mxnet] [MXNET-1095] [WIP] Support higher order gradient for cos,sin,negative,mul (#12821) @sxjscience<https://github.com/sxjscience> Can this PR be closed if the work is being followed up in another PR by @apeforest<https://github.com/apeforest> as commented above ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#12821 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AE8D7krU7VMtmzOIs2NoukTKyiA3lDqDks5ve94NgaJpZM4XbAw2>.