MKLDNN fallback when not recording gradients and calling backwards #12411

azai91 · 2018-08-30T16:29:07Z

Description

PR to address (#10994). There is a case where users may want to run the backwards pass but not record any gradients (https://mxnet.incubator.apache.org/api/python/autograd/autograd.html#mxnet.autograd.record - this should be addressed in a later PR as it does not make sense). MKLDNN does not handle this case and instead we will fallback.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

azai91 · 2018-08-30T22:05:28Z

This PR is waiting merge of #12019

stu1130 · 2018-09-18T01:09:53Z

@azai91 #12019 was merged. Is it still WIP?

azai91 · 2018-09-24T18:38:14Z

still working in progress. working with customer on different issue. will fix this next.

vandanavk · 2018-09-25T17:13:54Z

@mxnet-label-bot [pr-work-in-progress]

vrakesh · 2018-10-09T07:03:13Z

@azai91 requesting an update on this PR.

Roshrini · 2018-10-22T17:33:49Z

@azai91 Any update on this PR?

ankkhedia · 2018-10-30T19:06:00Z

@azai91 Thanks for the contribution!
Could you please update this PR?

anirudhacharya · 2018-11-13T00:36:18Z

@azai91 any update on this PR. you could close it and reopen the PR once the changes are ready.

@sandeep-krishnamurthy @anirudh2290

stu1130 · 2018-11-21T19:04:28Z

@azai91 could you address the CI failure?

vandanavk · 2018-11-27T22:00:10Z

@mxnet-label-bot add [pr-awaiting-testing]

azai91 · 2018-11-28T01:39:37Z

still investigating. this PR goes beyond the the original issue I was addressing.

azai91 · 2018-11-28T22:51:47Z

addressed original problem with issuse (#10994). However, I discovered another issue and created a new PR to address (#13445).

anirudh2290 · 2018-11-29T00:26:12Z

tests/python/unittest/test_gluon.py

-    check_hybrid_static_memory()
-    check_hybrid_static_memory(static_alloc=True)
-    check_hybrid_static_memory(static_alloc=True, static_shape=True)
+    check_hybrid_static_memory(train_mode=[True, False])


shouldnt it be train_modes ?

This reverts commit 895fd9c.

This reverts commit 6bda387.

This reverts commit 30e0636.

roywei · 2018-12-11T00:56:19Z

@azai91 Thanks for taking the time to dive into the issue, could you resolve the conflict and trigger CI if still working on this?

sandeep-krishnamurthy · 2018-12-26T21:31:14Z

@azai91 - can you please rebase and fix CI issues? @pengzhao-intel - You may be interested in this PR?

pengzhao-intel · 2018-12-27T02:47:57Z

Actually, this is a system-level issue that how do we handle the situation the backward is called with the is_train=False. I think it doesn't can be done by the simple bypass.

We will have the offline discussion first and back later @TaoLv .

TaoLv

Seems I missed some context of this issue. What's the expectation in mxnet if backward is called with is_train=false? What's the grad_output for backward function and if needed, what's the workspace for backward function?

Roshrini · 2019-01-08T00:41:00Z

@azai91 Can you please rebase this PR?

stu1130 · 2019-01-16T01:19:12Z

@azai91 ping again! thanks

pengzhao-intel · 2019-01-16T01:31:48Z

IMO, this usage and default behavior is not clear even in the original path.
@azai91 @stu1130 could you help get a clear picture first and then we can discuss how to handle it by MKLDNN backend?

sandeep-krishnamurthy · 2019-01-25T20:09:07Z

@apeforest - @TaoLv raised a good question here. Can you please help us answer -

What's the expectation in mxnet if backward is called with is_train=false? What's the grad_output >> for backward function and if needed, what's the workspace for backward function?

pengzhao-intel · 2019-03-14T06:51:44Z

ping, any update @azai91

abhinavs95 · 2019-03-27T17:34:33Z

@azai91 Could you please rebase and fix the CI issues?

abhinavs95 · 2019-03-27T17:35:34Z

@mxnet-label-bot update [pr-awaiting-response, pr-work-in-progress]

azai91 requested a review from anirudh2290 as a code owner August 30, 2018 16:29

azai91 force-pushed the fix/10994 branch 3 times, most recently from 1ccef95 to a56a306 Compare August 30, 2018 22:16

marcoabreu added the pr-work-in-progress PR is still work in progress label Sep 25, 2018

azai91 added 5 commits November 1, 2018 16:19

fallback if not training in batch norm

04bbb8f

add convolution fallback

783e626

fallback for remaining mkldnn operators backwards

d0eeee7

fall back if need_grad is false

7bcc962

add unit test for record false

64fe9c4

azai91 force-pushed the fix/10994 branch from a56a306 to 64fe9c4 Compare November 1, 2018 23:19

Merge branch 'master' into fix/10994

f638501

marcoabreu added the pr-awaiting-testing PR is reviewed and waiting CI build and test label Nov 27, 2018

azai91 added 4 commits November 27, 2018 17:40

Merge branch 'master' into fix/10994

95e360f

change to need_grad

30e0636

remove individual checks

6bda387

skip testing static_shape record false

7a7ca2f

azai91 changed the title ~~[WIP] MKLDNN fallback when not recording gradients and calling backwards~~ MKLDNN fallback when not recording gradients and calling backwards Nov 28, 2018

add todo issue numbmer

62dd2d0

anirudh2290 reviewed Nov 29, 2018

View reviewed changes

fix typo

e5c5976

azai91 force-pushed the fix/10994 branch from 5281885 to e5c5976 Compare November 29, 2018 00:30

set to need_grad other places

895fd9c

azai91 force-pushed the fix/10994 branch from 88ed8c0 to 895fd9c Compare November 29, 2018 00:43

azai91 added 4 commits November 28, 2018 17:44

retrigger

e9f3def

Revert "set to need_grad other places"

e774956

This reverts commit 895fd9c.

Revert "remove individual checks"

c346942

This reverts commit 6bda387.

Revert "change to need_grad"

44344a0

This reverts commit 30e0636.

TaoLv reviewed Dec 27, 2018

View reviewed changes

marcoabreu added pr-awaiting-response PR is reviewed and waiting for contributor to respond and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Mar 27, 2019

azai91 closed this Mar 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MKLDNN fallback when not recording gradients and calling backwards #12411

MKLDNN fallback when not recording gradients and calling backwards #12411

azai91 commented Aug 30, 2018

azai91 commented Aug 30, 2018

stu1130 commented Sep 18, 2018

azai91 commented Sep 24, 2018

vandanavk commented Sep 25, 2018

vrakesh commented Oct 9, 2018

Roshrini commented Oct 22, 2018

ankkhedia commented Oct 30, 2018

anirudhacharya commented Nov 13, 2018

stu1130 commented Nov 21, 2018

vandanavk commented Nov 27, 2018

azai91 commented Nov 28, 2018

azai91 commented Nov 28, 2018

anirudh2290 Nov 29, 2018

roywei commented Dec 11, 2018

sandeep-krishnamurthy commented Dec 26, 2018

pengzhao-intel commented Dec 27, 2018

TaoLv left a comment

Roshrini commented Jan 8, 2019

stu1130 commented Jan 16, 2019 •

edited

Loading

pengzhao-intel commented Jan 16, 2019

sandeep-krishnamurthy commented Jan 25, 2019

pengzhao-intel commented Mar 14, 2019

abhinavs95 commented Mar 27, 2019

abhinavs95 commented Mar 27, 2019

MKLDNN fallback when not recording gradients and calling backwards #12411

MKLDNN fallback when not recording gradients and calling backwards #12411

Conversation

azai91 commented Aug 30, 2018

Description

Checklist

Essentials

Changes

Comments

azai91 commented Aug 30, 2018

stu1130 commented Sep 18, 2018

azai91 commented Sep 24, 2018

vandanavk commented Sep 25, 2018

vrakesh commented Oct 9, 2018

Roshrini commented Oct 22, 2018

ankkhedia commented Oct 30, 2018

anirudhacharya commented Nov 13, 2018

stu1130 commented Nov 21, 2018

vandanavk commented Nov 27, 2018

azai91 commented Nov 28, 2018

azai91 commented Nov 28, 2018

anirudh2290 Nov 29, 2018

Choose a reason for hiding this comment

roywei commented Dec 11, 2018

sandeep-krishnamurthy commented Dec 26, 2018

pengzhao-intel commented Dec 27, 2018

TaoLv left a comment

Choose a reason for hiding this comment

Roshrini commented Jan 8, 2019

stu1130 commented Jan 16, 2019 • edited Loading

pengzhao-intel commented Jan 16, 2019

sandeep-krishnamurthy commented Jan 25, 2019

pengzhao-intel commented Mar 14, 2019

abhinavs95 commented Mar 27, 2019

abhinavs95 commented Mar 27, 2019

stu1130 commented Jan 16, 2019 •

edited

Loading