[MKLDNN]Enhance Quantization APIs and Tutorial #15448

xinyu-intel · 2019-07-03T01:45:27Z

Description

Create a MKL-DNN specific user-level api quantize_model_mkldnn which combines fusion and quantization.
Enable resnet50_v1b quantized model.
Split quantize_model API into three parts to make it flexible for users to integrate quantization flow into their project:
1)quantize_graph: quantize fp32 model to int8 model w/o calibration and return a collector for collecting calibration information in the next step.
2)[outside api]: users need only add a few lines together with mod.forward for collecting calibration information.
3)calib_graph: generate calibrated model based on filled collector.
Draft a tutorial to introduce How to quantize custom models for production-level inference with MKL-DNN backend.

@pengzhao-intel @TaoLv @ZhennanQin @ciyongch

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

update

pengzhao-intel · 2019-07-03T07:10:32Z

@anirudh2290 @ThomasDelteil as we discussed in the forum, we post the developer guide for the user who wants to integrate quantization flow into their script.

CC @reminisce @ZhennanQin @ElaineBao

Any suggestion is highly appreciated :)

Update MKLDNN_QUANTIZATION.md

…-intel/incubator-mxnet into enhance_quantization_api_1

docs/tutorials/mkldnn/MKLDNN_QUANTIZATION.md

xinyu-intel · 2019-07-03T08:13:19Z

http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-15448/12/tutorials/mkldnn/MKLDNN_QUANTIZATION.html

pengzhao-intel · 2019-07-03T10:08:46Z

@KellenSunderland please help take a review too :)

TaoLv · 2019-07-03T14:33:53Z

nit: use lower case in the name of document.

…-intel/incubator-mxnet into enhance_quantization_api_1

docs/tutorials/mkldnn/mkldnn_quantization.md

xinyu-intel · 2019-07-08T06:57:28Z

@ThomasDelteil Comments addressed, Please take a look at again:)

roywei · 2019-07-08T15:16:38Z

@mxnet-label-bot add [MKLDNN, Doc]

pengzhao-intel · 2019-07-09T02:02:03Z

@aaronmarkham could you help take a review for the new document?

pengzhao-intel · 2019-07-11T03:03:17Z

@ThomasDelteil Would you mind to take a review again?

ThomasDelteil · 2019-07-11T05:37:37Z

Will do, at a conference this week, limited bandwidth but next week I'll have some availability to look into quantization again and get back to you on the different email threads as well, apologies for the delay!

pengzhao-intel · 2019-07-11T05:45:17Z

Will do, at a conference this week, limited bandwidth but next week I'll have some availability to look into quantization again and get back to you on the different email threads as well, apologies for the delay!

Sure :) Have a good trip on AMLC.
Maybe you can bring more amazing ideas to improve the API and usability of quantization flow from the conference :) We're highly appreciated for the inputs and feedbacks.

Thanks in advance.

karan6181 · 2019-07-19T01:10:27Z

@ThomasDelteil Could you please review this PR once you have time? Thanks!

pengzhao-intel · 2019-07-23T00:53:16Z

@ThomasDelteil do you have a chance to review this week?
We have other improvements in GluonCV which depends on this PR.

pengzhao-intel · 2019-07-23T00:54:08Z

@ciyongch @ZhennanQin @ElaineBao please take a review and test in local.

ElaineBao · 2019-07-23T01:44:15Z

docs look good to me.

pengzhao-intel · 2019-08-01T06:02:45Z

@ThomasDelteil we are going to merge this PR in 24 hours if no further comments since other improvements depend on this.

pengzhao-intel

LGTM

pengzhao-intel · 2019-08-01T20:40:33Z

Merging now. We're continually improving the quantization flow so any suggestions and feedbacks are highly appreciated.

* enhance api and new tutorial * Update MKLDNN_QUANTIZATION.md update * fix lint * modify pics * skip test * add quantize layer in graph * update * remove center css flag * change requantize color * fix markdown pics * change to use png * Update MKLDNN_QUANTIZATION.md update * enable ipython script * fix png * fix lint * Update MKLDNN_QUANTIZATION.md * change title * trigger * use lower case * some typo * some typo * use dmlc web data * trigger * trigger

xinyu-intel and others added 3 commits July 3, 2019 09:26

enhance api and new tutorial

78d1606

rebase code

e663e53

Update MKLDNN_QUANTIZATION.md

f1808bd

update

xinyu-intel requested a review from szha as a code owner July 3, 2019 01:45

xinyu-intel and others added 10 commits July 3, 2019 10:15

fix lint

6567071

modify pics

872afce

skip test

381fa90

add quantize layer in graph

9ffa5ce

update

b4b4077

remove center css flag

4969479

change requantize color

b3776b4

fix markdown pics

dbf8870

change to use png

27e1492

Update MKLDNN_QUANTIZATION.md

ff5f253

update

Merge pull request #3 from pengzhao-intel/patch-2

a7d8924

Update MKLDNN_QUANTIZATION.md

xinyu-intel added 2 commits July 3, 2019 15:18

enable ipython script

6ff34f3

Merge branch 'enhance_quantization_api_1' of https://github.com/xinyu…

66f3386

…-intel/incubator-mxnet into enhance_quantization_api_1

ElaineBao reviewed Jul 3, 2019

View reviewed changes

docs/tutorials/mkldnn/MKLDNN_QUANTIZATION.md Outdated Show resolved Hide resolved

xinyu-intel added 3 commits July 3, 2019 15:30

fix png

c8326f6

fix lint

41b33a5

Update MKLDNN_QUANTIZATION.md

06162ed

change title

27dd471

trigger

e18a487

TaoLv mentioned this pull request Jul 3, 2019

[Doc] Improve the document for MKL-DNN backend #14399

Closed

5 tasks

use lower case

9d8fb59

Merge branch 'enhance_quantization_api_1' of https://github.com/xinyu…

6e63482

…-intel/incubator-mxnet into enhance_quantization_api_1

ThomasDelteil suggested changes Jul 4, 2019

View reviewed changes

xinyu-intel added 3 commits July 4, 2019 22:24

some typo

0f4ab4f

some typo

4ece731

fix some typo

73f2070

xinyu-intel mentioned this pull request Jul 5, 2019

Add mkldnn quantization tutorial pictures dmlc/web-data#193

Merged

use dmlc web data

316ec7e

trigger

c85b8e4

marcoabreu added Doc MKLDNN labels Jul 8, 2019

trigger

faf5e64

pengzhao-intel mentioned this pull request Jul 19, 2019

[Discussion] 1.6.0 Roadmap #15589

Closed

pengzhao-intel approved these changes Aug 1, 2019

View reviewed changes

pengzhao-intel merged commit b3064c5 into apache:master Aug 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MKLDNN]Enhance Quantization APIs and Tutorial #15448

[MKLDNN]Enhance Quantization APIs and Tutorial #15448

xinyu-intel commented Jul 3, 2019

pengzhao-intel commented Jul 3, 2019

xinyu-intel commented Jul 3, 2019 •

edited

Loading

pengzhao-intel commented Jul 3, 2019

TaoLv commented Jul 3, 2019

xinyu-intel commented Jul 8, 2019

roywei commented Jul 8, 2019

pengzhao-intel commented Jul 9, 2019

pengzhao-intel commented Jul 11, 2019

ThomasDelteil commented Jul 11, 2019

pengzhao-intel commented Jul 11, 2019

karan6181 commented Jul 19, 2019

pengzhao-intel commented Jul 23, 2019

pengzhao-intel commented Jul 23, 2019

ElaineBao commented Jul 23, 2019

pengzhao-intel commented Aug 1, 2019

pengzhao-intel left a comment

pengzhao-intel commented Aug 1, 2019

[MKLDNN]Enhance Quantization APIs and Tutorial #15448

[MKLDNN]Enhance Quantization APIs and Tutorial #15448

Conversation

xinyu-intel commented Jul 3, 2019

Description

Checklist

Essentials

Changes

Comments

pengzhao-intel commented Jul 3, 2019

xinyu-intel commented Jul 3, 2019 • edited Loading

pengzhao-intel commented Jul 3, 2019

TaoLv commented Jul 3, 2019

xinyu-intel commented Jul 8, 2019

roywei commented Jul 8, 2019

pengzhao-intel commented Jul 9, 2019

pengzhao-intel commented Jul 11, 2019

ThomasDelteil commented Jul 11, 2019

pengzhao-intel commented Jul 11, 2019

karan6181 commented Jul 19, 2019

pengzhao-intel commented Jul 23, 2019

pengzhao-intel commented Jul 23, 2019

ElaineBao commented Jul 23, 2019

pengzhao-intel commented Aug 1, 2019

pengzhao-intel left a comment

Choose a reason for hiding this comment

pengzhao-intel commented Aug 1, 2019

xinyu-intel commented Jul 3, 2019 •

edited

Loading