Pointwise fusion for GPU #15167

ptrendx · 2019-06-06T22:08:25Z

Description

This PR enables fusion of pointwise ops for GPU context, using runtime compilation (NVRTC). I will make and post a design doc in the near future.

Work done by me, @Caenorst, @nvchai and @MoisesHer.

Important note: this is NOT meant to compete with other long term compilation strategies for MXNet (like integration with TVM etc.). It is intended as a short term solution (and so e.g. does not tackle harder problems like fusion with convolutions/gemms) while those long term solutions are in development.

This is the first stage of this effort, focusing on pointwise ops that do not change the shape of the output (+ slices). Future PRs will tackle broadcast ops.

FYI @zheng-da @junrushao1994 @eric-haibin-lin @szha @KellenSunderland @nvchai

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

The fusion is controlled by MXNET_USE_FUSION env variable and currently on by default for GPU context (mostly to see how it will behave in the CI environment) - I will probably change it to opt-in later in the review process.
It supports both forward and backward pass
It supports rebinding with different shapes without the need of recompilation of the generated kernel.
It currently supports Symbol, Module and Gluon (only hybridized without inlining, so mostly static_alloc=True) API.

…ion, use the name of input nodes in the cuda code

…t, return a new graph to reset the indexed_graph

…ing instead of main graph topological ordering, add tvm.patch

…kward graph

… node_id, invert control_deps loop to modify topology of subgraph before calling its indexed_graph(), check that all node of the first DFSVisit are actually in the subgraph

Add powerscalar and rpowerscalar, fix return type of zero and one Cleaning, fixing lint Go back to proper TVM submodule

piyushghai · 2019-06-07T06:02:15Z

@ptrendx Can you look at the CI failures on this PR ?

@mxnet-label-bot Add [pr-awaiting-review, Operator]

ptrendx · 2019-06-07T21:13:17Z

Design doc: https://cwiki.apache.org/confluence/display/MXNET/GPU+Pointwise+fusion

Jerryzcn · 2019-10-25T00:15:41Z

@ptrendx Does this support AMP? Do you observe any speedup when training mask rcnn?

ptrendx · 2019-10-25T00:25:19Z

@Jerryzcn Yes, it does support AMP. I saw a (pretty small, 1-2%) speedup in the non-FPN version of MaskRCNN from GluonCV.

samskalicky

Approach looks great, have a few concerns about style/comments. Thanks for your effort here, clearly this is a ton of work!

src/executor/infer_graph_attr_pass.cc

src/executor/simple_partition_pass.h

DickJC123

An important new facility. Nice work! LGTM.

rondogency · 2019-11-05T20:36:33Z

Hi @DickJC123 looks like this PR increases the CI centos-gpu testing time by over 100%, which is caused by some gpu tests running time increasing by over 30 times. For example:

test_operator_gpu.test_sparse_mathematical_core goes from ~13s to ~350s, test_operator_gpu.test_lstm_bidirectional goes from ~15s to ~450s,
test_operator_gpu.test_rnnrelu_bidirectional goes from ~4s to ~250s.

So this PR may cause some regression, so can you take a look if you have time? I did the comparison between this PR and previous PRs.

rondogency · 2019-11-05T21:39:04Z

@ptrendx @Caenorst @nvchai

sxjscience · 2019-11-05T23:53:03Z

Would it be more appropriate to turn off fused_op by default? We can still provide documentation for the user to turn on it manually.

* Beginning of RTC of pointwise ops * Code generation from the given JSON * add initial simple_partition_pass and use it for pointwise fusion * fix the fusion, use a symbol.Copy() at the beginning of binding function, use the name of input nodes in the cuda code * Fixes * Adding support for attribute inference for backward nodes when fusing * keep proper input ordering for fused Op * instantiate the indexed_graph before starting the subgraph replacement, return a new graph to reset the indexed_graph * Fuse backward * fix ordering of subgraph node inputs using subgraph topological ordering instead of main graph topological ordering, add tvm.patch * excluse forward node fusion during the fusion of the nodes in the backward graph * Dealing with fused backward nodes inferattr * use subgraph.indexed_graph() instead of main for _FusedOpHelper nodes node_id, invert control_deps loop to modify topology of subgraph before calling its indexed_graph(), check that all node of the first DFSVisit are actually in the subgraph * Adding support for other reqs in codegen * Fix * Cleaning * Change the TVM submodule * More cleaning * Making linter happy * Do fusion only if default context is GPU * Fixes for tests Add powerscalar and rpowerscalar, fix return type of zero and one Cleaning, fixing lint Go back to proper TVM submodule * Fix the TVM commit * Fix lint * Guard fusion with MXNET_USE_CUDA * Fix * Fix clang-tidy * Add erf and erfinv backward * Gluon support for fusion * Cleaning * Cleaning and allow shape/type change in FusedOp * Fixing Gluon bugs * Fixing after rebase * Fixing race condition and guarding against races when using NVRTC * Cleaning and renaming FusedOp to _FusedOp * Going easy on Windows compiler * Disable fusion on Windows for now * Refactor InferAttr and InferShapeAttr * Added slice and half2 support to FusedOp * Fix lint errors * Added multiple types support for vector loading/storing * add slice fusion when it's at the beginning of subgraphs * Removed constant ndim assumption in fused op * Fix memory alignment issue in slice for FusedOp * Fixes * Fix lint errors * Do not include cuda_fp16.h * Refactor fused op op lists * Make linter happy * Changes from review * Fixes after rebase * Expand FusedOp support for slice * Fix for fp16 _zeros and _ones * Fix * Moving aux functions to unnamed namespace and detail namespace -> fusion namespace * Disabling fusion if it alters topological order of inputs * Print code only when env variable is set * Fix * Fix lint and 2 tests that specify the same names for multiple inputs * Fixes from review and disabling fusion of slice with non-default step * Add amp_cast to fusion, fixes * Add amp_multicast and its backward to the list of support ops * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Make clearer comment * Adding punctuation and capitalization to \brief descriptions * Fix * Fix * Add backward_cast to fusion * Adding unittests for fusion. Fix for erfinv_grad * Adding slice ops and add_n to tests * Fixes from review * Setting inplace option * Fix lint * Storing double in half * Retrigger CI * Slight relaxing of the relative tolerance in the test * Move the env variable check to the end * Fix a race condition between InferShape and scheduled Forward * Fix flakey test_fusion test involving fp32 erfinv op. * Fix from review * Added broadcast_like and slice_like to fused op * Minor fix and cleanup * Added negative axis support in slice_axis, temporarily disabled fusion of slice_like and broadcast_like * Added axes support to slice_like * Added axis support to broadcast_like * Add fast_load_slice function to fused op code * Added runtime switch for choosing fast and slow slice kernel * Fix lint and warning * Going easy on Windows compiler (again) * Fix slice_like * Debug broadcast_like fusion * Fix lint * Fix lint * Trigger CI * Get rid of the initializer list * Fix backward calls with different gradient type * avoid cycle when adding node specific for inputs of subgraph for pointwise fusion * Fix lint * Add namespace to the fusion implementations * Set launch bounds on the fused kernel * Fix NumPy tests * Test showcasing an issue fixed in PR #16553 * Cast scalarts to FP32 and perform (a*1.0/b) instead of (a/b) Fix lint errors Fix lint * Fix a bug in cycle detection for inputs only op in pointwise fusion * Add comments to simple_partition_pass.h file

* Beginning of RTC of pointwise ops * Code generation from the given JSON * add initial simple_partition_pass and use it for pointwise fusion * fix the fusion, use a symbol.Copy() at the beginning of binding function, use the name of input nodes in the cuda code * Fixes * Adding support for attribute inference for backward nodes when fusing * keep proper input ordering for fused Op * instantiate the indexed_graph before starting the subgraph replacement, return a new graph to reset the indexed_graph * Fuse backward * fix ordering of subgraph node inputs using subgraph topological ordering instead of main graph topological ordering, add tvm.patch * excluse forward node fusion during the fusion of the nodes in the backward graph * Dealing with fused backward nodes inferattr * use subgraph.indexed_graph() instead of main for _FusedOpHelper nodes node_id, invert control_deps loop to modify topology of subgraph before calling its indexed_graph(), check that all node of the first DFSVisit are actually in the subgraph * Adding support for other reqs in codegen * Fix * Cleaning * Change the TVM submodule * More cleaning * Making linter happy * Do fusion only if default context is GPU * Fixes for tests Add powerscalar and rpowerscalar, fix return type of zero and one Cleaning, fixing lint Go back to proper TVM submodule * Fix the TVM commit * Fix lint * Guard fusion with MXNET_USE_CUDA * Fix * Fix clang-tidy * Add erf and erfinv backward * Gluon support for fusion * Cleaning * Cleaning and allow shape/type change in FusedOp * Fixing Gluon bugs * Fixing after rebase * Fixing race condition and guarding against races when using NVRTC * Cleaning and renaming FusedOp to _FusedOp * Going easy on Windows compiler * Disable fusion on Windows for now * Refactor InferAttr and InferShapeAttr * Added slice and half2 support to FusedOp * Fix lint errors * Added multiple types support for vector loading/storing * add slice fusion when it's at the beginning of subgraphs * Removed constant ndim assumption in fused op * Fix memory alignment issue in slice for FusedOp * Fixes * Fix lint errors * Do not include cuda_fp16.h * Refactor fused op op lists * Make linter happy * Changes from review * Fixes after rebase * Expand FusedOp support for slice * Fix for fp16 _zeros and _ones * Fix * Moving aux functions to unnamed namespace and detail namespace -> fusion namespace * Disabling fusion if it alters topological order of inputs * Print code only when env variable is set * Fix * Fix lint and 2 tests that specify the same names for multiple inputs * Fixes from review and disabling fusion of slice with non-default step * Add amp_cast to fusion, fixes * Add amp_multicast and its backward to the list of support ops * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Make clearer comment * Adding punctuation and capitalization to \brief descriptions * Fix * Fix * Add backward_cast to fusion * Adding unittests for fusion. Fix for erfinv_grad * Adding slice ops and add_n to tests * Fixes from review * Setting inplace option * Fix lint * Storing double in half * Retrigger CI * Slight relaxing of the relative tolerance in the test * Move the env variable check to the end * Fix a race condition between InferShape and scheduled Forward * Fix flakey test_fusion test involving fp32 erfinv op. * Fix from review * Added broadcast_like and slice_like to fused op * Minor fix and cleanup * Added negative axis support in slice_axis, temporarily disabled fusion of slice_like and broadcast_like * Added axes support to slice_like * Added axis support to broadcast_like * Add fast_load_slice function to fused op code * Added runtime switch for choosing fast and slow slice kernel * Fix lint and warning * Going easy on Windows compiler (again) * Fix slice_like * Debug broadcast_like fusion * Fix lint * Fix lint * Trigger CI * Get rid of the initializer list * Fix backward calls with different gradient type * avoid cycle when adding node specific for inputs of subgraph for pointwise fusion * Fix lint * Add namespace to the fusion implementations * Set launch bounds on the fused kernel * Fix NumPy tests * Test showcasing an issue fixed in PR apache#16553 * Cast scalarts to FP32 and perform (a*1.0/b) instead of (a/b) Fix lint errors Fix lint * Fix a bug in cycle detection for inputs only op in pointwise fusion * Add comments to simple_partition_pass.h file

DickJC123 · 2019-11-06T20:48:51Z

Thanks for pointing out this large perf change. I will investigate.

ptrendx · 2019-11-07T00:45:56Z

I would say it is expected - the whole point of the feature is compile the portion of the model (which is much more expensive than just running that part) in other for subsequent iterations to run faster. Since CI runs unit tests, it creates a large number of models and never reuses them, so the compilation overhead can't be amortized. In order to reduce the overall ci time I would say the best way would be to enable fusion only on 1 or 2 configs in CI instead of all of them.

@sxjscience That approach traditionally did not work well - if you have a nonobvious feature that is not enabled by default users have no chance of actually getting benefit from it because that don't know they should be enabling it. Also, the CI example aside, there should never be a usecase where you do not want to enable fusion as it makes the training faster.

sxjscience · 2019-11-07T01:08:16Z

@ptrendx Okay, I just think that we may need more time to test for the cases in GluonNLP and GluonCV.

DickJC123 · 2019-11-08T00:43:46Z

After some investigation, I have an explanation and planned fix for the perf regression. To repeat what @ptrendx mentions, the real-time compilation of fused kernels takes additional time up front, with the idea that over many kernel invocations, the compile time will be more than made up for by the increased efficiency of the fused op. This matches the typical use case (unlike CI), so I believe that fusion should be left enabled by default.

Now one saving thing for each of the 3 tests mentioned by @rondogency is that most of the fused-ops in the test are duplicates of others seen earlier in the test. In fact <2% of the created fused-ops are unique. To fix this then I will submit a PR to introduce a 'fused op cache' that will map (source-code, gpu_arch) -> runnable kernel. This should eliminate most of the runtime compilations and correct in large part the issue flagged here.

* Beginning of RTC of pointwise ops * Code generation from the given JSON * add initial simple_partition_pass and use it for pointwise fusion * fix the fusion, use a symbol.Copy() at the beginning of binding function, use the name of input nodes in the cuda code * Fixes * Adding support for attribute inference for backward nodes when fusing * keep proper input ordering for fused Op * instantiate the indexed_graph before starting the subgraph replacement, return a new graph to reset the indexed_graph * Fuse backward * fix ordering of subgraph node inputs using subgraph topological ordering instead of main graph topological ordering, add tvm.patch * excluse forward node fusion during the fusion of the nodes in the backward graph * Dealing with fused backward nodes inferattr * use subgraph.indexed_graph() instead of main for _FusedOpHelper nodes node_id, invert control_deps loop to modify topology of subgraph before calling its indexed_graph(), check that all node of the first DFSVisit are actually in the subgraph * Adding support for other reqs in codegen * Fix * Cleaning * Change the TVM submodule * More cleaning * Making linter happy * Do fusion only if default context is GPU * Fixes for tests Add powerscalar and rpowerscalar, fix return type of zero and one Cleaning, fixing lint Go back to proper TVM submodule * Fix the TVM commit * Fix lint * Guard fusion with MXNET_USE_CUDA * Fix * Fix clang-tidy * Add erf and erfinv backward * Gluon support for fusion * Cleaning * Cleaning and allow shape/type change in FusedOp * Fixing Gluon bugs * Fixing after rebase * Fixing race condition and guarding against races when using NVRTC * Cleaning and renaming FusedOp to _FusedOp * Going easy on Windows compiler * Disable fusion on Windows for now * Refactor InferAttr and InferShapeAttr * Added slice and half2 support to FusedOp * Fix lint errors * Added multiple types support for vector loading/storing * add slice fusion when it's at the beginning of subgraphs * Removed constant ndim assumption in fused op * Fix memory alignment issue in slice for FusedOp * Fixes * Fix lint errors * Do not include cuda_fp16.h * Refactor fused op op lists * Make linter happy * Changes from review * Fixes after rebase * Expand FusedOp support for slice * Fix for fp16 _zeros and _ones * Fix * Moving aux functions to unnamed namespace and detail namespace -> fusion namespace * Disabling fusion if it alters topological order of inputs * Print code only when env variable is set * Fix * Fix lint and 2 tests that specify the same names for multiple inputs * Fixes from review and disabling fusion of slice with non-default step * Add amp_cast to fusion, fixes * Add amp_multicast and its backward to the list of support ops * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Make clearer comment * Adding punctuation and capitalization to \brief descriptions * Fix * Fix * Add backward_cast to fusion * Adding unittests for fusion. Fix for erfinv_grad * Adding slice ops and add_n to tests * Fixes from review * Setting inplace option * Fix lint * Storing double in half * Retrigger CI * Slight relaxing of the relative tolerance in the test * Move the env variable check to the end * Fix a race condition between InferShape and scheduled Forward * Fix flakey test_fusion test involving fp32 erfinv op. * Fix from review * Added broadcast_like and slice_like to fused op * Minor fix and cleanup * Added negative axis support in slice_axis, temporarily disabled fusion of slice_like and broadcast_like * Added axes support to slice_like * Added axis support to broadcast_like * Add fast_load_slice function to fused op code * Added runtime switch for choosing fast and slow slice kernel * Fix lint and warning * Going easy on Windows compiler (again) * Fix slice_like * Debug broadcast_like fusion * Fix lint * Fix lint * Trigger CI * Get rid of the initializer list * Fix backward calls with different gradient type * avoid cycle when adding node specific for inputs of subgraph for pointwise fusion * Fix lint * Add namespace to the fusion implementations * Set launch bounds on the fused kernel * Fix NumPy tests * Test showcasing an issue fixed in PR apache#16553 * Cast scalarts to FP32 and perform (a*1.0/b) instead of (a/b) Fix lint errors Fix lint * Fix a bug in cycle detection for inputs only op in pointwise fusion * Add comments to simple_partition_pass.h file

* Add cached op threadsafe version with corresponding C APIs, CPP Package changes, CI changes and tests * Fix download cmd in runtime_functions * Add CI changes * Add stage Fix indentation * Fix lint * Change to DEFAULT for C API * Fix mxnet_unit_tests path * export correct LD_LIBRARY_PATH * Add cpp include dirs * Build test with USE_CPP_PACKAGE * Add cached op threadsafe version with corresponding C APIs, CPP Package changes, CI changes and tests * Fix download cmd in runtime_functions * Merge * change mkldnn lib name * Add static_alloc, static_Shape support * Address review comments * Make GetCachedOpThreadSafeState similar to cached_op * Address review comments: comments for locking strategy * multithreaded inference tutorial * [Estimator] handle composite metrics in estimator (apache#16676) * handle composite metrics in estimator * fix composite metric case in handlers * remove unused import * [Estimator] refactor estimator to allow overriding evaluate/fit of a batch (apache#16678) * refactor estimator to allow overriding evaluate/fit of a batch * add doc to explain call structure and how to override * fix and doc * Pointwise fusion for GPU (apache#15167) * Beginning of RTC of pointwise ops * Code generation from the given JSON * add initial simple_partition_pass and use it for pointwise fusion * fix the fusion, use a symbol.Copy() at the beginning of binding function, use the name of input nodes in the cuda code * Fixes * Adding support for attribute inference for backward nodes when fusing * keep proper input ordering for fused Op * instantiate the indexed_graph before starting the subgraph replacement, return a new graph to reset the indexed_graph * Fuse backward * fix ordering of subgraph node inputs using subgraph topological ordering instead of main graph topological ordering, add tvm.patch * excluse forward node fusion during the fusion of the nodes in the backward graph * Dealing with fused backward nodes inferattr * use subgraph.indexed_graph() instead of main for _FusedOpHelper nodes node_id, invert control_deps loop to modify topology of subgraph before calling its indexed_graph(), check that all node of the first DFSVisit are actually in the subgraph * Adding support for other reqs in codegen * Fix * Cleaning * Change the TVM submodule * More cleaning * Making linter happy * Do fusion only if default context is GPU * Fixes for tests Add powerscalar and rpowerscalar, fix return type of zero and one Cleaning, fixing lint Go back to proper TVM submodule * Fix the TVM commit * Fix lint * Guard fusion with MXNET_USE_CUDA * Fix * Fix clang-tidy * Add erf and erfinv backward * Gluon support for fusion * Cleaning * Cleaning and allow shape/type change in FusedOp * Fixing Gluon bugs * Fixing after rebase * Fixing race condition and guarding against races when using NVRTC * Cleaning and renaming FusedOp to _FusedOp * Going easy on Windows compiler * Disable fusion on Windows for now * Refactor InferAttr and InferShapeAttr * Added slice and half2 support to FusedOp * Fix lint errors * Added multiple types support for vector loading/storing * add slice fusion when it's at the beginning of subgraphs * Removed constant ndim assumption in fused op * Fix memory alignment issue in slice for FusedOp * Fixes * Fix lint errors * Do not include cuda_fp16.h * Refactor fused op op lists * Make linter happy * Changes from review * Fixes after rebase * Expand FusedOp support for slice * Fix for fp16 _zeros and _ones * Fix * Moving aux functions to unnamed namespace and detail namespace -> fusion namespace * Disabling fusion if it alters topological order of inputs * Print code only when env variable is set * Fix * Fix lint and 2 tests that specify the same names for multiple inputs * Fixes from review and disabling fusion of slice with non-default step * Add amp_cast to fusion, fixes * Add amp_multicast and its backward to the list of support ops * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Apply wording suggestions from code review Co-Authored-By: Aaron Markham <[email protected]> * Make clearer comment * Adding punctuation and capitalization to \brief descriptions * Fix * Fix * Add backward_cast to fusion * Adding unittests for fusion. Fix for erfinv_grad * Adding slice ops and add_n to tests * Fixes from review * Setting inplace option * Fix lint * Storing double in half * Retrigger CI * Slight relaxing of the relative tolerance in the test * Move the env variable check to the end * Fix a race condition between InferShape and scheduled Forward * Fix flakey test_fusion test involving fp32 erfinv op. * Fix from review * Added broadcast_like and slice_like to fused op * Minor fix and cleanup * Added negative axis support in slice_axis, temporarily disabled fusion of slice_like and broadcast_like * Added axes support to slice_like * Added axis support to broadcast_like * Add fast_load_slice function to fused op code * Added runtime switch for choosing fast and slow slice kernel * Fix lint and warning * Going easy on Windows compiler (again) * Fix slice_like * Debug broadcast_like fusion * Fix lint * Fix lint * Trigger CI * Get rid of the initializer list * Fix backward calls with different gradient type * avoid cycle when adding node specific for inputs of subgraph for pointwise fusion * Fix lint * Add namespace to the fusion implementations * Set launch bounds on the fused kernel * Fix NumPy tests * Test showcasing an issue fixed in PR apache#16553 * Cast scalarts to FP32 and perform (a*1.0/b) instead of (a/b) Fix lint errors Fix lint * Fix a bug in cycle detection for inputs only op in pointwise fusion * Add comments to simple_partition_pass.h file * fix install dir (apache#16690) * [numpy] add numpy operator : append (apache#16564) * add operator : append ; fix op concatenate when axis = None * pylint disable remove mistake disable pylint * Initializer.__eq__ (apache#16680) * fix binary dependencies in CD and nightly (apache#16693) * [MKL-DNN] Add mxnet mkldnn cmake tutorial (apache#16688) * add mxnet mkldnn cmake instruction * imporve doc * OMP->OpenMP * Revert "[MKLDNN]Fix reorder2default (apache#16602)" (apache#16697) This reverts commit dd4eaf5. * [Estimator] refactor estimator and clarify docs (apache#16694) * refactor estimator and clarify docs * fix info message and test * clean up after releasing logging handler * Eliminate common expressions (apache#15657) * Eliminate common expressions from a graph * Guarding against optimizing out stateful ops and ops that require resource * Fix lint * Added THasDeterministicOutput to multiple ops * DDebug eliminate common expr * Added test * Expose get_optimized_symbol * Fix * Fix 2 * Add doc to the Python call * Add env var MXNET_ELIMINATE_COMMON_EXPR, default true * Add comments, improve readability of eliminate_common_expr_pass.cc * Expand testing * Lower priority of THasDeterministicOutput attr for equal Node test * Change mx.gpu() to mx.cpu() in tests * Skip CSE test on Windows (as env variable setting during test does not work there) * Add missing import sys * Add missing import logging * Backport of apache#16711, apache#16737, apache#16408 to 1.6 branch (apache#16763) * support mixed-precision true_divide (apache#16711) * [MKLDNN] use dim_t instead of int in slice/transpose operators (apache#16737) * use dim_t instead of int * fix same issue in pooling * rebase code * trigger CI * Add MXNet Ops for fast multihead attention (apache#16408) * add MXNet Ops for fast multihead attention * add cutlass as 3rdparty dependency * add cutlass to compilation flags * remove all cutlass stuff * add better error message and description and remove cutlass from compilation flags * change credit for the approach since the code have changed * fix typos * correct another typo * Add all the cuda/cublas helper functions * remove tests using kAddTo * only use cublasStridedBatchedGemm if CUDA >= 9.1 * add equivalent mxnet code in description of mha ops * remove a wrong copy-paste * add _contrib for namespace and add GPU only on description * add warning in bwd_ignore_zero_init description, also test with fp32 * add error return if bwd_ignore_zero_init is used without MXNET_EXEC_ENABLE_ADDTO * remove std::move for clang * remove bwd_ignore_zero_init flag * remove bwd_ignore_zero_init in test_operator_gpu.py * fix typo * fix another typo * Removed unrelated test * Add example and documentation for multi threaded inference * Add LICENSE * Add get_model.py * Add license for README * Refactor cached op and cached op threadsafe * Add limitation * Add tests for naive engine * Add latest test changes * Thread Safety tests in NaiveEngine mode * Thread Safety tests update * Update thread safety tests, add unsupported use cases * Changes to doc and refactor * Fix todo owner, indentation and mx_float->float * Refactor cached op code, remove num_threads arg from example * Fix lint * Fix warning * Add back cython, required for unix-gpu build * Fix for windows * Add bulking support for thread safe cached op version * Add support for subgraph testing * import mxnet before calling get_backend_symbol * Fix symbol json name * Refactor DynamicForward * Add comments * Add DMLC_ATTRIBUTE_UNUSED * Fix use_naive_run issue * Fix lint * Revert unittest_cpp to old test since it doesnt test thread safety * Fix doc Co-authored-by: Sheng Zha <[email protected]> Co-authored-by: Przemyslaw Tredak <[email protected]> Co-authored-by: Tao Lv <[email protected]> Co-authored-by: JiangZhaoh <[email protected]> Co-authored-by: Leonard Lausen <[email protected]> Co-authored-by: Xinyu Chen <[email protected]> Co-authored-by: Zhennan Qin <[email protected]>

ptrendx and others added 25 commits June 6, 2019 14:40

Beginning of RTC of pointwise ops

9653b67

Code generation from the given JSON

0e1774f

add initial simple_partition_pass and use it for pointwise fusion

8bf2945

fix the fusion, use a symbol.Copy() at the beginning of binding funct…

5cbb50d

…ion, use the name of input nodes in the cuda code

Fixes

fcf23c7

Adding support for attribute inference for backward nodes when fusing

892c18f

keep proper input ordering for fused Op

0a342a0

instantiate the indexed_graph before starting the subgraph replacemen…

07de800

…t, return a new graph to reset the indexed_graph

Fuse backward

975e8a6

fix ordering of subgraph node inputs using subgraph topological order…

6d9c0bf

…ing instead of main graph topological ordering, add tvm.patch

excluse forward node fusion during the fusion of the nodes in the bac…

384fbb0

…kward graph

Dealing with fused backward nodes inferattr

b9506ff

use subgraph.indexed_graph() instead of main for _FusedOpHelper nodes…

f30fbbb

… node_id, invert control_deps loop to modify topology of subgraph before calling its indexed_graph(), check that all node of the first DFSVisit are actually in the subgraph

Adding support for other reqs in codegen

1a2e30d

Fix

15fbed5

Cleaning

506b126

Change the TVM submodule

cf88753

More cleaning

b861af9

Making linter happy

d001b5d

Do fusion only if default context is GPU

48f1b94

Fixes for tests

37d4bbf

Add powerscalar and rpowerscalar, fix return type of zero and one Cleaning, fixing lint Go back to proper TVM submodule

Fix the TVM commit

616b932

Fix lint

56303c8

Guard fusion with MXNET_USE_CUDA

00e61cf

Fix

204ab30

ptrendx force-pushed the pr_fusion branch from b1b764b to 204ab30 Compare June 6, 2019 22:54

marcoabreu added Operator pr-awaiting-review PR is waiting for code review labels Jun 7, 2019

Caenorst and others added 2 commits October 28, 2019 17:08

Fix a bug in cycle detection for inputs only op in pointwise fusion

76aa154

Merge branch 'upstream' into pr_fusion

929b8e9

samskalicky reviewed Oct 30, 2019

View reviewed changes

src/executor/infer_graph_attr_pass.cc Show resolved Hide resolved

src/executor/simple_partition_pass.h Show resolved Hide resolved

src/executor/simple_partition_pass.h Show resolved Hide resolved

Add comments to simple_partition_pass.h file

3d1b5af

DickJC123 approved these changes Nov 1, 2019

View reviewed changes

DickJC123 merged commit 4149f8b into apache:master Nov 1, 2019

leezu mentioned this pull request Nov 7, 2019

Fused Op causes MXNetError #16747

Closed

zachgk mentioned this pull request Nov 8, 2019

FusedOp Failing Static Linked Build #16765

Closed

DickJC123 mentioned this pull request Nov 12, 2019

Speed fused_op compilation by caching ptx and jit-compiled device functions #16783

Merged

2 tasks

roywei mentioned this pull request Nov 12, 2019

keras-mxnet training failed with FusedOP #16785

Closed

DickJC123 mentioned this pull request Nov 16, 2019

USE_NVRTC -> ENABLE_CUDA_RTC to fix maven build. Add compile-guard to fusion. #16838

Merged

5 tasks

This was referenced Dec 5, 2019

[CD] dynamic libmxet pipeline fix + small fixes #16966

Merged

[CUDA 9.0] NVRTC Compilation failed #17020

Closed

RuRo mentioned this pull request Mar 16, 2020

Hybridized SoftReLU on the GPU is not numerically stable #17844

Closed

chinakook mentioned this pull request Apr 6, 2020

fix to skip node not in graph. apache/tvm#5238

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pointwise fusion for GPU #15167

Pointwise fusion for GPU #15167

ptrendx commented Jun 6, 2019 •

edited

Loading

piyushghai commented Jun 7, 2019

ptrendx commented Jun 7, 2019

Jerryzcn commented Oct 25, 2019

ptrendx commented Oct 25, 2019

samskalicky left a comment

DickJC123 left a comment

rondogency commented Nov 5, 2019 •

edited

Loading

rondogency commented Nov 5, 2019

sxjscience commented Nov 5, 2019

DickJC123 commented Nov 6, 2019

ptrendx commented Nov 7, 2019

sxjscience commented Nov 7, 2019

DickJC123 commented Nov 8, 2019 •

edited

Loading

Pointwise fusion for GPU #15167

Pointwise fusion for GPU #15167

Conversation

ptrendx commented Jun 6, 2019 • edited Loading

Description

Checklist

Essentials

Changes

Comments

piyushghai commented Jun 7, 2019

ptrendx commented Jun 7, 2019

Jerryzcn commented Oct 25, 2019

ptrendx commented Oct 25, 2019

samskalicky left a comment

Choose a reason for hiding this comment

DickJC123 left a comment

Choose a reason for hiding this comment

rondogency commented Nov 5, 2019 • edited Loading

rondogency commented Nov 5, 2019

sxjscience commented Nov 5, 2019

DickJC123 commented Nov 6, 2019

ptrendx commented Nov 7, 2019

sxjscience commented Nov 7, 2019

DickJC123 commented Nov 8, 2019 • edited Loading

ptrendx commented Jun 6, 2019 •

edited

Loading

rondogency commented Nov 5, 2019 •

edited

Loading

DickJC123 commented Nov 8, 2019 •

edited

Loading