Group Normalization

[haojin2]

Description

Group Normalization

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Forward
• Backward
• Unit tests
• Gluon block
• Unit test for Gluon

Comments

Related:
#14196 #11283
Flakiness Checks:

MXNET_TEST_COUNT=10000 nosetests tests/python/unittest/test_operator.py:test_groupnorm
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1680041162 to reproduce.
.
----------------------------------------------------------------------
Ran 1 test in 136.546s

OK

MXNET_TEST_COUNT=10000 nosetests tests/python/gpu/test_operator_gpu.py:test_groupnorm
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=2085401116 to reproduce.
.
----------------------------------------------------------------------
Ran 1 test in 172.874s

OK

[haojin2]

@junrushao1994 For review if you have time.

[szha]

cc @sxjscience

[hetong007]

@zhanghang1989 FYI

[zhreshold]

@haojin2 can you check why unitest is failing?

[haojin2]

@zhreshold Looking into it...

[haojin2]

@zhreshold The failure is on fp16 data type, previously it could endure 10000 trials on my end and now it could not, I'm looking into the issue.

[zhreshold]

@haojin2 is CI still failing?

[Jerryzcn]

Is there an issue related to CI failing?

[karan6181]

@sxjscience @zhreshold Could you please review this PR again?

@haojin2 Could you please re-trigger the CI and look into failure ? Thanks

[haojin2]

@zhreshold @Jerryzcn @sxjscience @eric-haibin-lin CI passed, ready for merge

[sxjscience]

I'll merge first. Need to make a PR to revise the test_util + add test for grad_req=kAddTo. Let's make sure all OPs are tested for grad_req=kAddTo (Need to combine the efforts with DeepNumpy).