[MXNET-580] Add SN-GAN example #12419

stu1130 · 2018-08-31T04:26:11Z

Description

Add Spectral Normalization GAN example

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Spectral Normalization GAN example

Comments

N/A

…to sn_gan_example

vandanavk · 2018-09-04T18:49:58Z

example/gluon/sn-gan/spectral_normalization_gan.py

+        _u = self.u.data(CTX)
+        _v = None
+
+        for _ in range(1):


This for loop is executed only once? Do we need a loop in that case?

The original paper use for loop to calculate the result. I should make it as a hyperparameter

vandanavk · 2018-09-04T18:50:12Z

example/gluon/sn-gan/spectral_normalization_gan.py

+
+        self.params.setattr('u', _u)
+
+        return w / sigma


Is there a chance that sigma would be 0?

The chance is quite small but I would deal with that

sandeep-krishnamurthy · 2018-09-05T01:01:35Z

Thanks for your contributions @stu1130
@ThomasDelteil - Can you please take a look at this? Thank you.

ThomasDelteil · 2018-09-05T01:06:46Z

@stu1130 thanks for your contribution, can you add a README.md in the folder of the sn_gan with reference to papers / sample output etc.
Have a look at this one for inspiration for example: https://github.com/apache/incubator-mxnet/tree/master/example/gluon/embedding_learning

vandanavk · 2018-09-05T20:11:49Z

Please execute pylint on these files using ci/other/pylintrc in incubator-mxnet folder and fix errors (if any).

pylint --rcfile=ci/other/pylintrc --ignore-patterns="..so$$,..dll$$,..dylib$$" example/gluon/sn-gan/*.py

vandanavk · 2018-09-05T20:14:15Z

example/gluon/sn-gan/spectral_normalization_gan.py

+BETA = 0.5  # beta1 for adam
+OUTPUT_DIR = './data'  # output directory
+MANUAL_SEED = random.randint(1, 10000)  # manual seed
+CTX = mx.gpu()  # change to gpu if you have gpu


Default should be cpu?

I think I should use argparse to let the user specify parameters. And default value would be cpu.

sandeep-krishnamurthy

Very nice work @stu1130 :-) Few minor comments, please address

@ThomasDelteil - Ping.

sandeep-krishnamurthy · 2018-09-11T05:02:42Z

example/gluon/sn-gan/README.md

@@ -0,0 +1,38 @@
+# Spectral Normalization GAN
+
+This example implements [Spectral Normalization for Generative Adversarial Networks](https://openreview.net/pdf?id=B1QRgziT-) based on [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset.


Please refer to arxiv - https://arxiv.org/abs/1802.05957

sandeep-krishnamurthy · 2018-09-11T05:03:27Z

example/gluon/sn-gan/README.md

+Example runs and the results:
+
+```python
+python train.py --use-gpu --data-path=data/CIFAR10


Write a note that user needs to download CIFAR10 dataset

sandeep-krishnamurthy · 2018-09-11T05:05:46Z

example/gluon/sn-gan/model.py

+
+    def __init__(self, num_filter, kernel_size,
+                 strides, padding, in_channels,
+                 ctx, iterations=1):


set default ctx=mx.cpu()

sandeep-krishnamurthy · 2018-09-11T05:07:52Z

example/gluon/sn-gan/model.py

+    g_net = gluon.nn.Sequential()
+    with g_net.name_scope():
+
+        g_net.add(gluon.nn.Conv2DTranspose(512, 4, 1, 0, use_bias=False))


It is always more clear for readers in these kind of examples to have named parameters for these layers.

…to sn_gan_example

ThomasDelteil

Solid contribution, thanks @stu1130

ThomasDelteil · 2018-09-11T19:01:27Z

example/gluon/sn-gan/data.py

+
+
+def get_training_data(batch_size):
+    """ hepler function to get dataloader"""


ThomasDelteil · 2018-09-11T19:03:23Z

example/gluon/sn-gan/model.py

+    def spectral_norm(self):
+        """ spectral normalization """
+        w = self.params.get('weight').data(self.ctx)
+        w_mat = w


I don't think this is necessary, you can simply use w in your nd.reshape(w_mat..

ThomasDelteil · 2018-09-11T19:04:35Z

example/gluon/sn-gan/model.py

+            self.u = self.params.get(
+                'u', init=mx.init.Normal(), shape=(1, num_filter))
+
+    def spectral_norm(self):


I would suggest using _spectral_norm(self) as it is a private function

ThomasDelteil · 2018-09-11T19:06:08Z

example/gluon/sn-gan/train.py

+parser.add_argument('--lr', type=float, default=0.0001,
+                    help='learning rate. default is 0.0001.')
+parser.add_argument('--lr-beta', type=float, default=0.5,
+                    help='learning rate for the beta in margin based loss. default is 0.5s.')


0.5s ? what does s stand for here?

ThomasDelteil · 2018-09-11T19:10:00Z

example/gluon/sn-gan/README.md

@@ -0,0 +1,40 @@
+# Spectral Normalization GAN


Please update https://github.com/apache/incubator-mxnet/tree/master/example#deep-learning-examples-in-the-mxnet-project-repository with your example, thanks!

ThomasDelteil · 2018-09-11T19:10:03Z

example/gluon/sn-gan/README.md

+## Learned Spectral Normalization
+
+![alt text](https://github.com/taki0112/Spectral_Normalization-Tensorflow/blob/master/assests/sn.png)
+


Could you add a few samples of the generated images? It always makes it more appealing for people looking to try new models. Thanks!

I am running the model with Xavier initializer and will update the image if it's better

…to sn_gan_example

…t image

stu1130

Thanks @ThomasDelteil

stu1130 · 2018-09-11T23:00:13Z

example/gluon/sn-gan/README.md

+## Learned Spectral Normalization
+
+![alt text](https://github.com/taki0112/Spectral_Normalization-Tensorflow/blob/master/assests/sn.png)
+


I am running the model with Xavier initializer and will update the image if it's better

stu1130 · 2018-09-11T23:20:34Z

example/gluon/sn_gan/model.py

+        """ spectral normalization """
+        w = self.params.get('weight').data(self.ctx)
+        # the w preserve the original weight value to be used in line 75
+        w_mat = w


w is needed to be used for calculation later in the line 75

assignment does not create a copy, reshape creates a copy. you can use:
w_mat = nd.reshape(w, [w.shape[0], -1])

ThomasDelteil · 2018-09-11T23:37:06Z

example/gluon/sn_gan/README.md

+  --epochs EPOCHS       number of training epochs. default is 100.
+  --lr LR               learning rate. default is 0.0001.
+  --lr-beta LR_BETA     learning rate for the beta in margin based loss.
+                        default is 0.5s.


nit: the s is also here still

ThomasDelteil

last minor change

sandeep-krishnamurthy

Thanks, a very useful contribution.

This reverts commit 46a5cee.

* Revert "Removing the re-size for validation data, which breaking the validation accuracy of CIFAR training (#12362)" This reverts commit ceabcaa. * Revert "[MXNET-580] Add SN-GAN example (#12419)" This reverts commit 46a5cee. * Revert "Remove regression checks for website links (#12507)" This reverts commit 619bc3e. * Revert "Revert "Fix flaky test: test_mkldnn.test_activation #12377 (#12418)" (#12516)" This reverts commit 7ea0533. * Revert "further bump up tolerance for sparse dot (#12527)" This reverts commit 90599e1. * Revert "Fix broken URLs (#12508)" This reverts commit 3d83c89. * Revert "Temporarily disable flaky tests (#12520)" This reverts commit 35ca13c. * Revert "Add support for more req patterns for bilinear sampler backward (#12386)" This reverts commit 4ee866f. * Revert "Change the way NDArrayIter handle the last batch (#12285)" This reverts commit 597a637.

* update sn-gan example * fix naming * add more comments * fix naming and refine comments * make power iteration as one hyperparameter * deal with divided by zero problem * replace 0.00000001 with EPSILON * refactor the example * add README * address the feedback * refine the composing * fix the typo, delete the redundant piece of code and update the result image * update folder name to align with others * update image name * add the variable back * remove the redundant piece of code and fix typo

* Revert "Removing the re-size for validation data, which breaking the validation accuracy of CIFAR training (apache#12362)" This reverts commit ceabcaa. * Revert "[MXNET-580] Add SN-GAN example (apache#12419)" This reverts commit 46a5cee. * Revert "Remove regression checks for website links (apache#12507)" This reverts commit 619bc3e. * Revert "Revert "Fix flaky test: test_mkldnn.test_activation apache#12377 (apache#12418)" (apache#12516)" This reverts commit 7ea0533. * Revert "further bump up tolerance for sparse dot (apache#12527)" This reverts commit 90599e1. * Revert "Fix broken URLs (apache#12508)" This reverts commit 3d83c89. * Revert "Temporarily disable flaky tests (apache#12520)" This reverts commit 35ca13c. * Revert "Add support for more req patterns for bilinear sampler backward (apache#12386)" This reverts commit 4ee866f. * Revert "Change the way NDArrayIter handle the last batch (apache#12285)" This reverts commit 597a637.

update sn-gan example

27019bb

stu1130 requested a review from szha as a code owner August 31, 2018 04:26

stu1130 added 3 commits August 30, 2018 22:00

fix naming

3d1601e

add more comments

8113551

fix naming and refine comments

e31ca88

sandeep-krishnamurthy added Gluon Example pr-awaiting-review PR is waiting for code review labels Aug 31, 2018

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

b7e44c6

…to sn_gan_example

vandanavk reviewed Sep 4, 2018

View reviewed changes

stu1130 added 2 commits September 4, 2018 13:07

make power iteration as one hyperparameter

0d1b80f

deal with divided by zero problem

4db5909

stu1130 changed the title ~~[MXNET-580][WIP] Add SN-GAN example~~ [MXNET-580] Add SN-GAN example Sep 4, 2018

replace 0.00000001 with EPSILON

3b944d2

vandanavk reviewed Sep 5, 2018

View reviewed changes

stu1130 added 2 commits September 6, 2018 22:04

refactor the example

b4ca8a3

add README

09e366a

stu1130 changed the title ~~[MXNET-580] Add SN-GAN example~~ [MXNET-580] [WIP] Add SN-GAN example Sep 7, 2018

sandeep-krishnamurthy reviewed Sep 11, 2018

View reviewed changes

stu1130 added 3 commits September 11, 2018 09:15

address the feedback

4085edb

refine the composing

a467c68

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

c3db826

…to sn_gan_example

ThomasDelteil suggested changes Sep 11, 2018

View reviewed changes

stu1130 added 2 commits September 11, 2018 15:55

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

dc982c3

…to sn_gan_example

fix the typo, delete the redundant piece of code and update the resul…

cc77f47

…t image

stu1130 commented Sep 11, 2018

View reviewed changes

stu1130 changed the title ~~[MXNET-580] [WIP] Add SN-GAN example~~ [MXNET-580] Add SN-GAN example Sep 11, 2018

stu1130 added 3 commits September 11, 2018 16:10

update folder name to align with others

f90190e

update image name

bd7a8bc

add the variable back

d7b54fc

stu1130 commented Sep 11, 2018

View reviewed changes

ThomasDelteil reviewed Sep 11, 2018

View reviewed changes

ThomasDelteil approved these changes Sep 11, 2018

View reviewed changes

remove the redundant piece of code and fix typo

4757682

sandeep-krishnamurthy approved these changes Sep 12, 2018

View reviewed changes

sandeep-krishnamurthy merged commit 46a5cee into apache:master Sep 12, 2018

zhreshold added a commit that referenced this pull request Sep 12, 2018

Revert "[MXNET-580] Add SN-GAN example (#12419)"

e838818

This reverts commit 46a5cee.

stu1130 deleted the sn_gan_example branch February 7, 2020 19:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-580] Add SN-GAN example #12419

[MXNET-580] Add SN-GAN example #12419

stu1130 commented Aug 31, 2018 •

edited

Loading

vandanavk Sep 4, 2018

stu1130 Sep 4, 2018

vandanavk Sep 4, 2018

stu1130 Sep 4, 2018

sandeep-krishnamurthy commented Sep 5, 2018

ThomasDelteil commented Sep 5, 2018

vandanavk commented Sep 5, 2018

vandanavk Sep 5, 2018

stu1130 Sep 5, 2018 •

edited

Loading

sandeep-krishnamurthy left a comment

sandeep-krishnamurthy Sep 11, 2018

sandeep-krishnamurthy Sep 11, 2018

sandeep-krishnamurthy Sep 11, 2018

sandeep-krishnamurthy Sep 11, 2018

ThomasDelteil left a comment

ThomasDelteil Sep 11, 2018

ThomasDelteil Sep 11, 2018

ThomasDelteil Sep 11, 2018

ThomasDelteil Sep 11, 2018

ThomasDelteil Sep 11, 2018

ThomasDelteil Sep 11, 2018

stu1130 Sep 11, 2018

stu1130 left a comment

stu1130 Sep 11, 2018

stu1130 Sep 11, 2018

ThomasDelteil Sep 11, 2018

ThomasDelteil Sep 11, 2018

ThomasDelteil left a comment

sandeep-krishnamurthy left a comment

		@@ -0,0 +1,38 @@
		# Spectral Normalization GAN

		This example implements [Spectral Normalization for Generative Adversarial Networks](https://openreview.net/pdf?id=B1QRgziT-) based on [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset.



		def get_training_data(batch_size):
		""" hepler function to get dataloader"""

		## Learned Spectral Normalization

		![alt text](https://github.com/taki0112/Spectral_Normalization-Tensorflow/blob/master/assests/sn.png)

[MXNET-580] Add SN-GAN example #12419

[MXNET-580] Add SN-GAN example #12419

Conversation

stu1130 commented Aug 31, 2018 • edited Loading

Description

Checklist

Essentials

Changes

Comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandeep-krishnamurthy commented Sep 5, 2018

ThomasDelteil commented Sep 5, 2018

vandanavk commented Sep 5, 2018

Choose a reason for hiding this comment

stu1130 Sep 5, 2018 • edited Loading

Choose a reason for hiding this comment

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThomasDelteil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stu1130 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThomasDelteil left a comment

Choose a reason for hiding this comment

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

stu1130 commented Aug 31, 2018 •

edited

Loading

stu1130 Sep 5, 2018 •

edited

Loading