[MXNET-854] SVRG Optimization in Python Module API #12376

StephanieYuan · 2018-08-27T22:12:08Z

Description

SVRG stands for Stochastic Variance Reduced Gradient, which is an optimization technique that complements SGD. It is introduced in the paper Accelerating Stochastic Gradient Descent using Predicative Variance Reduction in 2013.
Key Characteristics of SVRG:

Explicit Variance Reduction
Converge much faster compared to SGD on strongly convex smooth functions

SVRG Module should:

seamlessly supports both dense and sparse data, run on CPU and GPU instances on single machine and in distributed setting.
minimize the API differences with Python Module API.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-854], where MXNET-854 refers to the relevant JIRA issue created (except PRs with tiny changes)
All changes have test coverage:
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

SVRGModule inherits from Python Module and minimizes the API changes of Module API. Both intermediate level and high level APIs are supported
Functional tests and unittests have been tested for SVRGModule and SVRG Optimizer.

Comments

Design decisions of why using Python Module instead of Optimizer can be found in the design doc here
This version of SVRGModule supports single machine run with single cpu, single gpu and multiple-gpus. Distributed SVRGModule is soon to be updated.
Benchmarking has been performed on this single machine version SVRGModule on single cpu on linear regression model. Distributed SVRGModule to be updated soon. Benchmarking results can be found in contrib/svrg_optimization_python/benchmarks.

sandeep-krishnamurthy · 2018-08-28T17:32:14Z

Thanks @StephanieYuan - Welcome to MXNet community!

@piiswrong @eric-haibin-lin @anirudhacharya @Roshrini @vandanavk - You will be interested to have a look at this.

sandeep-krishnamurthy

Few high level comments for now. Will dive deep by building this change and playing around with MNIST.

Thanks for doing the benchmarks and adding the results. Looks promising and very useful.

It will be very helpful, if we can benchmark on MNIST as depicted in the paper, and see if we can achieve the results in paper, to verify correctness of the implementation.

sandeep-krishnamurthy · 2018-08-28T17:34:24Z

contrib/svrg_optimization_python/src/svrg_module.py

+    Examples
+    --------
+    >>> # An example of declaring and using SVRGModule.
+    >>> mod = mod = SVRGModule(symbol=lro, data_names=['data'], label_names=['lin_reg_label'], update_freq=2)


Please correct this example.

sandeep-krishnamurthy · 2018-08-28T17:36:57Z

contrib/svrg_optimization_python/src/svrg_module.py

+        if isinstance(update_freq, int):
+            self.update_freq = update_freq
+        else:
+            raise TypeError("update_freq must be an integer")


Can you make error more useful for user to understand the issue. Example: (update_freq in SVRGModule must be an integer. Example: 2. Given update_freq = ', update_freq)

sandeep-krishnamurthy · 2018-08-28T17:37:39Z

contrib/svrg_optimization_python/src/svrg_module.py

+        """Internal function to reset binded state."""
+        super(SVRGModule, self)._reset_bind()
+        self._mod_aux._reset_bind()
+


nit: extra spaces?

sandeep-krishnamurthy · 2018-08-28T17:54:04Z

contrib/svrg_optimization_python/src/svrg_module.py

+                self.logger.info('Epoch[%d] Train-%s=%f', epoch, name, val)
+            toc = time.time()
+            self.logger.info('Epoch[%d] Time cost=%.3f', epoch, (toc - tic))
+            print('Epoch[%d] Time cost=%.3f', epoch, eval_metric.get())


please use logger instead of print every where. example self.logger.info

sandeep-krishnamurthy · 2018-08-28T17:55:10Z

contrib/svrg_optimization_python/src/svrg_optimizer.py

+
+@mx.optimizer.register
+class SVRGOptimizer(mx.optimizer.Optimizer):
+    """SVRGOptimizer is a wrapper class for two optimizers: one for accumulating full gradients and the other


This is mainly useful with SGD right? Why allow any optimizer to be passed in here?

For the previous benchmarks SVRG is used with SGD. Meanwhile I did some exploration of using NAG + SVRG and Adam + SVRG. I think it will be valuable to benchmark those optimizers with SVRG too.

sandeep-krishnamurthy · 2018-08-28T17:58:17Z

contrib/svrg_optimization_python/test_svrg_train.py

@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one


tests should go tests. May be like below?

tests/python/train/test_contrib_svrg.py : Full end to end train and testing. See test_mlp.py as an example. Please, note, it should be faster (< 2 min at max?). If it is longer than that, we need to move it to nightly test.

tests/python/unittest/test_contrib_svrgmodule.py / ..._svrgoptimizer.py etc..

sandeep-krishnamurthy · 2018-08-28T18:01:48Z

contrib/svrg_optimization_python/test_svrg_train.py

+            mod.forward_backward(data_batch=batch)
+            mod.update()
+            mod.update_metric(metrics, batch.label)
+        print('Epoch[%d] Time cost=%.3f', e, metrics.get())


Please use logging.

sandeep-krishnamurthy · 2018-08-28T18:08:36Z

contrib/svrg_optimization_python/test_svrg_train.py

+    :return: an instance of mx.io.NDArrayIter
+    :return: an instance of mx.mod.svrgmodule for performing SVRG optimization
+    """
+    mx.random.seed(42)


Please do not use fixed seed. You can see other tests under tests/

Also, what are we asserting here?

sandeep-krishnamurthy · 2018-08-28T18:09:47Z

contrib/svrg_optimization_python/test_svrg_train.py

+    return di, mod
+
+# run as a script
+if __name__ == "__main__":


should it be? Please see other tests.

if __name__ == '__main__': import nose nose.runmodule()

StephanieYuan · 2018-08-29T02:06:53Z

Moved svrg_optimization to python/contrib package.

xcgoner · 2018-08-29T16:22:22Z

@StephanieYuan The original version of SVRG could be time-consuming due to the computation of the full gradient at the beginning of each epoch. Could you also include the cheap version in your implementation:
https://arxiv.org/abs/1511.01942

vandanavk

In general, please add documentation for the input params for functions and a description of return variables wherever missing.

Please execute pylint on these files using ci/other/pylintrc in incubator-mxnet folder.
Something like this
pylint --rcfile=ci/other/pylintrc --ignore-patterns="..so$$,..dll$$,..dylib$$" python/mxnet/contrib/svrg_optimization/.py example/svrg_module/.py tests/python/unittest/test_contrib_svrg

python/mxnet/contrib/svrg_optimization/svrg_optimizer.py

tests/python/unittest/test_contrib_svrg_optimizer.py

example/svrg_module/train.py

vandanavk · 2018-08-29T17:28:56Z

example/svrg_module/train.py

+parser.add_argument('-b', dest="batch_size", help="define the batch size for training", type=int,
+                    default=100, required=False)
+parser.add_argument('-m', dest='metrics', help="create eval metric", type=str, required=False)
+parser.add_argument('--gpus', type=str, help='list of gpus to run, e.g. 0 or 0,2,5. empty means using cpu')


Default value?

Default is mx.cpu() as defined in line 35.

tests/python/unittest/test_contrib_svrg_module.py

tests/python/unittest/test_contrib_svrg_optimizer.py

StephanieYuan · 2018-08-31T02:29:10Z

@xcgoner Thank you for the pointer! I'd like to try out the improvements in the later iterations of this project.

anirudhacharya

Can you update the API docs here - module.md and optimization.md with references to the SVRG module.

example/svrg_module/api_usage_example/example_inference.py

anirudhacharya · 2018-08-31T08:12:30Z

python/mxnet/contrib/svrg_optimization/svrg_optimizer.py

+
+@mx.optimizer.register
+class SVRGOptimizer(mx.optimizer.Optimizer):
+    """SVRGOptimizer is a wrapper class for two optimizers: self.aux_opt for accumulating full gradients and


Please write the doc string in the following format to maintain uniformity - https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/optimizer.py#L445-L488

tests/python/unittest/test_contrib_svrg_module.py

anirudhacharya · 2018-08-31T16:04:35Z

tests/python/unittest/test_contrib_svrg_module.py

+import numpy as np
+
+
+def set_up():


SVRGModule lacks test coverage. Can you add tests for module.save_checkpoint, module.load, module.forward, module.predict.

tests/python/unittest/test_contrib_svrg_optimizer.py

anirudhacharya · 2018-08-31T19:30:48Z

python/mxnet/contrib/svrg_optimization/README.md

+## SVRG Optimization in Python Module
+
+### Problem Description
+SVRG stands for Stochastic Variance Reduced Gradient, which was first introduced in the paper _Accelerating Stochastic 


This README should be removed from here and the SVRG description and characteristics should go into the documentation of the SVRG optimizer.

anirudhacharya · 2018-08-31T19:31:27Z

python/mxnet/contrib/svrg_optimization/README.md

+* test_svrg_module.py: unittests for SVRGModule API
+* test_svrg_optimizer.py: unittests for SVRGOptimizer API
+
+### Examples:


add these in the README of the examples folder.

anirudhacharya · 2018-08-31T19:32:40Z

python/mxnet/contrib/svrg_optimization/README.md

+* examples/svrg_module/example_api_train.py: a demo script for training a SVRGModule using intermediate level api and high level api
+* examples/svrg_module/example_inference.py: a demo script for SVRGModule inference
+
+#### Benchmarks:


as the benchmark graphs are present in the "examples" folder, move these lines to the README in the examples folder.

anirudhacharya · 2018-08-31T19:33:02Z

python/mxnet/contrib/svrg_optimization/README.md

+
+#### Testing:
+
+Unit Tests:


remove these lines about the Unit Tests.

anirudhacharya · 2018-08-31T19:34:22Z

example/svrg_module/linear_regression/README.md

+data_reader.py to download the data. 
+
+This example trains a linear regression model using SVRGModule. Logs of the training results can be found in 
+experiments.log


where is experiments.log?

It will be generated when runs the train script.

anirudhacharya

Run the following command to get the coverage results nosetests -v --with-coverage tests/python/unittest/test_contrib_svrg_*

mxnet/contrib/svrg_optimization/__init__.py                  3      0   100%
mxnet/contrib/svrg_optimization/svrg_module.py             175     92    47%
mxnet/contrib/svrg_optimization/svrg_optimizer.py           38      1    97%

If you notice svrg_module.py coverage is deficient. Please increase the coverage for that file, for example update_full_grads method is not tested and you have overriden the fit method of the base class Module, please add tests for them.

Also I am still not sure if the organization of the documentation for this new module is right. Marking @aaronmarkham to review the doc files.

anirudhacharya · 2018-09-04T21:33:53Z

tests/python/unittest/test_module.py

@@ -168,6 +168,7 @@ def test_module_layout():
    assert mod.get_outputs()[0].shape == dshape

    hdshape = (3, 4, 7)
+


nit: unnecessary blank line

aaronmarkham

Without trying it out, I just have a couple of suggestions for updates.
To be clear, at a minimum, you need to update api/python/index.md and add this module in the contrib section.

aaronmarkham · 2018-09-04T22:27:59Z

example/svrg_module/README.md

@@ -0,0 +1,26 @@
+## SVRG Module Example
+


Introduce what it is as well. From your other document:
"SVRGModule is an extension to the Module API that implements SVRG (Stochastic Variance Reduced Gradients) optimization ..."

aaronmarkham · 2018-09-04T22:31:48Z

example/svrg_module/README.md

+## SVRG Module Example
+
+#### API Usage Example
+SVRGModule provides both high-level and intermediate-level APIs while minimizing the changes with Module API.  


Link back to the docs. This should link back to this spot - where you should add an entry for this module.
https://github.com/apache/incubator-mxnet/blob/master/docs/api/python/index.md#contrib-package
Since this comes from the example folder and outside of the docs, use a full static link.

anirudhacharya · 2018-09-04T23:16:00Z

tests/python/unittest/save_000800.json

@@ -1,188 +0,0 @@
-{


Why delete this file and legacy_ndarray.v0?

It was done by mistake, I'll add them back when I push for more svrg_module test coverage. Thank you for the heads up!

anirudhacharya · 2018-09-05T00:33:09Z

@StephanieYuan Also the way svrg_optimizer.py is written it leaves a few questions open.

From your design document I understand that SVRGOptimizer cannot be used with Module API which is why you wrote the SVRGModule API which will use SVRGOptimizer under the hood instead of the usual SGD or other optimizers. Can you explain why in greater detail?

Also if this is indeed the case, then the SVRGOptimizer class needs to be declared private using a leading underscore in its name, else it will be very misleading to the user who might try to use SVRGOptimizer with the plain vanilla Module API. Also it should be made clear in the doc string for the SRVGOptimizer that this optimizer is to be used with the SVRGModule alone.

Also you might want to consider moving the SVRGOptimizer to the optimizer.py file and keep SVRGModule alone in the contrib folder. But I am not sure of this, you might want to get a second opinion on this. @piiswrong @eric-haibin-lin

sandeep-krishnamurthy · 2018-09-05T05:10:22Z

Great progress here. Thanks everyone.
@StephanieYuan

Please rebase and resolve conflicts.
Please comment about tests added and coverage.
We had discussed in above comment, to verify this implementation with MNIST to confirm correctness per the claims in paper. Please comment.

@anirudhacharya - I would still prefer keeping SVRG optimizer in contrib and allow it to evolve. As mentioned in above comments and design doc, Other combination like Adam+SVRG could also be used but not benchmarked. I would set the default optimizer for SVRGModule to be SVRGOptimizer but not remove the option altogether and make SVRGOptimizer internal. Thoughts?

StephanieYuan · 2018-09-05T14:11:46Z

Thanks everyone for the feedback and input!

I've rebased and resolved the conflict with master. I have also added some more unittests for SVRGModule especially on verifying SVRG gradients calculations and the test coverage is now:
mxnet/contrib/svrg_optimization/svrg_module.py 175 21 88%
mxnet/contrib/svrg_optimization/svrg_optimizer.py 38 1 97%

@sandeep-krishnamurthy I'm working on creating the MNIST example but I'm expecting it will take a few more days as I have other tasks in queue. Maybe I can include the example in my next PR on SVRG in distributed mode?

@anirudhacharya SVRGOptimizer is designed to be used with SVRGModule alone. The actual SVRG optimization logic is implemented in the SVRGModule and the SVRGOptimizer is designed to assist the full gradients accumulation in the KVStore. I've made SVRGOptimizer a private class and added the reasonings behind the design in greater detail in the docs/api/python/contrib/svrg_optimization. Hopefully it clarifies things a bit.

@aaronmarkham I've added pointer about the svrg_optimization in index.md and refined the docs in the example/svrg_module/README.

sandeep-krishnamurthy

LGTM. Left 2 minor comments, please fix. Thank you for your contributions.

This PR is ready to go after approval from @aaronmarkham for documentation and @anirudhacharya for his code review comments.

sandeep-krishnamurthy · 2018-09-05T16:13:37Z

tests/python/unittest/test_contrib_svrg_module.py

+    mod.init_params()
+    mod.init_optimizer(optimizer='sgd', optimizer_params={'learning_rate': 0.1})
+    mod.update()
+    mod.save_checkpoint('test', 0, save_optimizer_states=True)


Recommended to use tempfile. This avoid race condition in file access from other tests and also clean up is easy. Something like:

import tempfile import os .... tmp = tempfile.mkdtemp() tmpfile = os.path.join(tmp, 'svrg_test') mod.save_checkpoint(tmpfile, .....)

sandeep-krishnamurthy · 2018-09-05T16:14:55Z

tests/python/unittest/test_module.py

@@ -815,4 +815,4 @@ def test_forward_types():

 if __name__ == '__main__':
    import nose
-    nose.runmodule()
+    nose.runmodule()


nit: add a new line (pep)

anirudhacharya · 2018-09-05T17:33:11Z

@sandeep-krishnamurthy it now looks better after SVRGOptimizer was made private. Including SVRGOptimizer in optimizer.py file instead of the contrib folder was just a thought. I am okay either ways. Will take a look at @StephanieYuan 's changes and approve it accordingly.

anirudhacharya

This is how the documentation for this module renders right now - http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-12376/17/api/python/contrib/svrg_optimization.html
Private members like _SVRGOptimizer should not show up in the documentation

http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-12376/17/api/python/module/module.html#module-api

The description for the SVRGModule is not rendering. cc: @aaronmarkham

anirudhacharya

please address the comments on svrg_optimization.md file

StephanieYuan · 2018-09-05T18:13:34Z

@anirudhacharya I'll fix it soon. Is there a way that I can see the rendered view before it gets pushed?

anirudhacharya · 2018-09-05T18:26:29Z

@StephanieYuan some of the issues are with the content of the docs and few are with how the content renders. @aaronmarkham might be able to help with how the docs renders. You will probably have to build the docs and have to deploy it locally.

aaronmarkham · 2018-09-05T21:12:34Z

I just updated the how-to guide for building the docs.
https://cwiki.apache.org/confluence/display/MXNET/How+to+Build+the+Website
Let me know if you have any questions.

aaronmarkham · 2018-09-05T21:23:27Z

I'm not sure why those private members are showing up. Sphinx should recognize them as such if they're defined that way.
Look at http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html and maybe you can use :members or similar to limit what's displayed.
If you get your paths to be specific and accurate (without shortcuts) you should see the description populate assuming you have one. I noticed with ONNX that while mxnet.onnx.import_model might work in Python, it didn't work with Sphinx. The full path was required for the docs to generate properly mxnet.contrib.onnx.onnx2mx.import_model. I'm not sure why though.

Be sure to adjust settings.ini when testing the docs build, otherwise you'll be waiting forever each time while scala builds.

StephanieYuan · 2018-09-05T22:26:48Z

@aaronmarkham Thanks for the pointer!
I'm thinking to exclude SVRGModule from module.md since SVRGModule will be a contrib package at current stage.

python/mxnet/contrib/svrg_optimization/svrg_module.py

anirudhacharya

in the examples folder could you please add the code you used to generate the benchmarking graphs for the linear regression example. Since you are including the graphs in the examples folder, you should also have the code that can reproduce those results.

StephanieYuan · 2018-09-07T20:58:00Z

@anirudhacharya Sounds good! It's a Jupyter Notebook. I'll add it to the example folder.

…c. This version supports single machine SVRG with single cpu, gpu and multi-gpus.

anirudhacharya

LGTM

sandeep-krishnamurthy

Thanks for awesome work.
Having docs, example jupyter notebook, good test coverage is super useful.

…he#12376) Implemented a python SVRGModule for performing SVRG Optimization Logic. This version supports single machine SVRG with single cpu, gpu and multi-gpus. (apache#12376)

sandeep-krishnamurthy added Python pr-awaiting-review PR is waiting for code review labels Aug 28, 2018

sandeep-krishnamurthy reviewed Aug 28, 2018

View reviewed changes

StephanieYuan requested a review from szha as a code owner August 29, 2018 01:48

StephanieYuan force-pushed the SVRGOptimization branch from fc59615 to 9ecc673 Compare August 29, 2018 02:06

StephanieYuan force-pushed the SVRGOptimization branch 3 times, most recently from 495be3b to a107e3b Compare August 29, 2018 05:42

vandanavk reviewed Aug 29, 2018

View reviewed changes

StephanieYuan force-pushed the SVRGOptimization branch from a107e3b to b712796 Compare August 30, 2018 19:48

anirudhacharya reviewed Aug 31, 2018

View reviewed changes

anirudhacharya suggested changes Aug 31, 2018

View reviewed changes

StephanieYuan force-pushed the SVRGOptimization branch 3 times, most recently from 2f7b40d to 4a2b644 Compare September 4, 2018 21:47

anirudhacharya suggested changes Sep 4, 2018

View reviewed changes

aaronmarkham suggested changes Sep 4, 2018

View reviewed changes

anirudhacharya reviewed Sep 4, 2018

View reviewed changes

StephanieYuan force-pushed the SVRGOptimization branch 2 times, most recently from fa1753c to bb31363 Compare September 5, 2018 04:33

StephanieYuan force-pushed the SVRGOptimization branch 2 times, most recently from 4c48130 to c446dc0 Compare September 5, 2018 13:51

sandeep-krishnamurthy approved these changes Sep 5, 2018

View reviewed changes

aaronmarkham approved these changes Sep 5, 2018

View reviewed changes

anirudhacharya approved these changes Sep 5, 2018

View reviewed changes

anirudhacharya reviewed Sep 5, 2018

View reviewed changes

anirudhacharya suggested changes Sep 5, 2018

View reviewed changes

StephanieYuan force-pushed the SVRGOptimization branch 3 times, most recently from c6842a0 to d6e33f1 Compare September 5, 2018 20:57

StephanieYuan force-pushed the SVRGOptimization branch 2 times, most recently from dfddae1 to 3887fe9 Compare September 6, 2018 00:50

anirudhacharya suggested changes Sep 6, 2018

View reviewed changes

python/mxnet/contrib/svrg_optimization/svrg_module.py Outdated Show resolved Hide resolved

python/mxnet/contrib/svrg_optimization/svrg_module.py Outdated Show resolved Hide resolved

StephanieYuan force-pushed the SVRGOptimization branch 2 times, most recently from e39bf63 to 49220e0 Compare September 7, 2018 15:41

anirudhacharya reviewed Sep 7, 2018

View reviewed changes

StephanieYuan force-pushed the SVRGOptimization branch from 49220e0 to 4028bdf Compare September 8, 2018 20:19

Implemented a python SVRGModule for performing SVRG Optimization Logi…

cd54f73

…c. This version supports single machine SVRG with single cpu, gpu and multi-gpus.

StephanieYuan force-pushed the SVRGOptimization branch from 4028bdf to cd54f73 Compare September 8, 2018 20:20

anirudhacharya approved these changes Sep 10, 2018

View reviewed changes

sandeep-krishnamurthy approved these changes Sep 10, 2018

View reviewed changes

sandeep-krishnamurthy merged commit 3406845 into apache:master Sep 10, 2018

		@@ -0,0 +1,84 @@
		# Licensed to the Apache Software Foundation (ASF) under one

		@@ -168,6 +168,7 @@ def test_module_layout():
		assert mod.get_outputs()[0].shape == dshape

		hdshape = (3, 4, 7)

[MXNET-854] SVRG Optimization in Python Module API #12376

[MXNET-854] SVRG Optimization in Python Module API #12376

Conversation

StephanieYuan commented Aug 27, 2018 • edited Loading

Description

Checklist

Essentials

Changes

Comments

sandeep-krishnamurthy commented Aug 28, 2018

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StephanieYuan Aug 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StephanieYuan commented Aug 29, 2018 • edited Loading

xcgoner commented Aug 29, 2018

vandanavk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StephanieYuan commented Aug 31, 2018

anirudhacharya left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhacharya Aug 31, 2018 • edited Loading

Choose a reason for hiding this comment

anirudhacharya Aug 31, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhacharya left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronmarkham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhacharya commented Sep 5, 2018 • edited Loading

sandeep-krishnamurthy commented Sep 5, 2018

StephanieYuan commented Sep 5, 2018

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhacharya commented Sep 5, 2018

anirudhacharya left a comment

Choose a reason for hiding this comment

anirudhacharya left a comment • edited Loading

Choose a reason for hiding this comment

StephanieYuan commented Sep 5, 2018

anirudhacharya commented Sep 5, 2018

aaronmarkham commented Sep 5, 2018

aaronmarkham commented Sep 5, 2018

StephanieYuan commented Sep 5, 2018

anirudhacharya left a comment

Choose a reason for hiding this comment

StephanieYuan commented Sep 7, 2018

anirudhacharya left a comment

Choose a reason for hiding this comment

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

StephanieYuan commented Aug 27, 2018 •

edited

Loading

StephanieYuan Aug 28, 2018 •

edited

Loading

StephanieYuan commented Aug 29, 2018 •

edited

Loading

anirudhacharya left a comment •

edited

Loading

anirudhacharya Aug 31, 2018 •

edited

Loading

anirudhacharya Aug 31, 2018 •

edited

Loading

anirudhacharya left a comment •

edited

Loading

anirudhacharya commented Sep 5, 2018 •

edited

Loading

anirudhacharya left a comment •

edited

Loading