pull

CONTRIBUTING.md

@@ -0,0 +1,68 @@
+# Contributing to NNI
+
+Welcome, and thank you for your interest in contributing to NNI!
+
+There are many ways in which you can contribute, beyond writing code. The goal of this document is to provide a high-level overview of how you can get involved.
+
+# Provide feedback or ask a question
+
+* [File an issue](https://github.com/microsoft/nni/issues/new/choose) on GitHub.
+* Ask a question with NNI tags on [Stack Overflow](https://stackoverflow.com/questions/tagged/nni?sort=Newest&edited=true).
+* Discuss on the NNI [Gitter](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) in NNI.
+
+Join IM discussion groups:
+|Gitter||WeChat|
+|----|----|----|
+|![image](https://user-images.githubusercontent.com/39592018/80665738-e0574a80-8acc-11ea-91bc-0836dc4cbf89.png)| OR |![image](https://github.com/scarlett2018/nniutil/raw/master/wechat.png)|
+
+
+# Look for an existing issue
+Before you create a new issue, please do a search in [open issues](https://github.com/microsoft/nni/issues) to see if the issue or feature request has already been filed.
+
+Be sure to scan through the [most popular](https://github.com/microsoft/nni/issues?q=is%3Aopen+is%3Aissue+label%3AFAQ+sort%3Areactions-%2B1-desc) feature requests.
+
+If you find your issue already exists, make relevant comments and add your [reaction](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments). Use a reaction in place of a "+1" comment:
+
+* 👍 - upvote
+* 👎 - downvote
+
+If you cannot find an existing issue that describes your bug or feature, create a new issue using the guidelines below.
+
+# Writing good bug reports or feature requests
+File a single issue per problem and feature request. Do not enumerate multiple bugs or feature requests in the same issue.
+
+Provide as many information as you think might relevant to the context (thinking the issue is assigning to you, what kinds of info you will need to debug it!!!). To give you a general idea about what kinds of info are useful for developers to dig out the issue, we had provided issue template for you.
+
+Once you had submitted an issue, be sure to follow it for questions and discussions. 
+
+Once the bug is fixed or feature is addressed, be sure to close the issue.
+
+# Contributing fixes or examples
+
+This project welcomes contributions and suggestions. Most contributions require you to agree to a
+Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
+
+When you submit a pull request, a CLA bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+# Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
+contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
+
+# How to Contribute
+
+After getting familiar with contribution agreements, you are ready to create your first PR =), follow the NNI developer tutorials to get start:
+
+* We recommend new contributors to start with simple issues: ['good first issue'](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) or ['help-wanted'](https://github.com/microsoft/nni/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22).
+* [NNI developer environment installation tutorial](docs/en_US/Tutorial/SetupNniDeveloperEnvironment.md)
+* [How to debug](docs/en_US/Tutorial/HowToDebug.md)
+* If you have any questions on usage, review [FAQ](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/FAQ.md) first, if there are no relevant issues and answers to your question, try contact NNI dev team and users in [Gitter](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) or [File an issue](https://github.com/microsoft/nni/issues/new/choose) on GitHub.
+* [Customize your own Tuner](docs/en_US/Tuner/CustomizeTuner.md)
+* [Implement customized TrainingService](docs/en_US/TrainingService/HowToImplementTrainingService.md)
+* [Implement a new NAS trainer on NNI](docs/en_US/NAS/Advanced.md)
+* [Customize your own Advisor](docs/en_US/Tuner/CustomizeAdvisor.md)
+

docs/en_US/Tuner/BuiltinTuner.md

@@ -247,6 +247,7 @@ This is suggested when you have limited computational resources but have a relat
 * **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
 * **R** (*int, optional, default = 60*) - the maximum budget given to a trial (could be the number of mini-batches or epochs). Each trial should use TRIAL_BUDGET to control how long they run.
 * **eta** (*int, optional, default = 3*) - `(eta-1)/eta` is the proportion of discarded trials.
+* **exec_mode** (*serial or parallelism, optional, default = parallelism*) - If 'parallelism', the tuner will try to use available resources to start new bucket immediately. If 'serial', the tuner will only start new bucket after the current bucket is done.
 
 **Example Configuration:**
 

docs/en_US/Tuner/HyperbandAdvisor.md

@@ -7,7 +7,17 @@ Hyperband on NNI
 ## 2. Implementation with full parallelism
 First, this is an example of how to write an autoML algorithm based on MsgDispatcherBase, rather than Tuner and Assessor. Hyperband is implemented in this way because it integrates the functions of both Tuner and Assessor, thus, we call it Advisor.
 
-Second, this implementation fully leverages Hyperband's internal parallelism. Specifically, the next bucket is not started strictly after the current bucket. Instead, it starts when there are available resources.
+Second, this implementation fully leverages Hyperband's internal parallelism. Specifically, the next bucket is not started strictly after the current bucket. Instead, it starts when there are available resources. If you want to use full parallelism mode, set `exec_mode` with `parallelism`. 
+
+Or if you want to set `exec_mode` with `serial` according to the original algorithm. In this mode, the next bucket will start strictly after the current bucket.
+
+`parallelism` mode may lead to multiple unfinished buckets, and there is at most one unfinished bucket under `serial` mode. The advantage of `parallelism` mode is to make full use of resources, which may reduce the experiment duration multiple times. The following two pictures are the results of quick verification using [nas-bench-201](../NAS/Benchmarks.md), picture above is in `parallelism` mode, picture below is in `serial` mode.
+
+![parallelism mode](../../img/hyperband_parallelism.png "parallelism mode")
+
+![serial mode](../../img/hyperband_serial.png "serial mode")
+
+If you want to reproduce these results, refer to the example under `examples/trials/benchmarking/` for details.
 
 ## 3. Usage
 To use Hyperband, you should add the following spec in your experiment's YAML config file:
@@ -23,6 +33,8 @@ advisor:
     eta: 3
     #choice: maximize, minimize
     optimize_mode: maximize
+    #choice: serial, parallelism
+    exec_mode: parallelism
 ```
 
 Note that once you use Advisor, you are not allowed to add a Tuner and Assessor spec in the config file. If you use Hyperband, among the hyperparameters (i.e., key-value pairs) received by a trial, there will be one more key called `TRIAL_BUDGET` defined by user. **By using this `TRIAL_BUDGET`, the trial can control how long it runs**.

examples/trials/benchmarking/config_hyperband.yml

@@ -0,0 +1,27 @@
+authorName: default
+experimentName: example_mnist_hyperband
+trialConcurrency: 2
+maxExecDuration: 100h
+maxTrialNum: 10000
+#choice: local, remote, pai
+trainingServicePlatform: local
+searchSpacePath: search_space.json
+#choice: true, false
+useAnnotation: false
+advisor:
+  #choice: Hyperband, BOHB
+  builtinAdvisorName: Hyperband
+  classArgs:
+    #R: the maximum trial budget (could be the number of mini-batches or epochs) can be
+    #   allocated to a trial. Each trial should use trial budget to control how long it runs.
+    R: 60
+    #eta: proportion of discarded trials
+    eta: 3
+    #choice: maximize, minimize
+    optimize_mode: maximize
+    #choice: serial, parallelism
+    exec_mode: serial
+trial:
+  command: python3 main.py
+  codeDir: .
+  gpuNum: 0

examples/trials/benchmarking/main.py

@@ -0,0 +1,52 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+"""
+A test for hyperband, using nasbench201. So it need install the dependencies for nasbench201 at first.
+"""
+import argparse
+import logging
+import random
+import time
+
+import nni
+from nni.utils import merge_parameter
+from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats
+
+
+logger = logging.getLogger('test_hyperband')
+
+
+def main(args):
+    r = args.pop('TRIAL_BUDGET')
+    dataset = [t for t in query_nb201_trial_stats(args, 200, 'cifar100', include_intermediates=True)]
+    test_acc = random.choice(dataset)['intermediates'][r - 1]['ori_test_acc'] / 100
+    time.sleep(random.randint(0, 10))
+    nni.report_final_result(test_acc)
+    logger.debug('Final result is %g', test_acc)
+    logger.debug('Send final result done.')
+
+def get_params():
+    parser = argparse.ArgumentParser(description='Hyperband Test')
+    parser.add_argument("--0_1", type=str, default='none')
+    parser.add_argument("--0_2", type=str, default='none')
+    parser.add_argument("--0_3", type=str, default='none')
+    parser.add_argument("--1_2", type=str, default='none')
+    parser.add_argument("--1_3", type=str, default='none')
+    parser.add_argument("--2_3", type=str, default='none')
+    parser.add_argument("--TRIAL_BUDGET", type=int, default=200)
+
+    args, _ = parser.parse_known_args()
+    return args
+
+if __name__ == '__main__':
+    try:
+        # get parameters form tuner
+        tuner_params = nni.get_next_parameter()
+        logger.debug(tuner_params)
+        params = vars(merge_parameter(get_params(), tuner_params))
+        print(params)
+        main(params)
+    except Exception as exception:
+        logger.exception(exception)
+        raise

examples/trials/benchmarking/search_space.json

@@ -0,0 +1,8 @@
+{
+    "0_1": {"_type": "choice", "_value": ["none", "skip_connect", "conv_1x1", "conv_3x3", "avg_pool_3x3"]},
+    "0_2": {"_type": "choice", "_value": ["none", "skip_connect", "conv_1x1", "conv_3x3", "avg_pool_3x3"]},
+    "0_3": {"_type": "choice", "_value": ["none", "skip_connect", "conv_1x1", "conv_3x3", "avg_pool_3x3"]},
+    "1_2": {"_type": "choice", "_value": ["none", "skip_connect", "conv_1x1", "conv_3x3", "avg_pool_3x3"]},
+    "1_3": {"_type": "choice", "_value": ["none", "skip_connect", "conv_1x1", "conv_3x3", "avg_pool_3x3"]},
+    "2_3": {"_type": "choice", "_value": ["none", "skip_connect", "conv_1x1", "conv_3x3", "avg_pool_3x3"]}
+}

examples/trials/mnist-advisor/config_hyperband.yml

@@ -19,6 +19,8 @@ advisor:
     eta: 3
     #choice: maximize, minimize
     optimize_mode: maximize
+    #choice: serial, parallelism
+    exec_mode: parallelism
 trial:
   command: python3 mnist.py
   codeDir: .

examples/trials/mnist-advisor/config_pai.yml

@@ -19,6 +19,8 @@ advisor:
     eta: 3
     #choice: maximize, minimize
     optimize_mode: maximize
+    #choice: serial, parallelism
+    exec_mode: parallelism
 trial:
   command: python3 mnist.py
   codeDir: .

examples/trials/mnist-advisor/config_paiYarn.yml

@@ -19,6 +19,8 @@ advisor:
     eta: 3
     #choice: maximize, minimize
     optimize_mode: maximize
+    #choice: serial, parallelism
+    exec_mode: parallelism
 trial:
   command: python3 mnist.py
   codeDir: .

examples/trials/systems/rocksdb-fillrandom/config_smac.yml → examples/trials/systems_auto_tuning/rocksdb-fillrandom/config_smac.yml

@@ -1,21 +1,21 @@
-authorName: default
-experimentName: auto_rocksdb_SMAC
-trialConcurrency: 1
-maxExecDuration: 12h
-maxTrialNum: 256
-#choice: local, remote, pai
-trainingServicePlatform: local
-searchSpacePath: search_space.json
-#choice: true, false
-useAnnotation: false
-tuner:
-  #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner
-  #SMAC (SMAC should be installed through nnictl)
-  builtinTunerName: SMAC
-  classArgs:
-    #choice: maximize, minimize
-    optimize_mode: maximize
-trial:
-  command: python3 main.py
-  codeDir: .
-  gpuNum: 0
+authorName: default
+experimentName: auto_rocksdb_SMAC
+trialConcurrency: 1
+maxExecDuration: 12h
+maxTrialNum: 256
+#choice: local, remote, pai
+trainingServicePlatform: local
+searchSpacePath: search_space.json
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner
+  #SMAC (SMAC should be installed through nnictl)
+  builtinTunerName: SMAC
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+trial:
+  command: python3 main.py
+  codeDir: .
+  gpuNum: 0

examples/trials/systems/rocksdb-fillrandom/config_tpe.yml → examples/trials/systems_auto_tuning/rocksdb-fillrandom/config_tpe.yml

@@ -1,21 +1,21 @@
-authorName: default
-experimentName: auto_rocksdb_TPE
-trialConcurrency: 1
-maxExecDuration: 12h
-maxTrialNum: 256
-#choice: local, remote, pai
-trainingServicePlatform: local
-searchSpacePath: search_space.json
-#choice: true, false
-useAnnotation: false
-tuner:
-  #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner
-  #SMAC (SMAC should be installed through nnictl)
-  builtinTunerName: TPE
-  classArgs:
-    #choice: maximize, minimize
-    optimize_mode: maximize
-trial:
-  command: python3 main.py
-  codeDir: .
-  gpuNum: 0
+authorName: default
+experimentName: auto_rocksdb_TPE
+trialConcurrency: 1
+maxExecDuration: 12h
+maxTrialNum: 256
+#choice: local, remote, pai
+trainingServicePlatform: local
+searchSpacePath: search_space.json
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner
+  #SMAC (SMAC should be installed through nnictl)
+  builtinTunerName: TPE
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+trial:
+  command: python3 main.py
+  codeDir: .
+  gpuNum: 0

examples/trials/systems/rocksdb-fillrandom/search_space.json → examples/trials/systems_auto_tuning/rocksdb-fillrandom/search_space.json

@@ -1,14 +1,14 @@
-{
-    "write_buffer_size": {
-        "_type": "quniform",
-        "_value": [2097152, 16777216, 1048576]
-    },
-    "min_write_buffer_number_to_merge": {
-        "_type": "quniform",
-        "_value": [2, 16, 1]
-    },
-    "level0_file_num_compaction_trigger": {
-        "_type": "quniform",
-        "_value": [2, 16, 1]
-    }
-}
+{
+    "write_buffer_size": {
+        "_type": "quniform",
+        "_value": [2097152, 16777216, 1048576]
+    },
+    "min_write_buffer_number_to_merge": {
+        "_type": "quniform",
+        "_value": [2, 16, 1]
+    },
+    "level0_file_num_compaction_trigger": {
+        "_type": "quniform",
+        "_value": [2, 16, 1]
+    }
+}