Merge pull request #267 from microsoft/master

merge master

Author: SparkSnail

.gitignore

@@ -81,6 +81,3 @@ venv.bak/
 # VSCode
 .vscode
 .vs
-
-# In case you place source code in ~/nni/
-/experiments

README.md

@@ -215,7 +215,7 @@ Within the following table, we summarized the current NNI capabilities, we are g
 
 ### **Install**
 
-NNI supports and is tested on Ubuntu >= 16.04, macOS >= 10.14.1, and Windows 10 >= 1809. Simply run the following `pip install` in an environment that has `python 64-bit >= 3.5`.
+NNI supports and is tested on Ubuntu >= 16.04, macOS >= 10.14.1, and Windows 10 >= 1809. Simply run the following `pip install` in an environment that has `python 64-bit >= 3.6`.
 
 Linux or macOS
 

deployment/docker/Dockerfile

@@ -38,7 +38,7 @@ RUN cp /usr/bin/python3 /usr/bin/python
 #
 # update pip
 #
-RUN python3 -m pip install --upgrade pip==20.0.2 setuptools==39.1.0
+RUN python3 -m pip install --upgrade pip==20.0.2 setuptools==41.0.0
 
 # numpy 1.14.3  scipy 1.1.0
 RUN python3 -m pip --no-cache-dir install \

docs/en_US/Assessor/CustomizeAssessor.md

@@ -54,7 +54,7 @@ assessor:
 
 Please noted in **2**. The object `trial_history` are exact the object that Trial send to Assessor by using SDK `report_intermediate_result` function.
 
-The working directory of your assessor is `<home>/nni/experiments/<experiment_id>/log`, which can be retrieved with environment variable `NNI_LOG_DIRECTORY`,
+The working directory of your assessor is `<home>/nni-experiments/<experiment_id>/log`, which can be retrieved with environment variable `NNI_LOG_DIRECTORY`,
 
 More detail example you could see:
 > * [medianstop-assessor](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/medianstop_assessor)

docs/en_US/NAS/SPOS.md

@@ -75,7 +75,7 @@ Then search with evolution tuner.
 nnictl create --config config_search.yml
 ```
 
-The final architecture exported from every epoch of evolution can be found in `checkpoints` under the working directory of your tuner, which, by default, is `$HOME/nni/experiments/your_experiment_id/log`.
+The final architecture exported from every epoch of evolution can be found in `checkpoints` under the working directory of your tuner, which, by default, is `$HOME/nni-experiments/your_experiment_id/log`.
 
 ### Step 3. Train from Scratch
 

docs/en_US/SupportedFramework_Library.md

@@ -1,5 +1,5 @@
 # Framework and Library Supports 
-With the built-in Python API, NNI naturally supports the hyper parameter tuning and neural network search for all the AI frameworks and libraries who support Python models(`version >= 3.5`). NNI had also provided a set of examples and tutorials for some of the popular scenarios to make jump start easier.
+With the built-in Python API, NNI naturally supports the hyper parameter tuning and neural network search for all the AI frameworks and libraries who support Python models(`version >= 3.6`). NNI had also provided a set of examples and tutorials for some of the popular scenarios to make jump start easier.
 
 ## Supported AI Frameworks
 

docs/en_US/TrialExample/Trials.md

@@ -156,19 +156,19 @@ If NNI Annotation is used, the trial's converted code is in another temporary di
 #!/bin/bash
 cd /tmp/user_name/nni/annotation/tmpzj0h72x6 #This is the actual directory
 export NNI_PLATFORM=local
-export NNI_SYS_DIR=/home/user_name/nni/experiments/$experiment_id$/trials/$trial_id$
+export NNI_SYS_DIR=/home/user_name/nni-experiments/$experiment_id$/trials/$trial_id$
 export NNI_TRIAL_JOB_ID=nrbb2
-export NNI_OUTPUT_DIR=/home/user_name/nni/experiments/$eperiment_id$/trials/$trial_id$
+export NNI_OUTPUT_DIR=/home/user_name/nni-experiments/$eperiment_id$/trials/$trial_id$
 export NNI_TRIAL_SEQ_ID=1
 export MULTI_PHASE=false
 export CUDA_VISIBLE_DEVICES=
-eval python3 mnist.py 2>/home/user_name/nni/experiments/$experiment_id$/trials/$trial_id$/stderr
-echo $? `date +%s%3N` >/home/user_name/nni/experiments/$experiment_id$/trials/$trial_id$/.nni/state
+eval python3 mnist.py 2>/home/user_name/nni-experiments/$experiment_id$/trials/$trial_id$/stderr
+echo $? `date +%s%3N` >/home/user_name/nni-experiments/$experiment_id$/trials/$trial_id$/.nni/state
 ```
 
 ### Other Modes
 
-When running trials on other platforms like remote machine or PAI, the environment variable `NNI_OUTPUT_DIR` only refers to the output directory of the trial, while the trial code and `run.sh` might not be there. However, the `trial.log` will be transmitted back to the local machine in the trial's directory, which defaults to `~/nni/experiments/$experiment_id$/trials/$trial_id$/`
+When running trials on other platforms like remote machine or PAI, the environment variable `NNI_OUTPUT_DIR` only refers to the output directory of the trial, while the trial code and `run.sh` might not be there. However, the `trial.log` will be transmitted back to the local machine in the trial's directory, which defaults to `~/nni-experiments/$experiment_id$/trials/$trial_id$/`
 
 For more information, please refer to [HowToDebug](../Tutorial/HowToDebug.md).
 

docs/en_US/Tuner/CustomizeAdvisor.md

@@ -33,7 +33,7 @@ advisor:
     arg1: value1
 ```
 
-**Note that** The working directory of your advisor is `<home>/nni/experiments/<experiment_id>/log`, which can be retrieved with environment variable `NNI_LOG_DIRECTORY`.
+**Note that** The working directory of your advisor is `<home>/nni-experiments/<experiment_id>/log`, which can be retrieved with environment variable `NNI_LOG_DIRECTORY`.
 
 ## Example
 

docs/en_US/Tuner/CustomizeTuner.md

@@ -87,7 +87,7 @@ parameters = {"dropout": 0.3, "learning_rate": 0.4}
 value = 0.93
 ```
 
-**Note that** The working directory of your tuner is `<home>/nni/experiments/<experiment_id>/log`, which can be retrieved with environment variable `NNI_LOG_DIRECTORY`, therefore, if you want to access a file (e.g., `data.txt`) in the directory of your own tuner, you cannot use `open('data.txt', 'r')`. Instead, you should use the following:
+**Note that** The working directory of your tuner is `<home>/nni-experiments/<experiment_id>/log`, which can be retrieved with environment variable `NNI_LOG_DIRECTORY`, therefore, if you want to access a file (e.g., `data.txt`) in the directory of your own tuner, you cannot use `open('data.txt', 'r')`. Instead, you should use the following:
 
 ```python
 _pwd = os.path.dirname(__file__)

docs/en_US/Tuner/NetworkmorphismTuner.md

@@ -59,7 +59,7 @@ If you want to save and load the **best model**, the following methods are recom
 ```python
 # 1. Use NNI API
 ## You can get the best model ID from WebUI
-## or `nni/experiments/experiment_id/log/model_path/best_model.txt'
+## or `nni-experiments/experiment_id/log/model_path/best_model.txt'
 
 ## read the json string from model file and load it with NNI API
 with open("best-model.json") as json_file:

docs/en_US/Tutorial/InstallationLinux.md

@@ -6,7 +6,7 @@ Installation on Linux and macOS follow the same instructions, given below.
 
 ### Install NNI through pip
 
-  Prerequisite: `python 64-bit >= 3.5`
+  Prerequisite: `python 64-bit >= 3.6`
 
   ```bash
   python3 -m pip install --upgrade nni
@@ -16,7 +16,7 @@ Installation on Linux and macOS follow the same instructions, given below.
 
   If you are interested in special or the latest code versions, you can install NNI through source code.
 
-  Prerequisites: `python 64-bit >=3.5`, `git`, `wget`
+  Prerequisites: `python 64-bit >=3.6`, `git`, `wget`
 
   ```bash
   git clone -b v1.7 https://github.com/Microsoft/nni.git

docs/en_US/Tutorial/InstallationWin.md

@@ -2,7 +2,7 @@
 
 ## Prerequires
 
-* Python 3.5 (or above) 64-bit. [Anaconda](https://www.anaconda.com/products/individual) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html) is highly recommended to manage multiple Python environments on Windows.
+* Python 3.6 (or above) 64-bit. [Anaconda](https://www.anaconda.com/products/individual) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html) is highly recommended to manage multiple Python environments on Windows.
 
 * If it's a newly installed Python environment, it needs to install [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) to support build NNI dependencies like `scikit-learn`.
 

docs/en_US/Tutorial/QuickStart.md

@@ -2,7 +2,7 @@
 
 ## Installation
 
-We currently support Linux, macOS, and Windows. Ubuntu 16.04 or higher, macOS 10.14.1, and Windows 10.1809 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
+We currently support Linux, macOS, and Windows. Ubuntu 16.04 or higher, macOS 10.14.1, and Windows 10.1809 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.6`.
 
 ### Linux and macOS
 

docs/en_US/_templates/index.html

@@ -237,7 +237,7 @@ <h2 class="second-title">Install</h2>
     <p>
       NNI supports and is tested on Ubuntu >= 16.04, macOS >= 10.14.1,
       and Windows 10 >= 1809. Simply run the following `pip install`
-      in an environment that has `python 64-bit >= 3.5`.
+      in an environment that has `python 64-bit >= 3.6`.
     </p>
     <div class="command-intro">Linux or macOS</div>
     <div class="command">python3 -m pip install --upgrade nni</div>

examples/tuners/customized_tuner/setup.py

@@ -8,7 +8,7 @@
     version = '0.1',
     packages = setuptools.find_packages(exclude=['*test*']),
 
-    python_requires = '>=3.5',
+    python_requires = '>=3.6',
     classifiers = [
         'Programming Language :: Python :: 3',
         'License :: OSI Approved :: MIT License',

setup.py

@@ -27,7 +27,7 @@ def read(fname):
         'nni_gpu_tool':'tools/nni_gpu_tool'
     },
     package_data = {'nni': ['**/requirements.txt']},
-    python_requires = '>=3.5',
+    python_requires = '>=3.6',
     install_requires = [
         'astor',
         'hyperopt==0.1.2',

src/nni_manager/training_service/dlts/dltsTrainingService.ts

@@ -266,7 +266,7 @@ class DLTSTrainingService implements TrainingService {
     public async submitTrialJob(form: TrialJobApplicationForm): Promise<TrialJobDetail> {
         const trialJobId: string = uniqueString(5);
         const trialWorkingFolder: string = path.join(
-            '/nni/experiments', getExperimentId(),
+            '/nni-experiments', getExperimentId(),
             '/trials/', trialJobId);
         const trialJobDetail = new DLTSTrialJobDetail(
             trialJobId, // id

src/nni_manager/training_service/pai/paiTrainingService.ts

@@ -54,7 +54,7 @@ abstract class PAITrainingService implements TrainingService {
         this.metricsEmitter = new EventEmitter();
         this.trialJobsMap = new Map<string, PAITrialJobDetail>();
         this.jobQueue = [];
-        this.expRootDir = path.join('/nni', 'experiments', getExperimentId());
+        this.expRootDir = path.join('/nni-experiments', getExperimentId());
         this.experimentId = getExperimentId();
         this.paiJobCollector = new PAIJobInfoCollector(this.trialJobsMap);
         this.paiTokenUpdateInterval = 7200000; //2hours

src/nni_manager/training_service/remote_machine/shellExecutor.ts

@@ -175,7 +175,7 @@ class ShellExecutor {
     }
 
     public getRemoteExperimentRootDir(experimentId: string): string {
-        return this.joinPath(this.tempPath, 'nni', 'experiments', experimentId);
+        return this.joinPath(this.tempPath, 'nni-experiments', experimentId);
     }
 
     public joinPath(...paths: string[]): string {

src/sdk/pycli/setup.py

@@ -8,7 +8,7 @@
     version='999.0.0-developing',
     packages=setuptools.find_packages(),
 
-    python_requires='>=3.5',
+    python_requires='>=3.6',
     install_requires=[
         'requests'
     ],

src/sdk/pynni/setup.py

@@ -12,7 +12,7 @@ def read(fname):
     version = '999.0.0-developing',
     packages = setuptools.find_packages(exclude=['tests']),
 
-    python_requires = '>=3.5',
+    python_requires = '>=3.6',
     install_requires = [
         'hyperopt==0.1.2',
         'json_tricks',

src/webui/mock/dict-metrics-100.json

@@ -14,7 +14,7 @@
             "trainingServicePlatform": "local",
             "tuner": {
                 "builtinTunerName": "Random",
-                "checkpointDir": "/home/user/nni/experiments/TN5K0Qju/checkpoint"
+                "checkpointDir": "/home/user/nni-experiments/TN5K0Qju/checkpoint"
             },
             "versionCheck": true,
             "clusterMetaData": [

src/webui/mock/mnist-tfv1-nested.json

@@ -7,7 +7,7 @@
         "id": "AKVcExIf",
         "revision": 29,
         "execDuration": 172,
-        "logDir": "/home/v-yugzh/nni/experiments/AKVcExIf",
+        "logDir": "/home/v-yugzh/nni-experiments/AKVcExIf",
         "nextSequenceId": 10,
         "params": {
             "authorName": "default",

src/webui/mock/mnist-tfv1-running.json

@@ -7,7 +7,7 @@
         "id": "GD5MnU8G",
         "revision": 22,
         "execDuration": 172,
-        "logDir": "/home/user/nni/experiments/GD5MnU8G",
+        "logDir": "/home/user/nni-experiments/GD5MnU8G",
         "nextSequenceId": 3,
         "params": {
             "authorName": "default",

src/webui/mock/mnist-tfv2-stage0.json

@@ -7,7 +7,7 @@
     "id": "mw3lSbde",
     "revision": 3,
     "execDuration": 8,
-    "logDir": "/***/nni/experiments/mw3lSbde",
+    "logDir": "/***/nni-experiments/mw3lSbde",
     "nextSequenceId": 1,
     "params": {
       "authorName": "NNI Example",

src/webui/mock/mnist-tfv2-stage1.json

@@ -7,7 +7,7 @@
     "id": "mw3lSbde",
     "revision": 34,
     "execDuration": 233,
-    "logDir": "/***/nni/experiments/mw3lSbde",
+    "logDir": "/***/nni-experiments/mw3lSbde",
     "nextSequenceId": 9,
     "params": {
       "authorName": "NNI Example",

src/webui/mock/mnist-tfv2-stage2.json

@@ -7,7 +7,7 @@
     "id": "mw3lSbde",
     "revision": 39,
     "execDuration": 265,
-    "logDir": "/***/nni/experiments/mw3lSbde",
+    "logDir": "/***/nni-experiments/mw3lSbde",
     "nextSequenceId": 11,
     "params": {
       "authorName": "NNI Example",

tools/README.md

@@ -4,7 +4,7 @@ The NNI CTL module is used to control Neural Network Intelligence, including sta
 
 ```
 Ubuntu 16.04 or other Linux OS
-python >= 3.5
+python >= 3.6
 ```
 
 ## Installation

tools/bash-completion

@@ -75,7 +75,7 @@ _nnictl()
 
         # add experiment IDs to candidates if desired
         if [[ $__nnictl_2nd_expid_cmds =~ " ${COMP_WORDS[1]} " ]]; then
-            local experiments=$(ls ~/nni/experiments 2>/dev/null)
+            local experiments=$(ls ~/nni-experiments 2>/dev/null)
             COMPREPLY+=($(compgen -W "$experiments" -- $cur))
         fi
 

tools/nni_annotation/__init__.py

@@ -122,7 +122,7 @@ def _expand_file_annotations(src_path, dst_path, nas_mode):
 def _generate_specific_file(src_path, dst_path, exp_id, trial_id, module):
     with open(src_path) as src, open(dst_path, 'w') as dst:
         try:
-            with open(os.path.expanduser('~/nni/experiments/%s/trials/%s/parameter.cfg'%(exp_id, trial_id))) as fd:
+            with open(os.path.expanduser('~/nni-experiments/%s/trials/%s/parameter.cfg'%(exp_id, trial_id))) as fd:
                 para_cfg = json.load(fd)
             annotated_code = specific_code_generator.parse(src.read(), para_cfg["parameters"], module)
             if annotated_code is None:

tools/nni_cmd/nnictl_utils.py

@@ -445,9 +445,9 @@ def remote_clean(machine_list, experiment_id=None):
         sshKeyPath = machine.get('sshKeyPath')
         passphrase = machine.get('passphrase')
         if experiment_id:
-            remote_dir = '/' + '/'.join(['tmp', 'nni', 'experiments', experiment_id])
+            remote_dir = '/' + '/'.join(['tmp', 'nni-experiments', experiment_id])
         else:
-            remote_dir = '/' + '/'.join(['tmp', 'nni', 'experiments'])
+            remote_dir = '/' + '/'.join(['tmp', 'nni-experiments'])
         sftp = create_ssh_sftp_client(host, port, userName, passwd, sshKeyPath, passphrase)
         print_normal('removing folder {0}'.format(host + ':' + str(port) + remote_dir))
         remove_remote_directory(sftp, remote_dir)
@@ -520,7 +520,7 @@ def experiment_clean(args):
         home = str(Path.home())
         local_dir = nni_config.get_config('experimentConfig').get('logDir')
         if not local_dir:
-            local_dir = os.path.join(home, 'nni', 'experiments', experiment_id)
+            local_dir = os.path.join(home, 'nni-experiments', experiment_id)
         local_clean(local_dir)
         experiment_config = Experiments()
         print_normal('removing metadata of experiment {0}'.format(experiment_id))

tools/setup.py

@@ -8,7 +8,7 @@
     version = '999.0.0-developing',
     packages = setuptools.find_packages(exclude=['*test*']),
 
-    python_requires = '>=3.5',
+    python_requires = '>=3.6',
     install_requires = [
         'requests',
         'ruamel.yaml',