July iteration plan

[scarlett2018]

This is the draft plan for the upcoming iteration in July, it's a 4 weeks iteration, and it is targeted to release on early August.

Release Plan

• Release manager: @ultmaster
• Feature freeze date: 7/31
• Code freeze and demo date: 8/7 8/12
• Retrospective date: 8/14
• Release and next iteration plan date: 8/17 8/24

---

Let's try to adopt the legend of annotations:

Mark	Description
🏃‍♂️	work in progress
👪	work under review
✋	blocked task
💪	stretch goal for this iteration
🔴	missing issue / pull request reference
🔵	more investigation and survey required
⚫	under discussion within the team
⬛	a large work item, larger than one iteration

Framework and SDK support

• 👪 export and import experiment to make experiment be resumed or viewed on another machine Can I start the WebUI from the experiment folder? #2001 @SparkSnail (ETA 7/22 for finalize design) (7/31 feature freeze) Support save and open experiments #2750 rev: @QuanluZhang @chicm-ms
  • PR ready for demo
• 👪 export intermediate results and summary (both running and succeeded) nnictl change @tabVersion (Feature freeze ETA 7/24) feature: export experiment results #2706 rev: @QuanluZhang @SparkSnail pending resolving conflicts
• stop experiment with port (already supported, --port)
• 👪 Directly display log (downloadable link) in Web UI (needs to go through every training service and decide for each implement local mode only) legend PRs: Add REST API to download trial log #1706 Could download trial log in table open row #1738 PR: [webui] could view trial log | stderr on webui(only support local mode) #2731 support display trial log on local mode #2718 @JunweiSUN @Lijiaoa (ETA 7/21 summary of the current to be improved things on TS) rev: @liuzhe-lz @ultmaster
  • PR ready for demo
• Default log folder /home/user/nni/experiments -> /home/user/nni-experiments. @chicm-ms 7/17 rev: @ultmaster @JunweiSUN (PR Update nni exp dir #2686)
  • 👪 More fix in update experiment dir #2753 @JunweiSUN rev: @chicm-ms @SparkSnail
• 🔵 Fix evolution tuner runtime error and index error Fix evolution tuner RuntimeError and IndexError #2695
• 👪 Upgrade docker image tensorboard to 1.15 upgrade tensorflow version for docker image (#2732) #2735 @chicm-ms rev: @ultmaster @SparkSnail
• Remove thop from mnist-pytorch example
• ⚫ Parameter id starts from 0 when experiment resumed @ultmaster

Feature engineering

• 👪 Fix PyTorch import issue for feature engineering @chicm-ms 7/17 rev: @xuehui1991 @ultmaster @JunweiSUN (ETA 7/17 feature freeze) PR update gradient_selector dataloader iterator import #2690

NAS

• 🔵 TensorFlow v2 NAS performance issue TF NAS fix #2781 @liuzhe-lz rev: @ultmaster @QuanluZhang (ETA 7/17 feature freeze)
• 👪 Search space zoo ENAS and DRATS search space zoo #2589 @tabVersion rev: @ultmaster @QuanluZhang
• Add intermediate result API to NAS Benchmarks Add API to query intermediate results in NAS benchmark #2728 @ultmaster
• Fix NAS benchmarks bug nasbench201 api bug #2721 Add API to query intermediate results in NAS benchmark #2728 @ultmaster rev: @JunweiSUN @QuanluZhang

Web UI

• Add WebUI mock server and improve hyper-parameter parallel coordinates chart Add WebUI mock server and improve hyper-parameter parallel coordinates chart #2691 @ultmaster
• 🏃‍♂️ Redesign interpolate color of hyper-parameter chart @ultmaster Change line color range and rendering order #2759 @Lijiaoa @colorjam
• 🏃‍♂️ Fix issues for corner-case metrics/parameters (NaN/infinity/undefined/other-types) deal with all type metrics #2767 @Lijiaoa
• 👪 Trial details table pagination and refactor lines chart @colorjam add pagination & hotfix intermediate graph #2738 rev: @ultmaster @Lijiaoa
• 👪 Log files drawer cannot be closed via clicking overlay area. @Lijiaoa refactor experiment summary file(to fix componentWillReceiveProps waring) and click other areas to close panel #2734 rev: @ultmaster @liuzhe-lz
• 👪 Capitalization and spacing issues (before release). [webui] word format (such as Experiment summary, Log files) #2737 @Lijiaoa
• 🏃‍♂️ Trial No. should show sequence id instead of parameter id @JunweiSUN support display trial log on local mode #2718 needs another PR @Lijiaoa delete multiphase in webui #2760 rev: @squirrelsc @liuzhe-lz
• 💪 Backlog: WebUI, NasUI Enhancement Backlog #2683

Training Service

• bug: experiment cannot be resumed in pai mode @SparkSnail
• 👪 add log info for uploading codedir in training services Add log for copy data in pai mode #2702 @SparkSnail rev: @chicm-ms @ultmaster (ETA 7/24 feature freeze)
• 👪 add pai job detail url in webportal Add pai job detail uri in webportal #2703 @SparkSnail rev: @ultmaster @chicm-ms
• improve paimode example, change containerNFSMountPATH to /mnt/confignfs-data like paiStorageConfigName. Write placeholder in example. @SparkSnail Update doc for PAI #2685 reviewer: @squirrelsc @JunweiSUN @scarlett2018 (ETA 7/17 feature freeze)
• 👪 Training service scheduler - @squirrelsc had a PR, rev: @SparkSnail @chicm-ms (needs review and test Reusable environment support GPU scheduler, add test cases and refactoring. #2627)
  • 👪 PR submitted to v1.7.1 Hotfix for v1.7.1 #2692
  • subtask: write tests for refactored training service framework @squirrelsc rev: @SparkSnail @QuanluZhang
• Change message in

  nni/tools/nni_trial_tool/web_channel.py

  Line 26 in 0b9d6ce

  nni_log(LogType.Info, 'WebChannel: connected with info %s' % url)

• Set timeout for loop in trial runner @SparkSnail Add timeout for web_channel in trial_runner #2710
• 👪 test refactoring. CI issue: test sub-directory will fail the experiment resume test @linbinskn rev: @SparkSnail @chicm-ms (ETA 7/24 feature freeze) add sub-directory for testing pai resume #2694
  • by chicm-ms (8/12): add sub-directory for testing pai resume #2694 breaks windows local and remote pipelines, need to fix.
• bug: remote mode issue with multi-users remote mode issue with multi-users #2688 (PR fix chmod bug #2689) @JunweiSUN
• 👪 support more remote servers (>20) in remote mode @SparkSnail Fix remote machine connection logic #2725 rev: @chicm-ms @squirrelsc
• 👪 AML experiment turns into STOPPED status when azureml-sdk is not installed (needs double check) @SparkSnail Show environment error message in aml #2724 rev: @JunweiSUN @ultmaster
• 🔵 (by design) PAI job doesn't stop when experiment is stopped in reuse mode. @SparkSnail

Example, Tutorial, Jump start and doc

• 👪 nnicli intro and modify a few APIs @JunweiSUN (ETA 7/24 finalize survey) update nnicli #2713 @chicm-ms @QuanluZhang
• 👪 Colab support tutorial (nnicli/frp/ngrok) add tutorial for colab support #2700 @JunweiSUN rev: @ultmaster @QuanluZhang @SparkSnail
• 👪 Docker documentation (README outdated) update package version of docker doc #2720 revising upgrade tensorflow version for docker image (#2732) #2735 @JunweiSUN rev: @chicm-ms @SparkSnail

More scenario support (i.e. Data)

Model Compression

• 👪 TensorFlow v2 model compression support (pruner framework, one-shot algorithm) TensorFlow v2 Support #2127 @liuzhe-lz Compression for Tensorflow #2755 rev: @QuanluZhang @chicm-ms
  • Review on how the current NNI Pytorch model compression implementation, survey and compare the corresponding TF 2.0 supportability. ETA 7/22
• 👪 benchmark - speedup to support RetinaFace @suiguoxin compression benchmark #2742 rev: @chicm-ms @QuanluZhang
  • documentation Compression doc structure refactor #2676
• 👪 Model speedup enhancement (collaborate with Yuqing team) DenseNet, Inception @zheng-ningxin Support the List/Tuple Construct/Unpack operation for TorchModuleGraph #2609 Speedup enhancement #2719 rev: @QuanluZhang @chicm-ms
• 👪 Sensitivity pruner update Sensitivity pruner #2684 @zheng-ningxin rev: @chicm-ms @colorjam @QuanluZhang
• 👪 AMC (automl for model compression) paper implementation, missing docs and tests @chicm-ms rev: @QuanluZhang @colorjam (ETA 7/24 7/27 PR ready) AutoML for model compression #2573
  • PR ready for demo

Multiple Experiments Management @yangou1988 (ETA 7/24 finalize survey) #2779

1.7.1 candidate

• bug (1.7.1 candidate): improve reuse training service to handle earlier failure of jobs. for example, if user write incorrect path in openpai, it causes openpai job failed, but trial is waiting. The trial dispatcher will create openpai jobs, and fails continually. It needs a value like max failure to report this kind of env fatal error to nni manager, to make the whole experiment failed, and stop to create more env jobs. (@squirrelsc had submitted a PR, reviewer: @SparkSnail)

Deferred

• 🏃‍♂️ Support more tuning algorithms for classic NAS (regularized evolution) @QuanluZhang regularized evolution for image classifier architecture search #2708 rev: @ultmaster @liuzhe-lz (ETA 7/31 feature freeze) (deferred?)
  • PR ready for demo
• 🔵 NAS visualization improvements: refactor graph representation @ultmaster @zheng-ningxin (defer)
  • PR ready for demo
• 🔵 AutoML for data augmentation (image and NLP) @chicm-ms (ETA 7/24 7/31 finalize survey)
• support adding description to experiment to make users easily identify a certain experiment (defer)
• Add/Remove columns in hyper-parameter chart and trial details table and Refactor trial details table (click event) @ultmaster
• 👪 Cream2NNI @hwpengms rev: @ultmaster
  • PR ready for demo
• ✋ Tutorial to use tuner directly instead of using nni manager (like hyperopt?) @JunweiSUN (not started)

Backlog

• Search Space of Tuner, quniform +q or mode q Search Space of Tuner #2397
• Fix hyperband and BOHB: support various mode, mixed multiple rounds or strictly staged rounds.
• new CV NAS algo integration @ultmaster
• 🏃‍♂️ survey form pass internal compliance process @JunweiSUN (Pending for Jing's feedback)

[ultmaster]

Pending PR:

Fix pipeline (scikit-learn and python upgrade) @linbinskn
#2773 @Lijiaoa
#2768 @chicm-ms
#2753 Pending IT @chicm-ms
#2747 @tabVersion @QuanluZhang
#2713 @chicm-ms
#2573 @QuanluZhang @colorjam

[ultmaster]

Bug bash

target date: 8/24 8/26

• PAI training service @colorjam
  • Add log when copying data in PAI mode (Add log for copy data in pai mode #2702)
  • Add timeout for web_channel in trial_runner (Add timeout for web_channel in trial_runner #2710)
  • Add PAI trial job detail link (Add pai job detail uri in webportal #2703)
  • Fix resume in PAI: mkdir error in training service (Fix mkdir error #2673)
  • Non-reuse / reuse: exit at only one trial?
  • Improvements:
    • Unify the webui of reuse and non-reuse mode [v1.8 bug-bash] fix webui error  #2808
• AzureML training service @JunweiSUN @ninshang
  • Show environment error message in AzureML mode (Show environment error message in aml #2724)
  • Support GPU scheduler in reusable environment (Reusable environment support GPU scheduler, add test cases and refactoring. #2627) (Enable gpu scheduler in AML mode #2769)
  • Improvements:
    • Consistently launching new environments
• Local training service @colorjam @ninshang
  • Access trial log directly on WebUI (local mode only) (support display trial log on local mode #2718)
• Remote training service @liuzhe-lz
  • Fix bug when using chmod in remote training service (fix chmod bug #2689) (done)
  • Fix support for dozens of machines in remote training service (Fix remote machine connection logic #2725) @colorjam

WebUI, nnictl and nnicli

• WebUI @SparkSnail
  • Improve hyper-parameter parallel coordinates plot (Add WebUI mock server and improve hyper-parameter parallel coordinates chart #2691) (Change line color range and rendering order #2759)
  • Add pagination for trial job list (add pagination & hotfix intermediate graph #2738) (Change local sort per page to global search of all pages #2773)
  • Enable panel close when clicking overlay region (refactor experiment summary file(to fix componentWillReceiveProps waring) and click other areas to close panel #2734)
  • Remove support for Multiphase on WebUI (delete multiphase in webui #2760)
  • Fix several styling issues in WebUI (Some little style #2762 [webui] word format (such as Experiment summary, Log files) #2737)
  • Fix support for unusual types for metrics including NaN and Infinity (deal with all type metrics #2782)
  • Fix intermediate graph zooming issue (add pagination & hotfix intermediate graph #2738)
• nnictl @JunweiSUN
  • Support save and restore experiments (Support save and open experiments #2750)
  • Add intermediate results in export result (feature: export experiment results #2706)
  • Add command to list trial results with highest/lowest metrics (add nnictl command to list trial results with highest/lowest metric #2747)
  • Fix nnictl experiment delete (Fix nnictl experiment delete #2791)
• nnicli @tabVersion
  • Improve the user experience of nnicli (update nnicli #2713)

Neural architecture search

• Search space zoo: ENAS and DARTS (ENAS and DRATS search space zoo #2589) @QuanluZhang
• API to query intermediate results in NAS benchmark (Add API to query intermediate results in NAS benchmark #2728) @QuanluZhang
  • Fix issue when dict is unordered when querying NAS benchmark (Add API to query intermediate results in NAS benchmark #2728)
• Fix performance issue in NAS with TensorFlow (TF NAS fix #2781) @ultmaster

Model compression

• Speedup @chicm-ms
  • Support the List/Tuple Construct/Unpack operation for TorchModuleGraph (Support the List/Tuple Construct/Unpack operation for TorchModuleGraph #2609)
  • Add support the DenseNet and InceptionV3 for the speedup module (Speedup enhancement #2719)
  • Support the multiple successive tuple unpack operations (Successive unpack #2768)
• Compression benchmark (compression benchmark #2742) @chicm-ms
• Sensitivity pruner (Sensitivity pruner #2684) @chicm-ms
• TensorFlow v2 support in model compression (Compression for Tensorflow #2755) @chicm-ms
• Support AMC pruner (AutoML for model compression #2573) (Replace tensorboardX with torch.utils.tensorboard #2786) @colorjam
• Fix mask issue in SimulatedAnnealingPruner (fix SimulatedAnnealingPruner export mask issue #2736) @chicm-ms

Feature engineering

• Fix import issue for gradient selector dataloader iterator (update gradient_selector dataloader iterator import #2690) @liuzhe-lz

[QuanluZhang]

v1.8 released