NNI 2021 Jan~Feb Iteration Planning

[scarlett2018]

This is the plan for the iteration in Jan~Feb, it's a 4 weeks iteration.

Release Plan for v2.1

• Release manager: @J-shang
• Feature freeze date: 2.8 2.10
• Code freeze and demo date: 2.22 2.24 2.26
• Release date: 3.1 3.8 3.10

---

Let's try to adopt the legend of annotations:

Mark	Description
🏃‍♂️	work in progress
👪	work under review
✋	blocked task
💪	stretch goal for this iteration
🔴	missing issue / pull request reference
🔵	more investigation and survey required
⚫	under discussion within the team
⬛	a large work item, larger than one iteration

Features

nnictl & nni.experiment

• P0 - migrate nnicli to python approach @J-shang DL: 1.27 PR: migrate nnicli #3334 review: @liuzhe-lz @QuanluZhang
• 👪 P1 P0 - launch/view/ stop python approach @J-shang pr: add python api example and update doc #3396 review: @QuanluZhang @liuzhe-lz @SparkSnail
• P1 - Show a warning if user run trial.py without nnictl @liuzhe-lz PR Make standalone mode warning more obvious #3312 review: @ultmaster @J-shang
• Nnictl support launch from Python (merge .config & nni.sqlite) @J-shang PR Merge .config & nni.sqlite #3309 review: @SparkSnail @liuzhe-lz

Retiarii & NAS

• P0 - Improve graph generation and code generation logic (support control flow) @QuanluZhang DL: 1.29 2.5 PR: [retiarii] graph gen and code gen improvement #3330 [retiarii] refactor of pytorch operators #3365 review: @ultmaster @liuzhe-lz
  • Support member function, and more operators DL: 1.25 1.27
  • Support shared module DL: 2.1
  • Test graph convert on different pytorch versions
  • Support member variable
• P0 - Support APIs ValueChoice @ultmaster DL: 1.27 PR: [Retiarii] Improve high-level API interface and add implementation of ValueChoice #3349
  • support ValueChoice parameters DL: 2.2 PR: [Retiarii] Use ValueChoice inline in a serializable instance #3382 review: @QuanluZhang @liuzhe-lz
• P1 - Refactor Trainer @ultmaster DL: 2.5 PR: [Retiarii] Rewrite trainer with PyTorch Lightning #3359
• P1 - Support serialization @ultmaster
• P1 - Support 2 or 3 simple search algorithms (grid, random, evolution) @ultmaster PR: [Retiarii] Grid search, random and evolution strategy #3377 review: @QuanluZhang @liuzhe-lz

Model compression

• P2 - Make model compression user experience more user-friendly @colorjam DL: 1.29 Refactor model compression examples #3326 review: @QuanluZhang @J-shang
• P2 - Documentation improvement (update API reference for model compression in doc, deferred from v2.0) @colorjam DL: 1.29
• 👪 Refactor Compression Doc @colorjam Refactor Compression Doc #3371 review: @QuanluZhang @J-shang

Training service & NNI manager

• P1 - Formally support shared storage for reusable training service @J-shang DL: 1.22 1.27 PR: support shared storage for reusable mode #3354 review: @SparkSnail @liuzhe-lz
• P1 - Hybrid local windows support @SparkSnail DL: 1.25 PR: support local windows in hybrid mode #3353
• P1 - Remove PAIYarn code @SparkSnail DL: 1.22 PR: Remove paiYarn mode #3327
• P1 - AML outputs path & python process cannot be killed @J-shang PR fix aml outputs and python process not killed #3321 review: @SparkSnail @liuzhe-lz
• P1 - show the assigned training service of each trial in hybrid training service (show the assigned training service of each trial in hybrid training service #3261) @SparkSnail DL: 2.10 PR: Show trial environment information in reuse mode #3389 review: @J-shang @liuzhe-lz
• P1 - AML scheduling @SparkSnail DL: 2.5 2.7 PR: Add recently-idle environment scheduler in reuse mode #3375 review: @QuanluZhang @J-shang
• P2 - Reuse mode envs dir iterate @SparkSnail DL: 2.5 2.7 PR: Fix reuse mode env folder #3374 review: @liuzhe-lz @J-shang
• P2 - GPU metrics collector does not take preCommand in remote training service (GPU metrics collector does not take preCommand in remote training service #3284) @J-shang @acured Change command lines as serial. #3367
• P3 - Flatten nested JSON in RPC protocol (deferred from v2.0) @liuzhe-lz

WebUI

• P1- manage all experiments Support multiple selection for filter status @Lijiaoa DL: 1.29 PR: Support multiple selection for filter status (manage all experiments page) #3351
• Overview page improvement @Lijiaoa
  • P1 - Update default metric tooltip [webui enhancement] about the tooltip content for default metric chart #2964 DL: 1.22 PR: [webui] Update default metric tooltip #3316 review: @ultmaster @liuzhe-lz
  • P1 - Support show customized tuner name DL: 1.22 PR: Support show customized tuner/assessor name and get optimize_mode from tuner/advisor/assessor #3317
  • P1 - Support show trial details by clicking the point in trial results graph DL: 1.29 PR: Support show trial details by clicking the point in trial results graph #3352
• P1 - copy trial id in table @Lijiaoa PR: Support copy trial id, experiment id in the table #3378 review: @ultmaster @liuzhe-lz
• P1 - Hybrid show trial platforms in webui @Lijiaoa (need UX design, depends on show the assigned training service of each trial in hybrid training service #3261)

Pipelines

• P1 - Pipeline speedup @ultmaster DL: 1.25 PR: [CI] Introduce multi-stage pipeline for speedup #3313
• P1 - pipeline dependence migration DL: 2.3 2.5 @ultmaster PR: [CI] Fast-test cross-platform speedup #3364
• P1 - fix IT @J-shang PR: Fix pipeline #3366

Deferred

nnictl & nni.experiment

• P1 - Improve tuner debuggability, allow debugger tool @liuzhe-lz
• P1 - write v2 config files for examples @liuzhe-lz PR: Convert config files in examples to v2 schema #3388 (Depend on pr Add native support for v2 config #3315)
• P2 - Support nnictl tensorboard in Windows @J-shang
• P2 - Mask confidential config fields @liuzhe-lz
• P2 - nnictl auto-completion @liuzhe-lz
• P3 - launch NNI manager in remote container #modelbuilderneeds @liuzhe-lz

Retiarii & NAS

• P1 - Support one system-level optimization: CSE optimization to deduplicate computation @hzhua
• P1 - NAS visualization improvements: refactor graph representation (deferred from v2.0) @98may
• P2 - refactor strategy (strategy keep running when experiment stop)
• P2 - Add high level API for once-for-all related supports @ultmaster
• P3 - Test classic nas tuners on NNI NAS benchmark and report evaluation results (deferred from v2.0) @ultmaster

Model compression

• 👪 P1 - Quantization speedup tool @linbinskn DL: 1.22 PR: Support mixed-precision quantization speed up by using tensorrt #3220 Support mixed-precision quantization speed up by using tensorrt[original pr #3220 has been closed] #3356 review: @J-shang @liuzhe-lz @QuanluZhang
• P2 - Pruning model speedup improvement @zheng-ningxin
• P2 - Update TF model compression: support checkpoint (deferred from v2.0) @liuzhe-lz

Training service & NNI manager

• P1 - Native support for config v2 @liuzhe-lz PR: Add native support for v2 config #3315 review: @SparkSnail @J-shang
  • P0 - support view mode
• P2 - Reuse mode remote windows (@acured ask @SparkSnail )
• P2 - Hybrid kubeflow support @SparkSnail
• P2 - Reuse mode envs dir iterate @SparkSnail
• P2 - Documentation improvement @QuanluZhang
• P2 - Add more error message for remote training service (more error message for remote training service #3259) @QuanluZhang (fixed upstream, to verify the fix)
• P3 - Unify IPC mechanism for local and others @liuzhe-lz @ultmaster

Hyper-parameter tuning

• P2 - Improve hyperband, PBT with shared storage (dependency on shared storage implementation) @J-shang
• P3 - Formal Advisor API @liuzhe-lz
• P2 - Multi-objective optimization algorithm (deferred from v2.0) @ultmaster
• P4 - FBnetV3 tuning algorithm (deferred from v2.0) @J-shang
• P4 - Tutorial to use tuner directly instead of using nni manager (like hyperopt?) (deferred from v2.0) @liuzhe-lz

WebUI

• Overview page improvement @Lijiaoa
  • P2 - Adaptive UI (height) for high resolution, current window for high resolution need improvement @yangou1988 @Lijiaoa
  • P3 - Fix redundant margin at the bottom of the page
• P2 - Support tensorboard @Lijiaoa
  • UX design DL: 2.3
  • backend
• Detail Page improvement
  • P1 - Refine search parameters Refine search by paramters #3024 Improvement of Search Function in Trials Detail page  #3099 DL: 2.21
  • P2 - Put the add/remove button in the last column of the header Improvement of Search Function in Trials Detail page  #3099
• P3 - webUI dispatcher log / nnimanager log wrap ([Enhancement] View log message more conveniently  #3246) @Lijiaoa

Pipelines

• P2 - add reuse test @liuzhe-lz
• P3 - Support code coverage check, add this check to IT (deferred from v2.0) @ultmaster

Documentation

• P1 - Refactor readthedocs integration @ultmaster
• P2 - Improve crowdin workflow @ultmaster

Design Discussion Needed

• Design discussion for next steps of manage all experiments, including "actions" for Manage experiments, etc. HPO experiments management web dashboard #3100
• Design discussion for Duration issue [webui] about trial duration question #3012
• fail trials counted in total trial nums
• Improve a more typing-friendly experiment id pattern (Proposal: a more typing-friendly experiment id pattern #3102)
• Avoid reloading data for each trail (Did every trail need to load data? #3294) @QuanluZhang
• Experiment tag

Others

• P3 - Bump dependency version (deferred from v2.0) @liuzhe-lz
• P2 - Add the output of subprocess (Add output of subprocess in trial #3141) @liuzhe-lz
• P3 - Set NNI base URL (aka webui prefix) (Is there any option to set base url for the web report? #2771)
  • Check users' requirements @Lijiaoa @yiru
  • Web UI front-end @Lijiaoa
  • backend @J-shang
  • NNI manager & nnictl back-end @?
• P4 - AutoML for data augmentation (image and NLP) (research)
• P3 - Improve dev-install on Windows @liuzhe-lz

Backlog

• support IPV6
• update tuner search space

[J-shang]

bugbash

nnictl & nni.experiment

• P0 - migrate nnicli to python approach @kvartet
• P1 P0 - launch/view/ stop python approach @kvartet
• P1 - Show a warning if user run trial.py without nnictl @J-shang
• Nnictl support launch from Python (merge .config & nni.sqlite) (test nnictl resume) @liuzhe-lz

Retiarii & NAS

• P0 - Improve graph generation and code generation logic (support control flow) @ultmaster
  • Support member function, and more operators
  • Support shared module
  • Test graph convert on different pytorch versions
• P0 - Support APIs ValueChoice @QuanluZhang
  • support ValueChoice parameters
• P1 - Refactor Trainer @colorjam
• P1 - Support 2 or 3 simple search algorithms (grid, random, evolution) @98may

Model compression

• Compression Doc @kvartet
• Compression Example @linbinskn

Training service & NNI manager

• P1 - Formally support shared storage for reusable training service @SparkSnail
• P1 - Hybrid local windows support @J-shang
• P1 - AML outputs path & python process cannot be killed @SparkSnail
• P1 - show the assigned training service of each trial in hybrid training service (show the assigned training service of each trial in hybrid training service #3261) @J-shang
• P1 - AML scheduling @acured
• P2 - GPU metrics collector does not take preCommand in remote training service (GPU metrics collector does not take preCommand in remote training service #3284) @ultmaster

WebUI

• P1- manage all experiments Support multiple selection for filter status @liuzhe-lz
• Overview page improvement @J-shang
  • P1 - Update default metric tooltip [webui enhancement] about the tooltip content for default metric chart #2964
  • P1 - Support show customized tuner name
  • P1 - Support show trial details by clicking the point in trial results graph
• P1 - copy trial id in table @QuanluZhang
• P1 - Hybrid show trial platforms in webui @liuzhe-lz

[J-shang]

bugbash 2.0

nnictl & nni.experiment

• tuner name display on web UI (launch from python) @kvartet
• update_search_space should work when use start() now @kvartet

Retiarii & NAS

• Support APIs ValueChoice @QuanluZhang
  • support ValueChoice parameters
• Refactor Trainer @colorjam
• Support 2 or 3 simple search algorithms (grid, random, evolution) @98may
• lots of log

Training service & NNI manager

• Formally support shared storage for reusable training service @SparkSnail
• Hybrid v2 config convert @J-shang
• AML scheduling @acured
• GPU metrics collector does not take preCommand in remote training service (GPU metrics collector does not take preCommand in remote training service #3284) @ultmaster