Release 0.1.8rc2

[bouthilx]

Important changes

The python API is finally ready for release v0.1.8! 🎉

Python API

An API is now available to run experiments directly from python instead of using the commandline.

from orion.client import create_experiment

experiment = create_experiment(
   name='foo',
   space=dict(x='uniform(-50,50)'))

trial = experiment.suggest()

# Do something using trial.params['x']

results = [dict(
    name='dummy_objective',
    type='objective',
    value=dummy_objective)]

experiment.observe(trial, results)

Current API provides a simple function workon for cheap experiments that can be executed by a single worker, and a generic ExperimentClient (see example above) object for optimization with multiple workers.

See documentation for more details.

New Algorithms

Hyperband

Hyperband extends the Successive Halving algorithm by providing a way to exploit a fixed budget with different number of configurations for SuccessiveHalving algorithm to evaluate. It is especially useful when the trials are expensive to run and cheap noisy evaluations are possible. Think of it as using early evaluation during training to filter out bad candidates.

For more information on the algorithm, see original paper.

TPE

Tree-structured Parzen Estimator (TPE) algorithm is one of Sequential Model-Based Global Optimization (SMBO) algorithms, which will build models to propose new points based on the historical observed trials.

Instead of modeling p(y|x) like other SMBO algorithms, TPE models p(x|y) and p(y), and p(x|y) is modeled by transforming that generative process, replacing the distributions of the configuration prior with non-parametric densities.

TPE has the advantage of scaling particularly well compared to most Model-Based algorithm which are typically sequential. It does not model however dependencies between hyper-parameters, they
are assumed independent.

For more information on the algorithm, see original papers at:

• Algorithms for Hyper-Parameter Optimization
• Making a Science of Model Search: Hyperparameter Optimizationin Hundreds of Dimensions for Vision Architectures

Storage

To support integration with other tools and services such as MLFlow or Weight & Biases we wrapped our previous database backend with a storage backend. The database backends are now available within the Legagy storage backend. In addition, we now have a backend for Track. The latter is planned to serve as a bridge between Oríon and other experiment management platforms
or services. Track package development is on the ice for now, but contributions are very much welcomed. :)

Drop python 3.5, support 3.8

Although Oríon may still be compatible with python 3.5 we do not maintain it's support anymore. Python 3.8 is now officially supported.

Precision of real dimensions

By default Oríon now rounds hyperparameters to 4 decimals (ex 0.00041239123 would become 0.0004124). The rational is that little variations on continuous hyperparameters typically leads
to little variations in the in objective. When sharing hyperparameters (ex: in publications), one can now share the rounded values with the exact corresponding objectives instead of rounding the hyperparameters after the execution and risk sharing unreproducible results.

Rework of documentation

The documentation has been through a major rework.

• The introduction has been updated to better reflect current features.
• A section Getting Started and an Overview was added.
• The plugins documentation was updated with a new cookiecutter template to create algorithm plugins.
• A minimalist example of scikit-learn was added to serve as the simplest tutorial for Oríon.
• Extensive documentation on contributions has been added.

Detailed list of changes

New features

• Storage Backend with Track and Legacy (Track integration2 #289, remove ExperienceAdapter #302, make trial import lazy to avoid cyclic dependencies #316, Fix track backend API #318)
• Python API ( Refactor experiment builder  #297, Add Space definition in DB #299, Adding Python API #300, Expose trial.params as a dict #305, Handle broken trials with python API #307, Add support for space hierarchical structure #311, Fix formatting of trial with subdict params #312, Rework producer exception for the python API  #355, Validate provided status before updating trial #399, Catch keyboard interrupt in Python API #401)
• Add option to ignore some commandline options of user's script (Dev/ignore cli options #308)
• Add option to ignore code changes in user's script (Dev/ignore code changes #310)
• Add trial hash id that does not include fidelifty
  (useful to resume across fidelities)(Dev/recover experiment #313, Add hash params #322)
• Customize precision (PR: add precision to Real #331)
• Support commandline call where first argument is not user's script (Dev/support entry point #333, Changed documentation to support non-executable python files #338)
• Add support for non-standard -args (Add support for non-standard -args #341)
• Add more helper functions for cmdline client (Add more helper functions for cmdline client #343)
• Hyperband implementation (Hyperband oob #354, hyperband fixes #363, update asha and hyperband #383)
• TPE implementation (Add TPE algorithm into Orion #381, Update tpe test case #387, TPE discrete-categorical-loguniform space support #389)

Breaking changes

• Drop python 3.5 support in favor of 3.8 (Drop python3.5 support and add python3.8 #303)
• Remove deprecated score_handle (Remove score_handle since it is deprecated #315)
• Make global, local and cmdline args coherent (Make global, local and cmdline args coherent #349, Handle producer config in experiment section #404)

Bug Fixes

• Fix EphemeralDB index update after document deletion (Fix EphemeralDB index update after document deletion #306)
• Support small fidelity scales for ASHA (Support small fidelity scales for ASHA #314)
• Algo redefinition should not drop algo config (Algo redefinition should not drop algo config #319)
• Random algorithm will hang if search space is smaller than specified trials number (Random algorithm will hang if search space is smaller than specified trials number #336)
• Inconsistent output in orion status and orion list (Inconsistent output in orion status and orion list #342)
• Use EVC tree trials in producer (Use EVC tree trials in producer #347)
• Update fidelity sample method to honor required n_samples (Update fidelity sample method to honor required n_samples #351)
• Add max_trials to algorithm.is_done (Add max_trials to algorithm.is_done #360)
• Handle trials with corrupted status (Handle trials with corrupted status  #372)
• Making upper bound inclusive (Making upper bound inclusive #373)
• Fix --debug (Fix --debug  #374)
• Round ASHA budget properly (Round ASHA budget properly #402)
• Update Plugins documentation section (Fix mongodb count for v < 3.7 #406)
• Print help when calling orion alone (Print help when calling orion alone #408)

Other improvements

• improve error message for ASHA (improve error message for ASHA #317)
• Print important information at end of worker (Print important information at end of worker #345)
• Add comprehensive error message for branching err (Add comprehensive error message for branching err #346)
• Make global, local and cmdline args coherent (Make global, local and cmdline args coherent #349, Handle producer config in experiment section #404)
• Improve tox configuration (Enabling localized testing via tox #350, Update TOX configuration #398)
• Generic cardinality check in base algorithm (Generic cardinality check in base algorithm #352)
• Avoid duplicates in RandomSearch (Avoid duplicates in RandomSearch #361)
• Handle timeout in DBs (PickledDB in particular) (Handle timeout in DBs (PickledDB in particular) #362)
• Add stress test (Add stress test #364)
• Remove useless deprecation warnings (Remove useless deprecation warnings #376)
• Ignore test files for code coverage (Ignore test files for code coverage #377)
• Show proper yaml syntax in resolve_config deprecation warnings containing None (Show proper yaml syntax in resolve_config deprecation warnings containing None #391)
• Raise exception when no prior is provided (Raise exception when no prior is provided #410)

Documentation improvements

• Add citation section (Add citation section #301, Cite latest version of Oríon by default #379)
• Fix doc for bayesopt's alpha (Fix doc for bayesopt's alpha #309)
• Formatted commands and added a command description (Formatted commands and added a command description #327)
• Clone via HTTPS instead of SSH (Stacktrace when executing oríon without arguments #328)
• Minimalist example with scikit-learn (Minimalist example with scikit-learn #339)
• Full configuration documentation (Make global, local and cmdline args coherent #349)
• Contribution guidelines (Contribution guidelines #356)
• Update developer documentation (Update developer documentation #365, Fix typos in developer documentation #378)
• Release & Packaging (Release & Packaging #371, Add roadmap update to the list of steps for a release #411)
• Update the documentation structure (Update the documentation structure #375)
• Refreshed introduction of Oríon's features (Refreshed introduction of Oríon's features #380, Documentation guidelines for internal links on the README #384)
• Overhaul of the documentation for new comers (Overhaul of the documentation for new comers #385, Promote Oríon's agnosticity #403)
• Add Issue Template (Add issue templates #386)
• Add PR Template (Create pull_request_template.md #388)
• Add doc about label categories (Add doc about label categories #397)
• Update Plugins documentation section (Update Plugins documentation section #405)
• Document Windows compatibility (Document Windows compatibility #409)
• General documentation updates (Updated the roadmap for v0.1.8 #332, Added v0.1.9 to the roadmap #340, Removed empty sections and TODOs #344, Fix broken-link in monitoring #366, Remove relative links #367, Add link to cookiecutter template #370)

[codecov-commenter]

Codecov Report

Merging #414 into master will decrease coverage by 47.16%.
The diff coverage is 86.51%.

@@             Coverage Diff             @@
##           master     #414       +/-   ##
===========================================
- Coverage   94.69%   47.52%   -47.17%     
===========================================
  Files          62       70        +8     
  Lines        9826    13009     +3183     
  Branches      218      322      +104     
===========================================
- Hits         9305     6183     -3122     
- Misses        504     6800     +6296     
- Partials       17       26        +9     

Impacted Files	Coverage Δ
tests/functional/algos/test_algos.py	100.00% <ø> (ø)
...functional/backward_compatibility/test_versions.py	87.50% <ø> (-10.74%)	⬇️
tests/functional/branching/test_branching.py	99.29% <ø> (+0.13%)	⬆️
tests/functional/client/test_cli_client.py	100.00% <ø> (ø)
tests/functional/commands/conftest.py	88.07% <ø> (+1.51%)	⬆️
tests/functional/commands/test_db_commands.py	100.00% <ø> (ø)
tests/functional/commands/test_hunt_command.py	100.00% <ø> (ø)
tests/functional/commands/test_info_command.py	100.00% <ø> (ø)
...ests/functional/commands/test_init_only_command.py	100.00% <ø> (ø)
tests/functional/commands/test_insert_command.py	100.00% <ø> (ø)
... and 101 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d217af...eaa74af. Read the comment docs.