Merge branch 'develop' into develop

.travis.yml

@@ -21,9 +21,6 @@ jobs:
         after_success:
           - tox -e final-coverage
           - tox -e codecov
-        env: TOXENV=py35
-        python: 3.5
-      - <<: *test
         env: TOXENV=py36
         python: 3.6
       - <<: *test
@@ -34,7 +31,12 @@ jobs:
       - <<: *test
         env: TOXENV=py37
         python: 3.7
-        dist: xenial
+        dist: bionic
+        sudo: true
+      - <<: *test
+        env: TOXENV=py38
+        python: 3.8
+        dist: bionic
         sudo: true
       - <<: *test
         env: TOXENV=demo-random

conda/conda_build.sh

@@ -16,9 +16,10 @@ conda update -q conda
 conda info -a
 conda install conda-build anaconda-client
 
-conda build conda --python 3.5
+conda build conda --python 3.6
 conda build conda --python 3.6
 conda build conda --python 3.7
+conda build conda --python 3.8
 
 if [[ -n "${TRAVIS_TAG}" ]]
 then

docs/src/code/client.rst

@@ -6,6 +6,7 @@ Client helper functions
    :maxdepth: 1
    :caption: Modules
 
+   client/cli
    client/experiment
    client/manual
 

docs/src/code/client/cli.rst

@@ -0,0 +1,5 @@
+Commandline client
+==================
+
+.. automodule:: orion.client.cli
+   :members:

docs/src/code/core/utils.rst

@@ -6,6 +6,7 @@ Utilities
    :caption: Utility modules
 
    utils/format_trials
+   utils/format_terminal
 
 .. automodule:: orion.core.utils
    :members:

docs/src/code/core/utils/format_terminal.rst

@@ -0,0 +1,5 @@
+Format terminal
+===============
+
+.. automodule:: orion.core.utils.format_terminal
+   :members:

docs/src/examples/pytorch_a2c_ppo.rst

@@ -45,7 +45,7 @@ we add:
 .. code-block:: python
 
     #!/usr/bin/env python
-    from orion.client import report_results
+    from orion.client import report_objective
 
 
 and then we run
@@ -86,10 +86,7 @@ algorithm:
                                              args.cuda,
                                              eval_env_seeds)
 
-    report_results([dict(
-        name='validation_return',
-        type='objective',
-        value=np.mean(validation_returns))])
+    report_objective(name='validation_return', objective=np.mean(validation_returns))
 
 Now we're ready to go to run orion's hyperparameter optimization!
 

docs/src/examples/pytorch_cifar.rst

@@ -40,10 +40,7 @@ Last line of the main() function
 
         test_error_rate = test(epoch)
 
-    report_results([dict(
-        name='test_error_rate',
-        type='objective',
-        value=test_error_rate)])
+    report_objective(objective, name='test_error_rate')
 
 .. code-block:: bash
 

docs/src/user/api.rst

@@ -20,21 +20,27 @@ Suppose you normally execute your script with the following command
 
 .. code-block:: bash
 
-    $ ./main.py --lr 0.1
+    $ python main.py --lr 0.1
 
 Using the commandline API you can turn your script into a hyper-parameter process by wrapping it
 with Oríon.
 
 .. code-block:: bash
 
-    $ orion hunt -n exp-name ./main.py --lr~'loguniform(1e-5, 1.0)'
+    $ orion hunt -n exp-name python main.py --lr~'loguniform(1e-5, 1.0)'
 
 An experiment called ``exp-name`` will now be created and your script will be called with
-the argument ``--lr`` assigned to values sampled by the optimization algorthm.
+the argument ``--lr`` assigned to values sampled by the optimization algorithm.
 
 Configuration of the algorithm can be done inside a yaml file passed to ``--config`` as described in
 :ref:`Setup Algorithms`.
 
+To return the results to orion, you must add a call to
+:py:func:`orion.client.report_objective(value) <orion.client.cli.report_objective>`
+in your script at the end of the execution.
+
+See :py:mod:`orion.client.cli` for more information on all helper functions available.
+
 
 Python APIs
 ===========

docs/src/user/pytorch.rst

@@ -28,8 +28,9 @@ Adapting the code for Oríon
 To use Oríon with any code we need to do three things
 
 1. make the ``main.py`` file a python executable
-2. import the ``orion.client.report_results`` helper function
-3. call `report_results` on the final objective output to be minimized (e.g. final test error rate)
+2. import the ``orion.client.report_objective`` helper function
+3. call `report_objective` on the final objective output to be minimized (e.g. final test error
+   rate)
 
 After cloning pytorch examples repository, cd to mnist folder:
 
@@ -46,30 +47,28 @@ the ``main.py`` and make it executable, for example:
     $ chmod +x main.py
 
 2. At the top of the file, below the imports, add one line of import the helper function
-``orion.client.report_results()``:
+``orion.client.report_objective()``:
 
 .. code-block:: python
 
-    from orion.client import report_results
+    from orion.client import report_objective
 
 3. We need the test error rate so we're going to add a line to the function ``test()`` to return it
 
 .. code-block:: python
 
     return 1 - (correct / len(test_loader.dataset))
 
-Finally, we get back this test error rate and call ``report_results`` to
-return the final objective value to Oríon. Note that ``report_results`` is meant to
+Finally, we get back this test error rate and call ``report_objective`` to
+return the final objective value to Oríon. Note that ``report_objective`` is meant to
 be called only once because Oríon only looks at 1 ``'objective'`` value per run.
 
 .. code-block:: python
 
         test_error_rate = test(args, model, device, test_loader)
 
-    report_results([dict(
-        name='test_error_rate',
-        type='objective',
-        value=test_error_rate)])
+    report_objective(test_error_rate)
+
 
 Execution
 =========
@@ -156,8 +155,12 @@ validation set.
 Oríon will always **minimize** the objective so make sure you never try to
 optimize something like the accuracy of the model unless you are looking for very very bad models.
 
-You can also ``report_results`` of types ``'gradient'`` and ``'constraint'`` for
-algorithms which require those parameters as well.
+You can also report results of types ``'gradient'`` and ``'constraint'`` for
+algorithms which require those parameters as well, or ``'statistic'`` for metrics
+to be saved with the trial. See
+:py:func:`report_results() <orion.client.cli.report_results>`
+for more details.
+
 
 Debugging
 =========

setup.py

@@ -80,7 +80,7 @@
     'Topic :: Scientific/Engineering',
     'Topic :: Scientific/Engineering :: Artificial Intelligence',
 ] + [('Programming Language :: Python :: %s' % x)
-     for x in '3 3.5 3.6 3.7'.split()]
+     for x in '3 3.6 3.7 3.8'.split()]
 
 if __name__ == '__main__':
     setup(**setup_args)

src/orion/algo/base.py

@@ -21,6 +21,14 @@
 class BaseAlgorithm(object, metaclass=ABCMeta):
     """Base class describing what an algorithm can do.
 
+    Parameters
+    ----------
+    space : `orion.algo.space.Space`
+       Definition of a problem's parameter space.
+    kwargs : dict
+       Tunable elements of a particular algorithm, a dictionary from
+       hyperparameter names to values.
+
     Notes
     -----
     We are using the No Free Lunch theorem's [1]_[3]_ formulation of an
@@ -85,17 +93,6 @@ def observe(self, points, results):
     requires = []
 
     def __init__(self, space, **kwargs):
-        """Declare problem's parameter space and set up algo's hyperparameters.
-
-        Parameters
-        ----------
-        space : `orion.algo.space.Space`
-           Definition of a problem's parameter space.
-        kwargs : dict
-           Tunable elements of a particular algorithm, a dictionary from
-           hyperparameter names to values.
-
-        """
         log.debug("Creating Algorithm object of %s type with parameters:\n%s",
                   type(self).__name__, kwargs)
         self._space = space

src/orion/client/__init__.py

@@ -1,64 +1,30 @@
 # -*- coding: utf-8 -*-
 """
-:mod:`orion.client` -- Helper function for returning results from script
-==========================================================================
+:mod:`orion.client` -- Python API
+=================================
 
 .. module:: client
    :platform: Unix
    :synopsis: Provides functions for communicating with `orion.core`.
 
 """
-import os
-
+from orion.client.cli import (
+    interrupt_trial, report_bad_trial, report_objective, report_results)
 from orion.client.experiment import ExperimentClient
 import orion.core.io.experiment_builder as experiment_builder
 from orion.core.utils.exceptions import RaceCondition
 from orion.core.utils.tests import update_singletons
 from orion.core.worker.producer import Producer
 
 
-IS_ORION_ON = False
-_HAS_REPORTED_RESULTS = False
-RESULTS_FILENAME = os.getenv('ORION_RESULTS_PATH', None)
-if RESULTS_FILENAME and os.path.isfile(RESULTS_FILENAME):
-    import json
-    IS_ORION_ON = True
-
-if RESULTS_FILENAME and not IS_ORION_ON:
-    raise RuntimeWarning("Results file path provided in environmental variable "
-                         "does not correspond to an existing file.")
-
-
-def report_results(data):
-    """Facilitate the reporting of results for a user's script acting as a
-    black-box computation.
-
-    :param data: A dictionary containing experimental results
-
-    .. note:: To be called only once in order to report a final evaluation
-       of a particular trial.
-
-    .. note:: In case that user's script is not running in a orion's context,
-       this function will act as a Python `print` function.
-
-    .. note:: For your own good, this can be called **only once**.
-
-    """
-    global _HAS_REPORTED_RESULTS  # pylint:disable=global-statement
-    if _HAS_REPORTED_RESULTS:
-        raise RuntimeWarning("Has already reported evaluation results once.")
-    if IS_ORION_ON:
-        with open(RESULTS_FILENAME, 'w') as results_file:
-            json.dump(data, results_file)
-    else:
-        print(data)
-    _HAS_REPORTED_RESULTS = True
+__all__ = ['interrupt_trial', 'report_bad_trial', 'report_objective', 'report_results']
 
 
 # pylint: disable=too-many-arguments
-def create_experiment(name, version=None, space=None, algorithms=None,
-                      strategy=None, max_trials=None, storage=None, branching=None,
-                      working_dir=None):
+def create_experiment(
+        name, version=None, space=None, algorithms=None,
+        strategy=None, max_trials=None, storage=None, branching=None,
+        working_dir=None):
     """Create an experiment
 
     There is 2 main scenarios