Merge pull request #363 from donglinjy/hyperband-enhance

hyperband fixes

docs/src/code/algo.rst

@@ -14,4 +14,5 @@ TODO
    algo/space
    algo/base
    algo/random
+   algo/hyperband
    algo/asha

docs/src/code/algo/hyperband.rst

@@ -0,0 +1,5 @@
+Hyperband Algorithm
+===================
+
+.. automodule:: orion.algo.hyperband
+   :members:

docs/src/user/algorithms.rst

@@ -58,6 +58,60 @@ Configuration
 
 Seed for the random number generator used to sample new trials. Default is ``None``.
 
+.. _hyperband-algorithm:
+
+Hyperband
+---------
+
+`Hyperband`_ extends the `SuccessiveHalving`_ algorithm by providing a way to exploit a
+fixed budget with different number of configurations for ``SuccessiveHalving`` algorithm to
+evaluate. Each run of ``SuccessiveHalving`` will be defined as a ``bracket`` in Hyperband.
+Hyperband requires two inputs (1) ``R``, the maximum amount of resource that can be allocated
+to a single configuration, and (2) ``eta``, an input that controls the proportion of
+configurations discarded in each round of SuccessiveHalving.
+
+To use Hyperband in Oríon, you must specify one parameter with ``fidelity(low, high, base)``
+as the prior, ``low`` will be ignored, ``high`` will be taken as the maximum resource ``R``
+and ``base`` will be taken as the reduction factor ``eta``.
+
+Number of epochs usually can be used as the resource but the algorithm is generic and can be
+applied to any multi-fidelity setting. That is, you can use training time, specifying the
+fidelity with ``--epochs~fidelity(low=1, high=81, base=3)``
+(assuming your script takes this argument in commandline),
+but you could also use other fidelity
+such as dataset size ``--dataset-size~fidelity(low=500, high=50000)``
+(assuming your script takes this argument and adapt dataset size accordingly).
+
+
+.. _SuccessiveHalving: https://arxiv.org/abs/1502.07943
+
+.. note::
+
+   Current implementation does not support more than one fidelity dimension.
+
+Configuration
+~~~~~~~~~~~~~
+
+.. code-block:: yaml
+
+    algorithms:
+       hyperband:
+          seed: null
+          repetitions: 1
+
+
+``seed``
+
+Seed for the random number generator used to sample new trials. Default is ``None``.
+
+``repetitions``
+
+Number of executions for Hyperband. A single execution of Hyperband takes a finite
+budget of ``(log(R)/log(eta) + 1) * (log(R)/log(eta) + 1) * R``, and ``repetitions`` allows you
+to run multiple executions of Hyperband. Default is ``numpy.inf`` which means to run Hyperband
+until no new trials can be suggested.
+
+
 .. _ASHA:
 
 ASHA