diff --git a/README.md b/README.md index e7bf8826a7b..ce08317fbc0 100644 --- a/README.md +++ b/README.md @@ -17,19 +17,33 @@ Adaptive experimentation is the machine-learning guided process of iteratively exploring a (possibly infinite) parameter space in order to identify optimal configurations in a resource-efficient manner. Ax currently supports Bayesian optimization and bandit optimization as exploration strategies. Bayesian -optimization in Ax is powered by [BoTorch](https://github.com/facebookexternal/botorch), -a modern library for Bayesian optimization research built on PyTorch. +optimization in Ax is powered by +[BoTorch](https://github.com/facebookexternal/botorch), a modern library for +Bayesian optimization research built on PyTorch. For full documentation and tutorials, see the [Ax website](https://ax.dev) ## Why Ax? -* **Versatility**: Ax supports different kinds of experiments, from dynamic ML-assisted A/B testing, to hyperparameter optimization in machine learning. -* **Customization**: Ax makes it easy to add new modeling and decision algorithms, enabling research and development with minimal overhead. -* **Production-completeness**: Ax comes with storage integration and ability to fully save and reload experiments. -* **Support for multi-modal and constrained experimentation**: Ax allows for running and combining multiple experiments (e.g. simulation with a real-world "online" A/B test) and for constrained optimization (e.g. improving classification accuracy without significant increase in resource-utilization). -* **Efficiency in high-noise setting**: Ax offers state-of-the-art algorithms specifically geared to noisy experiments, such as simulations with reinforcement-learning agents. -* **Ease of use**: Ax includes 3 different APIs that strike different balances between lightweight structure and flexibility. Using the most concise Loop API, a whole optimization can be done in just one function call. The Service API integrates easily with external schedulers. The most elaborate Developer API affords full algorithm customization and experiment introspection. +- **Versatility**: Ax supports different kinds of experiments, from dynamic + ML-assisted A/B testing, to hyperparameter optimization in machine learning. +- **Customization**: Ax makes it easy to add new modeling and decision + algorithms, enabling research and development with minimal overhead. +- **Production-completeness**: Ax comes with storage integration and ability to + fully save and reload experiments. +- **Support for multi-modal and constrained experimentation**: Ax allows for + running and combining multiple experiments (e.g. simulation with a real-world + "online" A/B test) and for constrained optimization (e.g. improving + classification accuracy without significant increase in resource-utilization). +- **Efficiency in high-noise setting**: Ax offers state-of-the-art algorithms + specifically geared to noisy experiments, such as simulations with + reinforcement-learning agents. +- **Ease of use**: Ax includes 3 different APIs that strike different balances + between lightweight structure and flexibility. The Service API (recommended + for the vast majority of use-cases) provides an extensive, robust, and + easy-to-use interface to Ax; the Loop API enables particularly concise usage; + and the Developer API enables advanced experimental and methodological + control. ## Getting Started @@ -63,20 +77,22 @@ artificial evaluation function): ## Installation ### Requirements + You need Python 3.10 or later to run Ax. The required Python dependencies are: -* [botorch](https://www.botorch.org) -* jinja2 -* pandas -* scipy -* sklearn -* plotly >=2.2.1 +- [botorch](https://www.botorch.org) +- jinja2 +- pandas +- scipy +- sklearn +- plotly >=2.2.1 ### Stable Version #### Installing via pip + We recommend installing Ax via pip (even if using Conda environment): ``` @@ -84,26 +100,35 @@ conda install pytorch torchvision -c pytorch # OSX only (details below) pip install ax-platform ``` -Installation will use Python wheels from PyPI, available for [OSX, Linux, and Windows](https://pypi.org/project/ax-platform/#files). +Installation will use Python wheels from PyPI, available for +[OSX, Linux, and Windows](https://pypi.org/project/ax-platform/#files). -*Note*: Make sure the `pip` being used to install `ax-platform` is actually the one from the newly created Conda environment. -If you're using a Unix-based OS, you can use `which pip` to check. +_Note_: Make sure the `pip` being used to install `ax-platform` is actually the +one from the newly created Conda environment. If you're using a Unix-based OS, +you can use `which pip` to check. -*Recommendation for MacOS users*: PyTorch is a required dependency of BoTorch, and can be automatically installed via pip. -However, **we recommend you [install PyTorch manually](https://pytorch.org/get-started/locally/#anaconda-1) before installing Ax, using the Anaconda package manager**. -Installing from Anaconda will link against MKL (a library that optimizes mathematical computation for Intel processors). -This will result in up to an order-of-magnitude speed-up for Bayesian optimization, as at the moment, installing PyTorch from pip does not link against MKL. +_Recommendation for MacOS users_: PyTorch is a required dependency of BoTorch, +and can be automatically installed via pip. However, **we recommend you +[install PyTorch manually](https://pytorch.org/get-started/locally/#anaconda-1) +before installing Ax, using the Anaconda package manager**. Installing from +Anaconda will link against MKL (a library that optimizes mathematical +computation for Intel processors). This will result in up to an +order-of-magnitude speed-up for Bayesian optimization, as at the moment, +installing PyTorch from pip does not link against MKL. -If you need CUDA on MacOS, you will need to build PyTorch from source. Please consult the PyTorch installation instructions above. +If you need CUDA on MacOS, you will need to build PyTorch from source. Please +consult the PyTorch installation instructions above. #### Optional Dependencies To use Ax with a notebook environment, you will need Jupyter. Install it first: + ``` pip install jupyter ``` If you want to store the experiments in MySQL, you will need SQLAlchemy: + ``` pip install SQLAlchemy ``` @@ -116,7 +141,8 @@ You can install the latest (bleeding edge) version from Git. First, see recommendation for installing PyTorch for MacOS users above. -At times, the bleeding edge for Ax can depend on bleeding edge versions of BoTorch (or GPyTorch). We therefore recommend installing those from Git as well: +At times, the bleeding edge for Ax can depend on bleeding edge versions of +BoTorch (or GPyTorch). We therefore recommend installing those from Git as well: ``` pip install git+https://github.com/cornellius-gp/linear_operator.git @@ -141,9 +167,11 @@ To support plotly-based plotting in newer Jupyter notebook versions pip install "notebook>=5.3" "ipywidgets==7.5" ``` -[See Plotly repo's README](https://github.com/plotly/plotly.py#jupyter-notebook-support) for details and JupyterLab instructions. +[See Plotly repo's README](https://github.com/plotly/plotly.py#jupyter-notebook-support) +for details and JupyterLab instructions. If storing Ax experiments via SQLAlchemy in MySQL or SQLite: + ``` pip install git+https://github.com/facebook/Ax.git#egg=ax-platform[mysql] ``` @@ -152,13 +180,18 @@ pip install git+https://github.com/facebook/Ax.git#egg=ax-platform[mysql] ### Getting help -Please open an issue on our [issues page](https://github.com/facebook/Ax/issues) with any questions, feature requests or bug reports! If posting a bug report, please include a minimal reproducible example (as a code snippet) that we can use to reproduce and debug the problem you encountered. +Please open an issue on our [issues page](https://github.com/facebook/Ax/issues) +with any questions, feature requests or bug reports! If posting a bug report, +please include a minimal reproducible example (as a code snippet) that we can +use to reproduce and debug the problem you encountered. ### Contributing See the [CONTRIBUTING](CONTRIBUTING.md) file for how to help out. -When contributing to Ax, we recommend cloning the [repository](https://github.com/facebook/Ax) and installing all optional dependencies: +When contributing to Ax, we recommend cloning the +[repository](https://github.com/facebook/Ax) and installing all optional +dependencies: ``` pip install git+https://github.com/cornellius-gp/linear_operator.git @@ -175,8 +208,8 @@ See recommendation for installing PyTorch for MacOS users above. The above example limits the cloned directory size via the [`--depth`](https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt) -argument to `git clone`. If you require the entire commit history you may remove this -argument. +argument to `git clone`. If you require the entire commit history you may remove +this argument. ## License diff --git a/docs/api.md b/docs/api.md index d56a36be036..c517537c3d2 100644 --- a/docs/api.md +++ b/docs/api.md @@ -21,9 +21,10 @@ From most lightweight to fullest functionality, our APIs are: optimization is complete. **Use this API only for the simplest use cases where running a single trial is fast and only one trial should be running at a time.** -- **Service API** ([tutorial](/tutorials/gpei_hartmann_service.html)) can be - used as a lightweight service for parameter-tuning applications where trials - might be evaluated in parallel and data is available asynchronously (e.g. +- **[RECOMMENDED] Service API** + ([tutorial](/tutorials/gpei_hartmann_service.html)) can be used as a + lightweight service for parameter-tuning applications where trials might be + evaluated in parallel and data is available asynchronously (e.g. hyperparameter or simulation optimization). It requires little to no knowledge of Ax data structures and easily integrates with various schedulers. In this mode, Ax suggests one-[arm](glossary.md#arm) trials to be evaluated by the diff --git a/docs/trial-evaluation.md b/docs/trial-evaluation.md index 38b0d1e2a5b..7b675058948 100644 --- a/docs/trial-evaluation.md +++ b/docs/trial-evaluation.md @@ -3,11 +3,21 @@ id: trial-evaluation title: Trial Evaluation --- -There are 3 paradigms for evaluating [trials](glossary.md#trial) in Ax. -Note: ensure that you are using the [appropriate type of trials](core.md#trial-vs-batched-trial) for your experiment, before proceeding to trial evaluation. - -## Service API -The [```AxClient```](/api/service.html#module-ax.service.ax_client) exposes [```get_next_trial```](/api/service.html#ax.service.ax_client.AxClient.get_next_trial), as well as [```complete_trial```](/api/service.html#ax.service.ax_client.AxClient.complete_trial). The user is responsible for evaluating the trial parameters and passing the results to [```complete_trial```](/api/service.html#ax.service.ax_client.AxClient.complete_trial). +There are 3 paradigms for evaluating [trials](glossary.md#trial) in Ax. Note: +ensure that you are using the +[appropriate type of trials](core.md#trial-vs-batched-trial) for your +experiment, before proceeding to trial evaluation. + +## [RECOMMENDED] Service API + +The Service API [`AxClient`](/api/service.html#module-ax.service.ax_client) +exposes +[`get_next_trial`](/api/service.html#ax.service.ax_client.AxClient.get_next_trial), +as well as +[`complete_trial`](/api/service.html#ax.service.ax_client.AxClient.complete_trial). +The user is responsible for evaluating the trial parameters and passing the +results to +[`complete_trial`](/api/service.html#ax.service.ax_client.AxClient.complete_trial). ```python ... @@ -19,16 +29,29 @@ for i in range(25): ### Evaluating Trial Parameters -In the Service API, the [```complete_trial```](/api/service.html#ax.service.ax_client.AxClient.complete_trial) requires `raw_data` evaluated from the parameters suggested by [```get_next_trial```](/api/service.html#ax.service.ax_client.AxClient.get_next_trial). +In the Service API, the +[`complete_trial`](/api/service.html#ax.service.ax_client.AxClient.complete_trial) +method requires `raw_data` evaluated from the parameters suggested by +[`get_next_trial`](/api/service.html#ax.service.ax_client.AxClient.get_next_trial). The data can be in the form of: + - A dictionary of metric names to tuples of (mean and [SEM](glossary.md#sem)) - A single (mean, SEM) tuple - A single mean -In the second case, Ax will assume that the mean and the SEM are for the experiment objective (if the evaluations are noiseless, simply provide an SEM of 0.0). In the third case, Ax will assume that observations are corrupted by Gaussian noise with zero mean and unknown SEM, and infer the SEM from the data (this is equivalent to specifying an SEM of None). Note that if the observation noise is non-zero (either provided or inferred), the "best arm" suggested by Ax may not always be the one whose evaluation returned the best observed value (as the "best arm" is selected based on the model-predicted mean). +In the second case, Ax will assume that the mean and the SEM are for the +experiment objective (if the evaluations are noiseless, simply provide an SEM of +0.0). In the third case, Ax will assume that observations are corrupted by +Gaussian noise with zero mean and unknown SEM, and infer the SEM from the data +(this is equivalent to specifying an SEM of None). Note that if the observation +noise is non-zero (either provided or inferred), the "best arm" suggested by Ax +may not always be the one whose evaluation returned the best observed value (as +the "best arm" is selected based on the model-predicted mean). -For example, this evaluation function computes mean and SEM for [Hartmann6](https://www.sfu.ca/~ssurjano/hart6.html) function and for the L2-norm. We return `0.0` for SEM since the observations are noiseless: +For example, this evaluation function computes mean and SEM for +[Hartmann6](https://www.sfu.ca/~ssurjano/hart6.html) function and for the +L2-norm. We return `0.0` for SEM since the observations are noiseless: ```python from ax.utils.measurement.synthetic_functions import hartmann6 @@ -38,7 +61,9 @@ def hartmann_evaluation_function(parameterization): return {"hartmann6": (hartmann6(x), 0.0), "l2norm": (np.sqrt((x ** 2).sum()), 0.0)} ``` -This function computes just the objective mean and SEM, assuming the [Branin](https://www.sfu.ca/~ssurjano/branin.html) function is the objective of the experiment: +This function computes just the objective mean and SEM, assuming the +[Branin](https://www.sfu.ca/~ssurjano/branin.html) function is the objective of +the experiment: ```python from ax.utils.measurement.synthetic_functions import branin @@ -62,16 +87,32 @@ def branin_evaluation_function_unknown_sem(parameterization): ``` ## Loop API -The [```optimize```](/api/service.html#ax.service.managed_loop.optimize) function requires an `evaluation_function`, which accepts parameters and returns raw data in the format described above. -It can also accept a `weight` parameter, a nullable `float` representing the fraction of available data on which the parameterization should be evaluated. For example, this could be a downsampling rate in case of hyperparameter optimization (what portion of data the ML model should be trained on for evaluation) or the percentage of users exposed to a given configuration in A/B testing. This weight is not used in unweighted experiments and defaults to `None`. + +The [`optimize`](/api/service.html#ax.service.managed_loop.optimize) function +requires an `evaluation_function`, which accepts parameters and returns raw data +in the format described above. It can also accept a `weight` parameter, a +nullable `float` representing the fraction of available data on which the +parameterization should be evaluated. For example, this could be a downsampling +rate in case of hyperparameter optimization (what portion of data the ML model +should be trained on for evaluation) or the percentage of users exposed to a +given configuration in A/B testing. This weight is not used in unweighted +experiments and defaults to `None`. ## Developer API -The Developer API is supported by the [```Experiment```](/api/core.html#module-ax.core.experiment) class. In this paradigm, the user specifies: - * [`Runner`](../api/core.html#ax.core.runner.Runner): Defines how to deploy the experiment. - * List of [`Metrics`](../api/core.html#ax.core.metric.Metric): Each defines how to compute/fetch data for a given objective or outcome. +The Developer API is supported by the +[`Experiment`](/api/core.html#module-ax.core.experiment) class. In this +paradigm, the user specifies: + +- [`Runner`](../api/core.html#ax.core.runner.Runner): Defines how to deploy the + experiment. +- List of [`Metrics`](../api/core.html#ax.core.metric.Metric): Each defines how + to compute/fetch data for a given objective or outcome. + +The experiment requires a `generator_run` to create a new trial or batch trial. +A generator run can be generated by a model. The trial then has its own `run` +and `mark_complete` methods. -The experiment requires a `generator_run` to create a new trial or batch trial. A generator run can be generated by a model. The trial then has its own `run` and `mark_complete` methods. ```python ... sobol = Models.SOBOL(exp.search_space) @@ -90,7 +131,12 @@ for i in range(15): ### Custom Metrics -Similar to a trial evaluation in the Service API, a custom metric computes a mean and SEM for each arm of a trial. However, the metric's `fetch_trial_data` method will be called automatically by the experiment's [```fetch_data```](/api/core.html#ax.core.base_trial.BaseTrial.fetch_data) method. If there are multiple objectives or outcomes that need to be optimized for, each needs its own metric. +Similar to a trial evaluation in the Service API, a custom metric computes a +mean and SEM for each arm of a trial. However, the metric's `fetch_trial_data` +method will be called automatically by the experiment's +[`fetch_data`](/api/core.html#ax.core.base_trial.BaseTrial.fetch_data) method. +If there are multiple objectives or outcomes that need to be optimized for, each +needs its own metric. ```python class MyMetric(Metric): @@ -110,11 +156,23 @@ class MyMetric(Metric): ### Adding Your Own Runner -In order to control how the experiment is deployed, you can add your own runner. To do so, subclass [`Runner`](../api/core.html#ax.core.runner.Runner) and implement the [`run`](../api/core.html#ax.core.runner.Runner.run) method and [`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required) property. - -The [`run`](../api/core.html#ax.core.runner.Runner.run) method accepts a [`Trial`](../api/core.html#ax.core.trial.Trial) and returns a JSON-serializable dictionary of any necessary tracking info to fetch data later from this external system. A unique identifier or name for this trial in the external system should be stored in this dictionary with the key `"name"`, and this can later be accessed via `trial.deployed_name`. - -The [`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required) indicates whether the trial requires an intermediate staging period before evaluation begins. This property returns False by default. +In order to control how the experiment is deployed, you can add your own runner. +To do so, subclass [`Runner`](../api/core.html#ax.core.runner.Runner) and +implement the [`run`](../api/core.html#ax.core.runner.Runner.run) method and +[`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required) +property. + +The [`run`](../api/core.html#ax.core.runner.Runner.run) method accepts a +[`Trial`](../api/core.html#ax.core.trial.Trial) and returns a JSON-serializable +dictionary of any necessary tracking info to fetch data later from this external +system. A unique identifier or name for this trial in the external system should +be stored in this dictionary with the key `"name"`, and this can later be +accessed via `trial.deployed_name`. + +The +[`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required) +indicates whether the trial requires an intermediate staging period before +evaluation begins. This property returns False by default. An example implementation is given below: diff --git a/website/tutorials.json b/website/tutorials.json index 36a282b0ead..de2e5e579e7 100644 --- a/website/tutorials.json +++ b/website/tutorials.json @@ -1,12 +1,12 @@ { "API Comparison": [ { - "id": "gpei_hartmann_loop", - "title": "Loop API" + "id": "gpei_hartmann_service", + "title": "[RECOMMENDED] Service API" }, { - "id": "gpei_hartmann_service", - "title": "Service API" + "id": "gpei_hartmann_loop", + "title": "Loop API" }, { "id": "gpei_hartmann_developer",