Which acquisition function should I use to iteratively improve model accuracy rather than optimize a target value?

[sgbaird]

It took me a while to track down the issue and comment related to this, so I wanted to surface this in a new issue and immediately close it for better searchability.

I think there are two approaches. One option is to reformulate it as a problem where you minimize an error metric instead of a target value; this requires having a test set or some form of cross-validation that you trust. The other option which seems preferable, especially if one is adding new data or starting from scratch takes from @Balandat's comment in #460 (comment) to use the botorch.acquisition.active_learning.qNegIntegratedPosteriorVariance (qNIPV) acquisition function:

Yeah qNIPV is agnostic to the direction, the goal is to minimize a global measure of uncertainty of the model, so there is no better or worse w.r.t. the function values.

If I understand it correctly, this implies no exploitation, but rather pure exploration.

[Balandat]

If I understand it correctly, this implies no exploitation, but rather pure exploration.

correct

[iandoxsee]

Jumping in here to ask a follow-on: Are there any recommended acquisition functions within Botorch for purely exploration-based active learning of multi-objective spaces? qNIPV appears limited to single-objective, if I'm understanding correctly.

My use case is in real-world experimentation, where I would typically use a design of experiments (DoE) approach to map out some n-dimensional variable space as accurately as possible under some experimental budget. This DoE approach works great, but is labor-intensive, so an active learning approach is appealing. For example, some of my variables may end up having little/no effect on one or more of my objectives, but they contribute equally to my (predefined) experimental burden using DoE.

Thanks all!

[sgbaird]

@iandoxsee great question about multi-objective equivalents. I'm sure you'll get an answer from one of the devs, but I"ll include my thoughts, too. For a multi-objective problem, you still need some way to encode trade-offs between objectives. I could imagine a sum of qNIPV values across each of the objectives, or some variation on an expected hypervolume improvement with one axis per qNIPV. Probably something off about the latter thought, though.

Also, curious to hear why your use case is something where minimization of global uncertainty is more appealing than optimizing the objectives directly.

[iandoxsee]

@sgbaird Thanks for your thoughts on this.

In my use case, there is a need to fully understand the space to ensure that objectives are consistently above/below individually-specified thresholds throughout the entire space. It's a manufacturing setting, so the ranges for this active learning experiment represent the expected routine variation that we wish to accommodate.

My impression is that minimizing global uncertainty is similar to the goal of our DoE experimentation: Identifying as good a model as possible, given some experimental budget. Optimizing our objectives directly is a problem elsewhere in the development timeline, but the devs' code has been tremendously helpful already in this regard :)

[sgbaird]

@iandoxsee thanks for clarifying! Also, what is your experimental budget (100's, 1000's, etc.), and if you have any existing data, how much, and how many design variables will you be looking at? Of these, how many are categorical vs. numerical? Approximate is fine.

[iandoxsee]

Good points, should have included that already!

Budget of <50 experiments, ideally <20-30 to be competitive with most of my DoE studies. Existing data on the order of ~3-5 experiments, 3-8 variables which will all be numerical, and 1-4 responses (1 should be already satisfied by qNIPV).

[eytan]

@iandoxsee , you may be interested in https://botorch.org/tutorials/constraint_active_search which aims to cover all designs which exceed some pre-specified threshold across multiple outcomes (constraints).

[iandoxsee]

Hi @eytan, this looks like a really exciting approach that directly gets to what we're after. Thanks for your input!