BO as Assistant: Using Bayesian Optimization for Asynchronously Generating Design Suggestions
National Institute of Advanced Industrial Science and Technology (AIST)
National Institute of Advanced Industrial Science and Technology (AIST)
Bayesian optimization (BO) is a powerful technique characterized by its intelligent sampling. Previous work has used BO in human (designer)-in-the-loop frameworks for efficiently solving design problems. However, such frameworks are not flexible for designers, which may reduce agency and creativity.
We use BO as an assistant (rather than in human-in-the-loop), where designers can take the initiative in the design process while also benefiting from BO's intelligent sampling strategy. The system generates design suggestions using BO just by observing slider manipulation (without requiring explicit input).
The proposed framework is domain-agnostic. We demonstrate its generality by applying it to diverse design domains in computer graphics.
Many design tasks involve parameter adjustment, and designers often struggle to find desirable parameter value combinations by manipulating sliders back and forth. For such a multi-dimensional search problem, Bayesian optimization (BO) is a promising technique because of its intelligent sampling strategy; in each iteration, BO samples the most effective points considering both exploration (i.e., prioritizing unexplored regions) and exploitation (i.e., prioritizing promising regions), enabling efficient searches. However, existing BO-based design frameworks take the initiative in the design process and thus are not flexible enough for designers to freely explore the design space using their domain knowledge. In this paper, we propose a novel design framework, BO as Assistant, which enables designers to take the initiative in the design process while also benefiting from BO's sampling strategy. The designer can manipulate sliders as usual; the system monitors the slider manipulation to automatically estimate the design goal on the fly and then asynchronously provides unexplored-yet-promising suggestions using BO's sampling strategy. The designer can choose to use the suggestions at any time. This framework uses a novel technique to automatically extract the necessary information to run BO by observing slider manipulation without requesting additional inputs. Our framework is domain-agnostic, demonstrated by applying it to photo color enhancement, 3D shape design for personal fabrication, and procedural material design in computer graphics.
Figure: Concept of BO as Assistant, a framework for assisting designers in finding appropriate design parameter values by using Bayesian optimization (BO) techniques. The system (1) monitors the designer's slider manipulation, (2) automatically estimates the designer's design goal, and (3) asynchronously provides suggestions sampled using BO's strategy. The designer can choose to use suggestions or ignore them.
Yuki Koyama and Masataka Goto. 2022. BO as Assistant: Using Bayesian Optimization for Asynchronously Generating Design Suggestions. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22), pp.77:1–77:14.
DOI: 10.1145/3526113.3545664Our Bayesian optimization implementation is reproducible by using an open-source library and following the instruction below.
Our framework uses preferential Bayesian optimization (PBO), a variant of BO that runs with relative preferential information as input. First, you need to install a Python library that implements PBO, sequential-line-search, to your Python environment. This library can be installed using pip, but before that, you need to install CMake and Eigen.
macOS:
brew install cmake eigen
Ubuntu 18.04:
sudo apt install cmake libeigen3-dev
Once Cmake and Eigen are installed, then the library (v0.3 is the version used in the paper) can be installed by the pip command.
pip install git+https://github.com/yuki-koyama/sequential-line-search.git@v0.3
In your code, first you need to import the relevant libraries.
import pySequentialLineSearch as pysls
import numpy as np
PreferentialBayesianOptimizer is the main class. As written in the paper, we use Gaussian Process Upper Confidence Bound (GP-UBC). The GP-UBC hyperparameter value is 0.5. We do not use the maximum a posteriori (MAP) estimation of kernel hyperparameters.
TYPE = pysls.AcquisitionFuncType.GaussianProcessUpperConfidenceBound
NUM_SUGGESTIONS = 3
NUM_PARAMS = 12
optimizer = pysls.PreferentialBayesianOptimizer(
num_dims=NUM_PARAMS,
use_map_hyperparams=False,
acquisition_func_type=TYPE,
num_options=NUM_SUGGESTIONS + 1)
optimizer.set_gaussian_process_upper_confidence_bound_hyperparam(0.50)
Note that the value of the num_options argument should be the number of the suggestions plus one as above.
At the beginning of the design process, there is no data available. So, we generate initial suggestions just randomly.
suggestions = np.random.rand(NUM_SUGGESTIONS, NUM_PARAMS)
This is the main part of the PBO process. First, you need to extract a new preferential data entry from slider observation. Then, feed the data to the optimizer to update the internal model, and ask the optimizer to generate new suggestions. Finally, you need to retrieve the suggestions (and provide them to the user).
# Extract preferential feedback data
preferred_option, other_options = extract_preference_data(...)
# Submit the preferential data and update the internal model
optimizer.submit_custom_feedback_data(preferred_option, other_options)
# Calculate new suggestions
optimizer.determine_next_query()
# Retrieve the options sampled by Bayesian optimization
for i in range(NUM_SUGGESTIONS):
suggestions[i] = optimizer.get_current_options()[i + 1]
Note that optimizer.get_current_options()[0] always stores the "current best" option and so can be just ignored in this work.
Kenta Yamamoto, Yuki Koyama, and Yoichi Ochiai
UIST 2022
Yuki Koyama, Issei Sato, and Masataka Goto
ACM Transactions on Graphics (SIGGRAPH 2020)
Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi
ACM Transactions on Graphics (SIGGRAPH 2017)