Bayesian optimization, a technique used to optimize functions, is dominated by computer scientists and optimization researchers, not statisticians. This is surprising, given that Bayesian optimization is a Bayesian method. Most research in this area focuses on developing new acquisition strategies with no-regret guarantees, rather than improving the statistical modeling of the objective function. The Gaussian Process surrogate model, a crucial component of Bayesian optimization, is often treated as a fixed black box, with little attention paid to prior misspecification, posterior consistency, or model calibration.
This division between the computer science and statistics communities might be due to a deeper epistemic difference. However, the statistical structure of the surrogate model is crucial to its performance, and this is an area where statisticians can contribute. The convergence behavior of Bayesian optimization is governed by how quickly the Gaussian Process posterior concentrates around the true function, which is controlled directly by the choice of kernel. Regret bounds, such as those in the canonical GP-UCB framework, are driven by the maximal information gain, which depends on the eigenvalue decay of the kernel’s integral operator and the RKHS norm of the latent function.
In practice, most Bayesian optimization implementations use generic Matern or RBF kernels, regardless of the structure of the objective. These kernels impose strong and often inappropriate assumptions, such as stationarity, isotropy, and homogeneity of smoothness. Domain knowledge is rarely incorporated into the kernel, although this can dramatically reduce the effective complexity of the hypothesis space and accelerate learning.
This raises an important question: is there an opportunity for statistical expertise to improve both the theory and practice of Bayesian optimization? By incorporating statistical knowledge into the development of Bayesian optimization methods, we may be able to create more efficient and effective algorithms.