TypeError: predict() got an unexpected keyword argument 'return_mean_grad'

Question

Using scikit-optimize and scikit-learn, I get the following error on calling the gp_minimize function:

TypeError: predict() got an unexpected keyword argument 'return_mean_grad'

I think it may be some version incompatability but they are both latest versions (according to pip)

• scikit-optimize 0.9.0
• scikit-learn 1.2.0

What could be the problem?

Sample code I am using

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern
from skopt import gp_minimize

# Define the kernel for the Gaussian process
kernel = Matern(nu=2.5)

# Initialize the Gaussian process regressor
gpr = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=10)

# Define the objective function
def f(x):
    x1, x2, x3 = x
    return -x1**2 - x2**2 - x3**2

# Define the variable bounds
bounds = [(0, 1), (0, 1), (0, 1)]

# Perform the optimization
res = gp_minimize(f, bounds, n_calls=20, random_state=0, verbose=True, n_random_starts=10, acq_func="EI", base_estimator=gpr)

# Print the optimal variables and function value
print("x1 = {:.4f}, x2 = {:.4f}, x3 = {:.4f}, f(x) = {:.4f}".format(res.x[0], res.x[1], res.x[2], res.fun))

Answer 1

you're using 2 packages Scikit-Optimize & Scikit-Learn - I can advise you to learn the hierarchy of classes in each of them as well as OOP principles in general, including SOLID principles for your project architecture design. You have tried to do Dependency Invertion having not created the proper implementation, if you really needed it. BTW, I see no need to write such useless wrapper, better use existing classes & methods correct!

• you CANNOT use GaussianProcessRegressor from sklearn as best_estimator for gp_minimize from skopt :

Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions.

you can use Optimizer in Regressor, but not vc.vs. ! - as so as Regressor is assigned with param Optimizer, you cannot assign it to optimizer(minimizer) from another package again, even if its returned values could suit that skopt.minimizer -- it would provide a cyclic Dependency - that is the horror in development for all times.

For the Regressor you can even create your own custom optimizer to use it as any other optimizer (at least given by default), but in any case you will need to provide correct Black-Box function's return values [in order they could be utilized by Regressor as params], therefore can build-in black-box function into yourOptimizer function (e.g.), concerning your example:

import numpy as np
from sklearn.datasets import make_friedman2
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern
from skopt import gp_minimize

# Define the kernel for the Gaussian process
kernel = Matern(nu=2.5)
# Define the variable bounds
bounds = [(0, 1), (0, 1), (0, 1)]
initial_theta= [1.,1.,1.]

X, y = make_friedman2(n_samples=500, noise=0, random_state=0)

# Define the objective function
def black_box_function(x):
    x1, x2, x3 = x
    y= -x1**2 - x2**2 - x3**2
    return y

def my_opt(black_box_function, initial_theta, bounds):
        rng = np.random.RandomState(0)
        theta_opt, func_min = \
            initial_theta, black_box_function(initial_theta, eval_gradient=False)
        for _ in range(50):
            theta = np.atleast_1d(rng.uniform(np.maximum(-2, bounds[:, 0]),
                                              np.minimum(1, bounds[:, 1])))
            f = black_box_function(theta, eval_gradient=False)
            if f < func_min:
                theta_opt, func_min = theta, f
        return theta_opt, func_min

# Perform the optimization
opt = gp_minimize(black_box_function,  bounds,  n_calls=20, random_state=1, verbose=True, n_random_starts=10, acq_func="EI")
print(opt)
# Print the optimal variables and function value
print("x1 = {:.4f}, x2 = {:.4f}, x3 = {:.4f}, f(x) = {:.4f}".format(opt.x[0], opt.x[1], opt.x[2], opt.fun))
print('======\n')

# Initialize the Gaussian process regressor
gpr = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=10, normalize_y=False, optimizer=my_opt)
gpr.fit(X, y)
print("score: ", gpr.score(X, y)) 

But for Bayesian optimization you can even use bayesian-optimization package if you do not know how to use either sklearn or skopt - skopt-bayes-opt, but not spoiling the logics of MLE calculations with faulty inversions of operations & ugly architecture of dependencies.

Thus, your code's problems:

• do not mix packages mindless
• learn OOP: classes, inheritance vs. delegation
• follow hierarchy
• explore docs of package/library for its methods & params of methods [best_estimator required objects, as you wanted], always pay attention to return_value to suit it for the next operation usage
• provide correct input (incl. params) to any function to recieve correct result
• remember, that optimization is the tool for the process of any regression (either mle, or ols)
• p.s. supply good examles (not mindless) - Applied math - first, software packages - last... and you will be lucky with your optimization

Answer 2

though I have already posted all my dislikes about the architecture of packages & interactions between them - but still CORRECTION to your code found: acq_optimizer="sampling" to gp_minimize() (default was: "lbfgs"):

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern
from skopt import gp_minimize

# Define the kernel for the Gaussian process
kernel = Matern(nu=2.5)

# Initialize the Gaussian process regressor
gpr = GaussianProcessRegressor( kernel =kernel , n_restarts_optimizer=10)

# Define the objective function
def f(x):
    x1, x2, x3 = x
    return -x1**2 - x2**2 - x3**2

# Define the variable bounds
bounds = [(0, 1), (0, 1), (0, 1)]

# Perform the optimization
res = gp_minimize(f, bounds, n_calls=20, random_state=0, verbose=True, n_random_starts=10, acq_func="EI", acq_optimizer="sampling", base_estimator=gpr)

# Print the optimal variables and function value
print("x1 = {:.4f}, x2 = {:.4f}, x3 = {:.4f}, f(x) = {:.4f}".format(res.x[0], res.x[1], res.x[2], res.fun))

# x1 = 1.0000, x2 = 1.0000, x3 = 1.0000, f(x) = -3.0000

my advice remains the same for the cases when you have such troubles:

explore docs of package/library for its methods & params of methods

I still can just assume, that there could be some bug in the lib itself, because in the lib - warnings are marked in some places to the state "ignore" -- therefore it is not always seen where the bug really arised from...

P.S. BTW, skopt (aka scikit-optimize) described as

a member of the sklearn club which is to say it works well, reliably, predictably, universally has amazing tooling, but is not that fast and few modern fancy fripperies.