gpflow: Non-Independent Multioutput Kernel

Question

I'm trying to implement my own MultioutputKernel (MOK) in gpflow, however I'm stuck at the Multiple Dispatch for the (Kernel, Inducing Variable) combinations.

According to the docs, the fallback method for the combination of a MultioutputKernel with InducingPoints should be calling fully_correlated_conditional (via gpf.conditionals.multioutput.inducing_point_conditional).

However, I can't get any MOK working with non-independent inducing variables, not even the pre-implemented one. Here's a minimum not-working example for SharedIndependent:

######################## toy data
d = 1
X = np.random.normal(0, 10, (100, d))
xx = np.linspace(-10, 10, 200).reshape(-1, 1)
f = lambda x: x ** 2
f_ = lambda x: 2 * x
Y = f(X)
Y_ = f_(X)

Y_combined = np.hstack((Y, Y_))
data = (X, Y_combined)

######################### gpflow stuff
kernel = gpf.kernels.RBF(lengthscales=[1.0] * d)
Z = X.copy()
# create multi-output inducing variables from Z
iv = gpf.inducing_variables.InducingPoints(Z)

MOK_K = gpf.kernels.SharedIndependent(kernel, output_dim=2)

m = gpf.models.SVGP(likelihood=gpf.likelihoods.Gaussian(), kernel=MOK_K, num_latent_gps=2,
                    inducing_variable=iv)

optimizer = gpf.optimizers.Scipy()
optimizer.minimize(
    m.training_loss_closure(data),
    variables=m.trainable_variables,
    method="l-bfgs-b",
    options={"disp": True, "maxiter": 1000},
)

This does not work, unless one switches out the inducing points with

iv = gpf.inducing_variables.SharedIndependentInducingVariables(
    gpf.inducing_variables.InducingPoints(Z)
)

But for my custom non-independent Kernel, I need the fully correlated conditionals. The error I get is

ValueError: base_conditional() arguments [Note that this check verifies the shape of an alternative representation of Kmn. See the docs for the actual expected shape.]

It seems in the inducing_point_conditional method it tries to "flatten" (to 2d) the kernel matrix to the classical multi-output representation before passing it to base_conditional. However, I don't understand where it goes wrong, as the shapes should be fine. They are just as defined in the docs.

What would I have to change to get it running with the fully correlated conditionals?

Answer 1

The issue were the dimensions of the variational distribution parameters, q_mu and q_sqrt.

One has to manually define them in advance, so that they have the shapes [N*P, 1] and [1, N*P, N*P] respectively, where P is the output dimension (2 in my example).

Note that this is slightly different to the docs, they say q_mu has to be [1, N*P].

My code from the question is running correctly with:

# [200, 1]    
q_mu = tf.zeros([200, 1], dtype=gpf.config.default_float())

# [1, 200, 200]
q_sqrt = tf.eye(200, dtype=gpf.config.default_float())[tf.newaxis, ...]

m = gpf.models.SVGP(likelihood=gpf.likelihoods.Gaussian(), kernel=MOK_K, num_latent_gps=2,
                    inducing_variable=iv, q_mu=q_mu, q_sqrt=q_sqrt)