TypeError: If no scoring is specified, the estimator passed should have a 'score' method

Question

I have been working with a PyTorch neural network for a while now. I decided I wanted to add a permutation feature importance scorer, and this started to cause some issues.

I get" TypeError: If no scoring is specified, the estimator passed should have a 'score' method. The estimator <class 'skorch.net.NeuralNet'>[uninitialized]( module=<class 'main.run..MultiLayerPredictor'>, ) does not. " - error message. Here's my code:

class MultiLayerPredictor(torch.nn.Module):
    def __init__(self, input_shape=9152, output_shape=1, hidden_dim=1024, **kwargs):
        super().__init__()
        self.fc1 = torch.nn.Linear(in_features=input_shape, out_features=hidden_dim)
        self.fc2 = torch.nn.Linear(in_features=hidden_dim, out_features=hidden_dim)
        self.fc3 = torch.nn.Linear(in_features=hidden_dim, out_features=output_shape)

    def forward(self, x):
        l1 = torch.relu(self.fc1(x))
        l2 = torch.relu(self.fc2(l1))
        return torch.sigmoid(self.fc3(l2)).reshape(-1)

print("Moving to wrapping the neural net")
net = NeuralNet(
    MultiLayerPredictor,
    criterion=nn.MSELoss,
    max_epochs=10,
    optimizer=optim.Adam,
    lr=0.1,
    iterator_train__shuffle=True
)

print("Moving to finding optimal hyperparameters")

lr = (10**np.random.uniform(-5,-2.5,1000)).tolist()
params = {
    'optimizer__lr': lr,
    'max_epochs':[300,400,500],
    'module__num_units': [14,20,28,36,42],
    'module__drop' : [0,.1,.2,.3,.4]
}

gs = RandomizedSearchCV(net,params,refit=True,cv=3,scoring='neg_mean_squared_error',n_iter=100)
gs.fit(X_train_scaled,y_train);

def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidates = np.flatnonzero(results['rank_test_score'] == i)
    for candidate in candidates:
        print("Model with rank: {0}".format(i))
        print("Mean validation score: {0:.3f} (std: {1:.3f})".format(
              results['mean_test_score'][candidate],
              results['std_test_score'][candidate]))
        print("Parameters: {0}".format(results['params'][candidate]))
        print("")

print(report(gs.cv_results_,10))

epochs = [i for i in range(len(gs.best_estimator_.history))]
train_loss = gs.best_estimator_.history[:,'train_loss']
valid_loss = gs.best_estimator_.history[:,'valid_loss']

plt.plot(epochs,train_loss,'g-');
plt.plot(epochs,valid_loss,'r-');
plt.title('Training Loss Curves');
plt.xlabel('Epochs');
plt.ylabel('Mean Squared Error');
plt.legend(['Train','Validation']);
plt.show()

r = permutation_importance(net, X_test, y_test, n_repeats=30,random_state=0)

for i in r.importances_mean.argsort()[::-1]:
    if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
        print(f"{metabolites.feature_names[i]:<8}"
              f"{r.importances_mean[i]:.3f}"
              f" +/- {r.importances_std[i]:.3f}")

y_pred_acc = gs.predict(X_test)
print('Accuracy : ' + str(accuracy_score(y_test,y_pred_acc)))

Stacktrace would point that the error stems from the line where I set the permutation importance. How can I fix this?

Full stacktrace:

*Traceback (most recent call last):
  File "//ad..fi/home/h//Desktop/neuralnet/neuralnet_wrapped.py", line 141, in <module>
    run()
  File "//ad..fi/home/h//Desktop/neuralnet/neuralnet_wrapped.py", line 119, in run
    r = permutation_importance(net, X_test, y_test,
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\inspection\_permutation_importance.py", line 132, in permutation_importance
    scorer = check_scoring(estimator, scoring=scoring)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\metrics\_scorer.py", line 425, in check_scoring
    raise TypeError(
TypeError: If no scoring is specified, the estimator passed should have a 'score' method. The estimator <class 'skorch.net.NeuralNet'>[uninitialized](
  module=<class '__main__.run.<locals>.MultiLayerPredictor'>,
) does not.*

Answer 1

From the docs:

NeuralNet still has no score method. If you need it, you have to implement it yourself.

This is the problem. The NeuralNet has no score method, as the error says. And the documentation says that "you have to implement it yourself". You can check that looking at the source-code too.

Answer 2

As Berriel said, this fails since your neural network instance does not implement a score() method. This is the default as it is unclear what score should be returned for an arbitrary learning task.

This happens in with sklearn grid searches as well and you circumvented this by passing scoring='neg_mean_squared_error'. You can do this here as well:

r = permutation_importance(net, X_test, y_test, 
        scoring='neg_mean_squared_error', n_repeats=30, random_state=0)

Alternatively, say because you need scoring in other places as well, you can implement the score method yourself:

class MyNet(NeuralNetwork):
    def score(self, X, y):
        y = self.predict(X)
        return sklearn.metrics.mean_squared_error(y, y_pred)