Why the best loss doesn't updated?

Question

I'm attempting to run parameter optimization using HYPEROPT, but I'm not seeing any changes in the printing of the best loss value.

I attempted to change the accuracy sign, but it did not help. I tried testing models in my own randomized trials, and the results were much better. How could I optimize the parameters?

I followed this notebook.

Minimal code example:

import pandas as pd
from sklearn.metrics import roc_auc_score
from hyperopt import STATUS_OK, Trials, fmin, hp, tpe
import xgboost as xgb

def objective(space):
    clf = xgb.XGBClassifier(
        n_estimators=space['n_estimators'], max_depth=int(space['max_depth']), gamma=space['gamma'],
        reg_alpha=int(space['reg_alpha']), min_child_weight=int(space['min_child_weight']),
        colsample_bytree=int(space['colsample_bytree']))

    evaluation = [(train, train_labels), (test, test_labels)]

    clf.fit(train, train_labels,
            eval_set=evaluation, eval_metric="auc",
            early_stopping_rounds=10, verbose=True)

    pred = clf.predict(test)
    accuracy = roc_auc_score(test_labels, pred)
    print("ROC:", accuracy)
    return {'loss': -accuracy, 'status': STATUS_OK}

space = {'max_depth': hp.quniform("max_depth", 3, 300, 1),
         'gamma': hp.uniform('gamma', 1, 9),
         'reg_alpha': hp.quniform('reg_alpha', 5, 180, 1),
         'reg_lambda': hp.uniform('reg_lambda', 0, 1),
         'colsample_bytree': hp.uniform('colsample_bytree', 0.1, 1),
         'min_child_weight': hp.quniform('min_child_weight', 0, 10, 1),
         'n_estimators': 300,
         'seed': 0
         }

train, train_labels, train_Ids = pd.read_csv("train.csv")
test, test_labels, test_Ids = pd.read_csv("test.csv")

trials = Trials()

best_hyperparams = fmin(fn=objective,
                        space=space,
                        algo=tpe.suggest,
                        max_evals=400,
                        trials=trials)

print("The best hyperparameters are : ", "\n")
print(best_hyperparams)

The result, which is repeated at the start of each iteration, is for example:

2%|▏         | 9/400 [00:07<05:31,  1.18trial/s, best loss: -0.5]
...
5%|▍         | 19/400 [00:17<05:58,  1.06trial/s, best loss: -0.5]
...

Answer 1

I cannot recreate your exact issue without your dataset, but I tried it with the sklearn dataset load_breast_cancer. I quickly get scores better than 0.5, but there are many scores with that baseline score. I think it's because your reg_alpha range is too high, so that some of the models end up pruned to nothing. Hopefully after your optimization samples some smaller alphas, the tpe algorithm will start to focus on more useful values.

You might check:

import numpy as np
alphas = [(trial['misc']['vals']['reg_alpha'][0], trial['result']['loss']) for trial in trials.trials]
print(np.array([alpha for alpha, score in alphas if score == -0.5]).min())
print(np.array([alpha for alpha, score in alphas if score != -0.5]).max())

which for me gives 85.0 and 89.0; there's a little overlap, but broadly speaking, alphas greater than 85 kill the model.