searching for best hyper parameters of XGBRegressor using HalvingGridSearchCV

Question

I am searching for best hyper parameters of XGBRegressor using HalvingGridSearchCV. Here is the code:

base_estimator = XGBRegressor(seed=1234,use_label_encoder=False,base_score=0.5,max_delta_step=0,
                              scale_pos_weight=1,nthread=12)

params = {'learning_rate': [0.2],
        'max_depth': [500],
         'min_child_weight': [50],
         'gamma': [1.5],
         'reg_alpha': [0.7],
        'reg_lambda':[50],
        'subsample':[1],
        'colsample_bytree': [0.5],
         'n_estimators':[1000]}

sh = HalvingGridSearchCV(base_estimator, param_grid=params, cv=5,
                         factor=2, max_resources=7926,resource='n_samples',
                         aggressive_elimination=True).fit(x_train, y_train,early_stopping_rounds=10,eval_metric='rmse', eval_set=[(x_test, y_test)], verbose=True)
     
print("Best: %f using %s" % (sh.best_score_, sh.best_params_))

Best: 0.058512 using {'colsample_bytree': 0.5, 'gamma': 1.5, 'learning_rate': 0.2, 'max_depth': 500, 'min_child_weight': 50, 'n_estimators': 1000, 'reg_alpha': 0.7, 'reg_lambda': 50, 'subsample': 1}

I've tried many options, but the result is still small, Can you help me and tell me why? Is anything bad in code or ? i do not know.

Thank you so much

Answer 1

Can you include more parameters for search? I can see that you use the same parameters over each step of iteration. You should add some more values to increase the probability to find the ones which will be "the best". For instance, you can use this code:

params={"learning_rate": [0.1,0.2, 0.3, 0.4, 0.5, 0.8, 1,10, 20 ,0.01, 0.001, 0.0001 ],
                            "max_depth": [ 3, 4, 5, 6, 8],
                            "min_child_weight": [ 1, 3, 5, 7],
                            "gamma": [0.1,0.2, 0.3, 0.4, 0.5, 0.8, 1,10, 20 ,0.01, 0.001, 0.0001 ],
                            "colsample_bytree":[ 0.3, 0.4],
                           "early_stopping_rounds": [3]},

Answer 2

It really depends on the data. If you have tried many hyperparameters combinations and still you have bad performance, you should try to train a model with better data.