GridSearchCV with XGBRegressor

2.1k Views Asked by At

I am using R^2 (from sklearn.metrics) as my scoring function, but when the grid search finishes it throws a best score of -282.3. Before trying to tune the parameters for this model I ran XGBRegressor on my training data with a set of (what I thought to be) reasonable parameters and got an R^2 score of 0.62, and when running grid search i made sure to include those initial parameters in the ranges I passed in. I am confused as to why this is happening and would appreciate some clarification and a work around if possible. Ive read in a few different places that the scoring API always minimizes values (so scores are negated), but but I am not sure how this would affect me because R^2 can be negative if the model does not fit the data well.

I include the code below, though I do not think it is necessary to answer the question.

params = { 'max_depth': [2, 4, 6],
           'n_estimators': [100, 500, 1000],
           'colsample_bytree': [0.2, 0.6, 0.8],
           'min_child_weight': [3, 5, 7],
           'gamma': [0.3, 0.5, 0.7],
           'subsample': [0.4, 0.6, 0.8]}

model = XGBRegressor()
grid = GridSearchCV(estimator=model, 
                   param_grid=params,
                   scoring='r2', 
                   verbose=1)
start = time.time()
grid.fit(X, y)
end = time.time()
print(end-start)
print("Best parameters:", grid.best_params_)
print("Best Score (R2): ", (grid.best_score_))

For reference, this is the model I used prior trying a GridSearch which gave me an R^2 score of 0.62:

regressor = XGBRegressor(colsample_bytree=0.6, gamma=0.3, max_depth=4, min_child_weight=5, n_estimators=100,  subsample=0.8, objective='reg:squarederror')

Thanks in advance!

0

There are 0 best solutions below