I am using R^2 (from sklearn.metrics) as my scoring function, but when the grid search finishes it throws a best score of -282.3. Before trying to tune the parameters for this model I ran XGBRegressor on my training data with a set of (what I thought to be) reasonable parameters and got an R^2 score of 0.62, and when running grid search i made sure to include those initial parameters in the ranges I passed in. I am confused as to why this is happening and would appreciate some clarification and a work around if possible. Ive read in a few different places that the scoring API always minimizes values (so scores are negated), but but I am not sure how this would affect me because R^2 can be negative if the model does not fit the data well.
I include the code below, though I do not think it is necessary to answer the question.
params = { 'max_depth': [2, 4, 6],
'n_estimators': [100, 500, 1000],
'colsample_bytree': [0.2, 0.6, 0.8],
'min_child_weight': [3, 5, 7],
'gamma': [0.3, 0.5, 0.7],
'subsample': [0.4, 0.6, 0.8]}
model = XGBRegressor()
grid = GridSearchCV(estimator=model,
param_grid=params,
scoring='r2',
verbose=1)
start = time.time()
grid.fit(X, y)
end = time.time()
print(end-start)
print("Best parameters:", grid.best_params_)
print("Best Score (R2): ", (grid.best_score_))
For reference, this is the model I used prior trying a GridSearch which gave me an R^2 score of 0.62:
regressor = XGBRegressor(colsample_bytree=0.6, gamma=0.3, max_depth=4, min_child_weight=5, n_estimators=100, subsample=0.8, objective='reg:squarederror')
Thanks in advance!