How do I get the MAE, RMSE, MSE and R^2 on a Pycaret model?

Question

I am trying to get the MAE, RMSE, MSE and R^2 on a model, but actually it only gives me some metrics that are used mostly on classification, not on regression .

These are the metrics that the model gives me:

I have already read the Pycaret documentation, but I only found the option of add_metric() but I don't if this function will work for that (also I didn't understood how add_metric() function works)

My setup function:

exp = setup(data = dataset, target = 'Lower Salary', categorical_features = cat_f,
               ignore_features= ['Job Title','Headquarters','Founded','Type of ownership','Competitors','company_txt','job_title_sim','seniority_by_title','Salary Estimate','Job Description','Industry','Hourly','Employer provided'],
               normalize = True,session_id = 123)

My create_model function:

logit = create_model('lr')

Answer 1

The actual problem is that you are using logistic regression. That is a classification model, not a regression model. That is why you are only seeing metrics for classification models.

I'm actually working on a time series project right now and need to calculate these metrics. This is what I do.

I haven't used pycaret but this would require you having the ability to get your test dataset and an output of your model's predictions.

You will need to import sort from the math library, and also install sklearn for r2_score(), mean_squared_error(), and mean_absolute_error().

print(f'Mean Absolute Error = {mean_absolute_error(actual,pred)}')
print(f'Mean Squared Error = {mean_squared_error(actual,pred)}')
print(f'Root Mean Squared Error = {math.sqrt(mean_squared_error(actual,pred))}')
print(f'r2 = {r2_score(actual,pred)}')

Looking at the pycaret docs, it looks like get_leaderboard() might work for your case. Here's a link

Here is a notebook that has code examples for you.