Poisson Model (Binary Response) Evaluation Metrics

Question

I am unsure about the correct evaluation metrics to use in my model, and would appreciate your advice.

Model Info:
I fitted a GLM Poisson model in Python on a dataset, where each row of data has a different exposure between 0 to 1 and the response variable is binary. So for example the exposure and response of the first 4 rows of data could look like below in a data frame:

Exposure: 0.345, 0.123, 0.8, 0.00387....

Response: 0, 1, 0, 0.....

My model has 6 factors, all showing statistically significance. The model prediction is between 0.01 to 6.

I used Mean_Square_Error and R2_Score originally as the evaluation metrics (see code below), but I got odd results. I got Mean Square Error: 0.01 and R2_Score: 0.03 which is a very poor score. I got a negative R2_Score if I didn't multiply the Model_Prediction by Exposure.

from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error

print("Mean Square Error")
print(round(mean_squared_error(Response,  Model_Prediction * Exposure),2))

print("R2_Score")
print(round(r2_score(Response,  Model_Prediction * Exposure),2))

My questions are:

1. Have I used the mean_squared_error and r2_score incorrectly?
2. Given the response is 0/1, can I use ROC and AUC in a Poisson regression problem?

Answer 1

Not clear how you got the Model_prediction in your code, but in general, Poisson regression is for count variables and hence the prediction can be above 1. If the rate of 1s in your data is not very small (>10%), I would expect a fair number of predictions being above 1. It is fine to use it for binary outcomes though, however, it is more often done for estimating group risk ratios, which are the coefficients of the regression. I suggest trying Logistic Regression, which has logit transformation of a regression and makes outcome to be between 0 and 1 (interpreted as probability of positive binary outcome). Good discussion here https://stats.stackexchange.com/questions/18595/poisson-regression-to-estimate-relative-risk-for-binary-outcomes?noredirect=1&lq=1