I'm working on a set of 50 severely imbalanced (1:10,000-100,000) classification problems. Using equal class weights (and default hyperparameters) the LightGBM model achieves minimum performance (0.5 AUC) over all problems. Adjusting the positive class weight (scale_pos_weight) to the recommended value of count(negative class)/count(positive class) = 10,000-100,000, we achieve a moderate AUC ranging between 0.65 and 0.85. However, when we took the reciprocal of the recommended class weight value, i.e. 1/10,000 - 1/100,000, our AUC improved by about 5-10% consistently across all 50 problems. Note all AUCs are from the test set.
How would weighing the positive class (which is already a rare class) drastically less improve the model at all, let alone better than balancing the classes?
Is this an issue with the AUC metric? I've read that it is overly optimistic for severely imbalanced problems. Can we still not conclude anything until we have done cross validation? Would large positive weights increase likelihood of over-fitting compared to small weights, resulting in lower AUCs?
I'm working on performing some more tests and researching online to try understand this better (with little luck), although if you have any theories on what is going on, I'd love to hear them so I can direct my investigations better.