Encog binary classification score for ROC

Question

I am working on a binary classifier using Encog (via Java). I have it set up using an SVM or neural network, and I am want to evaluate the quality of the different models using (in part) the area under the ROC curve.

More specifically, I would ideally like to convert the output of the model into a some kind of prediction confidence score that can be used for rank ordering in the ROC, but I have yet to find anything in the documentation.

In the code, I get the model results with something like:

MLData result = ((MLRegression) method).compute( pair.getInput() );
String classification = normHelper.denormalizeOutputVectorToString( result )[0]; 

How do I also get a numerical confidence of the classification?

Answer 1

Encog has no direct support for ROC curves. A ROC curve is more of a visualization than an actual model type, which is primarily the focus of Encog.

Generating a ROC curve for SVM's and Neural Networks is somewhat different. For a neural network, you must establish thresholds for the classification neurons. There is a good paper about that here: http://www.lcc.uma.es/~jja/recidiva/048.pdf

I may eventually add direct support for ROC curves into Encog in the future. They are becoming a very common visualization.

Answer 2

I have found a way to coax prediction probabilities out of SVM inside the encog framework. This method relies upon the equivalent of the -b option for libSVM (see http://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html)

To do this, override the SVM class from encog. The constructor will enable probability estimates via the smv_parameter object (see below). Then, when doing the calculation, call the method svm_predict_probability as shown below.

Caveat: below is only a code fragment and in order to be useful you will probably need to write other constructors and to pass the resulting probabilities out of the methods below. This fragment is based upon encog version 3.3.0.

public class MySVMProbability extends SVM {

public MySVMProbability(SVM method) {
    super(method.getInputCount(), method.getSVMType(), method.getKernelType());
    // Enable probability estimates 
    getParams().probability = 1;
}


@Override
public int classify(final MLData input) {
    svm_model model = getModel();
    if (model == null) {
        throw new EncogError(
                "Can't use the SVM yet, it has not been trained, " 
                + "and no model exists.");
    }

    final svm_node[] formattedInput = makeSparse(input);
    final double probs[] = new double[svm.svm_get_nr_class(getModel())];
    final double d = svm.svm_predict_probability(model, formattedInput, probs);

    /* probabilities for each class are in probs[] */

    return (int) d;
}


@Override
public MLData compute(MLData input) {
    svm_model model = getModel();
    if (model == null) {
        throw new EncogError(
                "Can't use the SVM yet, it has not been trained, "
                + "and no model exists.");
    }

    final MLData result = new BasicMLData(1);

    final svm_node[] formattedInput = makeSparse(input);

    final double probs[] = new double[svm.svm_get_nr_class(getModel())];
    final double d = svm.svm_predict_probability(model, formattedInput, probs);
    /* probabilities for each class are in probs[] */
    result.setData(0, d);

    return result;
}
}