calculating accuracy for Monte carlo Dropout on pytorch

Question

I have found an implementation of the Monte carlo Dropout on pytorch the main idea of implementing this method is to set the dropout layers of the model to train mode. This allows for different dropout masks to be used during the different various forward passes. The implementation illustrate how multiple predictions from the various forward passes are stacked together and used for computing different uncertainty metrics.

import sys

import numpy as np

import torch
import torch.nn as nn


def enable_dropout(model):
    """ Function to enable the dropout layers during test-time """
    for m in model.modules():
        if m.__class__.__name__.startswith('Dropout'):
            m.train()

def get_monte_carlo_predictions(data_loader,
                                forward_passes,
                                model,
                                n_classes,
                                n_samples):
    """ Function to get the monte-carlo samples and uncertainty estimates
    through multiple forward passes

    Parameters
    ----------
    data_loader : object
        data loader object from the data loader module
    forward_passes : int
        number of monte-carlo samples/forward passes
    model : object
        keras model
    n_classes : int
        number of classes in the dataset
    n_samples : int
        number of samples in the test set
    """

    dropout_predictions = np.empty((0, n_samples, n_classes))
    softmax = nn.Softmax(dim=1)
    for i in range(forward_passes):
        predictions = np.empty((0, n_classes))
        model.eval()
        enable_dropout(model)
        for i, (image, label) in enumerate(data_loader):

            image = image.to(torch.device('cuda'))
            with torch.no_grad():
                output = model(image)
                output = softmax(output) # shape (n_samples, n_classes)
            predictions = np.vstack((predictions, output.cpu().numpy()))

        dropout_predictions = np.vstack((dropout_predictions,
                                         predictions[np.newaxis, :, :]))
        # dropout predictions - shape (forward_passes, n_samples, n_classes)
    
    # Calculating mean across multiple MCD forward passes 
    mean = np.mean(dropout_predictions, axis=0) # shape (n_samples, n_classes)

    # Calculating variance across multiple MCD forward passes 
    variance = np.var(dropout_predictions, axis=0) # shape (n_samples, n_classes)

    epsilon = sys.float_info.min
    # Calculating entropy across multiple MCD forward passes 
    entropy = -np.sum(mean*np.log(mean + epsilon), axis=-1) # shape (n_samples,)

    # Calculating mutual information across multiple MCD forward passes 
    mutual_info = entropy - np.mean(np.sum(-dropout_predictions*np.log(dropout_predictions + epsilon),
                                            axis=-1), axis=0) # shape (n_samples,)

What I'm trying to do is to calculate accuracy across different forward passes, can anyone please help me on how to get the accuracy and make any changes on the dimensions used in this implementation

I am using the CIFAR10 dataset and would like to use the dropout on test time The code for the data_loader

 testset = torchvision.datasets.CIFAR10(root='./data', train=False,download=True, transform=test_transform)

 #loading the test set
data_loader = torch.utils.data.DataLoader(testset, batch_size=n_samples, shuffle=False, num_workers=4) ```

Answer 1

Accuracy is the percentage of correctly classified samples. You can create a boolean array that indicates whether a certain prediction is equal to its corresponding reference value, and you can get the mean of these values to calculate accuracy. I have provided a code example of this below.

import numpy as np

# 2 forward passes, 4 samples, 3 classes
# shape is (2, 4, 3)
dropout_predictions = np.asarray([
    [[0.2, 0.1, 0.7], [0.1, 0.5, 0.4], [0.9, 0.05, 0.05], [0.25, 0.74, 0.01]],
    [[0.1, 0.5, 0.4], [0.2, 0.6, 0.2], [0.8, 0.10, 0.10], [0.25, 0.01, 0.74]]
])

# Get the predicted value for each sample in each forward pass.
# Shape of output is (2, 4).
classes = dropout_predictions.argmax(-1)
# array([[2, 1, 0, 1],
#        [1, 1, 0, 2]])

# Test equality among the reference values and predicted classes.
# Shape is unchanged.
y_true = np.asarray([2, 1, 0, 1])
elementwise_equal = np.equal(y_true, classes)
# array([[ True,  True,  True,  True],
#        [False,  True,  True, False]])

# Calculate the accuracy for each forward pass.
# Shape is (2,).
elementwise_equal.mean(axis=1)
# array([1. , 0.5])

In the example above, you can see that the accuracy for the first forward pass was 100%, and the accuracy for the second forward pass was 50%.

Answer 2

@jakub's answer is correct. However, I wanted to propose an alternate approach that may be better especially for more nascent researchers.

Scikit-learn comes with many built in performance measurement functions, including accuracy. To get those approaches to work with PyTorch, you only need to convert your torch tensor to numpy arrays:

  x = torch.Tensor(...) # Fill-in as needed
  x_np = x.numpy() # Convert to numpy

Then, you simply import scikit-learn:

   from sklearn.metrics import accuracy_score
   y_pred = [0, 2, 1, 3]
   y_true = [0, 1, 2, 3]
   accuracy_score(y_true, y_pred)

This simply returns 0.5. Easy peasy and less likely to have a bug.