Precision--recall curves in image retrieval domain

Question

I am working on loop-closure detection problem in two different seasons, e.g., summer, and fall. I need to make precision-recall curves. Suppose, I have taken 500 image from summer and 500 image from fall season. I have distance matrix. enter image description here

But I am totally confused, how to make precison recall curves. Like, for each image from one season, I will get 500 nearest images in ascending (distance) order. I know the definition of precision and recall, but i can't get close to the solution of this problem. Looking forward for any kind of help or comments or advice. thanks in advance.

Answer 1

In precision-recall plots each point is a pair of precision and recall values. In your case, I guess, you'd need to compute those values for each image and then average them.

Imagine you have 1000 images in total and only 100 images that belong to summer. If you take 500 closest images to some "summer" image, precision in the best case (when the first images always belong to the class) would be:

precision(summer) = 100 / (100 + 400) = (retrieved summer images) / (retrieved summer images + other retrieved images) = 0.2

And recall:

recall(summer) = 100 / (100 + 0) = (retrieved summer images) / (retrieved summer images + not retrieved summer images) = 1

As you can see, it has high recall because all the summer images were retrieved, but low precision, because there are only 100 images, and 400 other images don't belong to the class.

Now, if you take the first 100 images instead of 500, both recall and precision would equal 1.

If you take 50 first images, then precision would be still 1, but recall would drop to 0.5.

So, by varying the number of images you can get points for the precision-recall curve. For the above-described example these points would be (0.2, 1), (1, 1), (1, 0.5).

You could compute these values for each of the 1000 images using different thresholds.