How to use FaceNet and DBSCAN on multiple embeddings identities?

Question

I have the following setting:

1. A surveillance system take photos of people's faces (there are a varying number of photos for each person).
2. I run FaceNet for each photo and get a list of embedding vectors for each person (each person is represented by a list of embeddings, not by a single one).

The problem:

I want to cluster observed people using DBSCAN, but I need to guarantee that face embeddings from the same people go to the same cluster (remember we can have multiple photos of the same people, and we already know they must belong to the same cluster).

One solution could be to get a "mean" or average embedding for each person, but I believe this data loss is going to produce bad results.

Another solution could be to concatenate N embeddings (with N constant) in a single vector and pass that 512xN vector to DBSCAN, but the problem with this is that the order in which the embeddings are appended to this vector is going to produce different results.

Anyone has faced this same problem?

Answer 1

deepface wraps facenet face recognition model. The regular face recognition process is shown below.

#!pip install deepface
from deepface import DeepFace

my_set = [
   ["img1.jpg", "img2.jpg"],
   ["img1.jpg", "img3.jpg"],
]
obj = DeepFace.verify(my_set, model_name = 'Facenet')

for i in obj:
   print(i["distance"])

If you need the embeddings generated by facenet, you can adopt deepface as well.

from deepface.commons import functions
from deepface.basemodels import Facenet

model = Facenet.loadModel() 

#this detects and aligns faces. Facenet expects 160x160x3 shaped inputs.
img1 = functions.preprocess_face("img1.jpg", target_size = (160, 160))
img2 = functions.preprocess_face("img2.jpg", target_size = (160, 160))

#this finds embeddings for images
img1_embedding = model.predict(img1)
img2_embedding = model.predict(img2)

Embeddings will be 128 dimensional vectors for Facenet. You can run any clustering algorithm to embeddings. I have applied k-means for this kind of a study. I haven't any experience about dbscan but you can apply it if you have the embeddings.

Besides, you can adopt different face recognition models within deepface such as vgg-face, openface, facebook deepface, deepid and dlib.