Detect faces without eyebrows and jaw using opencv

Question

I am trying to detect faces (specifically, faces with opened eyes) using OpenCV haar cascade classifier. However, I had a problem detecting the faces that do not have eyebrows and/or jaw, as shown in the following image. I had tried many haar cascade for face detection such as haarcascade_frontalface_default.xml, haarcascade_frontalface_alt_tree.xml, etc. But all of these did not work.

Here is my code:

import cv2
import os
import glob

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
eye_cascade = cv2.CascadeClassifier('haarcascade_eye_tree_eyeglasses.xml')

count = 0
path = "./test/*.png"
for index, filename in enumerate(glob.glob(path)):
    img = cv2.imread(filename)
    basename = os.path.splitext(os.path.basename(filename))[0]

    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(gray, 1.3, 5)

    for (x,y,w,h) in faces:
        # cv2.rectangle(img,(x,y),(x+w, y+h),(255,0,0), 2)
        roi_gray = gray[y:y+h, x:x+w]
        roi_color = img[y:y+h, x:x+w]
        eyes = eye_cascade.detectMultiScale(roi_gray)

        if len(eyes) >= 2:
            count = count + 1
            output_dir = './test/output'
            cv2.imwrite(f'{output_dir}/{basename}.png', img)

cv2.waitKey(0)
cv2.destroyAllWindows()

Thank you in advance!

Answer 1

If you have tensorflow installed you can use a neural net to detect faces which give much better accuracy than the simple haar classifier.

Here's an example using the MTCNN detector which uses tensorflow as the backend.

from mtcnn.mtcnn import MTCNN
from PIL import Image
import numpy as np

img = Image.open('6qNFu.png') # load the image
img = np.asarray(img, dtype='uint8') # convert to numpy array
img = img[:,:,0:3] # drop the alpha channel

detector = MTCNN() # initialize MTCNN detector
print(detector.detect_faces(img)) # use MTCNN detector to return bounding box and face metrics

Using the bounding box you can extract the face from the image. Note: if the face is truncated like in the instances above, it might return a negative coordinate which is an extrapolation of where it thinks the face might be.

Here is a the documentation on MTCNN library: https://pypi.org/project/mtcnn/ It also tells you how to install it.

Answer 2

Use facial landmarks with dlib, this method may work for you, see these two links:

• facial-landmarks-dlib-opencv-python
• opencv-face-recognition

Also, see this link:

• dlib-and-deep-learning