ASL hand sign detection opencv

Question

I am a bit new to opencv and could use some help. I want to detect ASL hand signs.

For detecting hands, I can use either detection by skin color or a haar classifier. I already detect hands, but the problem is detecting the hand shape.

I can get the curent hand shape using the algorithm described here, so the problem is how do I compare this shape to my database of shapes?

I tried comparing them using the algorithm described here, which detects similar features images have. The problem is that this will match it with all the hands, since...well it detects them as hands. For instance, check this image, it should point only to V, but it detects features in W and R, too.

I want my final result to be like here, so how can I compare image shapes? Is my approach wrong?

I was thinking that detecting by convexity hull won't work, because most of the signs are closed fists. Check O, for instance, it has no open fingers, so I thought that trying to compare contours would be the best. How to compare them, though? FLANN doesn't seem to work. Or I'm doing it wrong.

Would a Haar cascade classifier work? Or would it detect two hands in different positions as hands as well?

Or is there another way to match shapes? That could solve my problem, but I couldn't find any example that does for custom shapes, only for ones like rectangles, circles and triangles.

Update

Ok, I've been playing a bit with matchShapes as berak told me. Here's my code below(it's a bit messy as I'm testing currently).

    #include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include <iostream>
#include <stdio.h>
#include <stdlib.h>

using namespace cv;
using namespace std;

Mat src; Mat src_gray;
int thresh = 10;
int max_thresh = 300;

/// Function header
void thresh_callback(int, void* );

/** @function main */
int main( int argc, char** argv )
{
    /// Load source image and convert it to gray
    src = imread( argv[1], 1 );

    /// Convert image to gray and blur it
    cvtColor( src, src_gray, CV_BGR2GRAY );
    blur( src_gray, src_gray, Size(3,3) );

    /// Create Window
    char* source_window = "Source";
    namedWindow( source_window, CV_WINDOW_AUTOSIZE );
    imshow( source_window, src );

    createTrackbar( " Canny thresh:", "Source", &thresh, max_thresh, thresh_callback );
    thresh_callback( 0, 0 );

    waitKey(0);
    return(0);
}

/** @function thresh_callback */
void thresh_callback(int, void* )
{
    Mat canny_output; 
    vector<vector<Point> > contours;
    vector<Vec4i> hierarchy;
    double largest_area=0;
    int largest_contour_index=0;
    Rect bounding_rect;
    /// Detect edges using canny
    Canny( src_gray, canny_output, thresh, thresh*2, 3 );
    /// Find contours
    findContours( canny_output, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, Point(0, 0) );
    cout<<contours.size()<<endl;
    /// Draw contours
    Mat drawing = Mat::zeros( canny_output.size(), CV_8UC3 );
     vector<vector<Point> >hull( contours.size() );
    for( int i = 0; i< contours.size(); i++ )
    {      Scalar color = Scalar( 255,255,255 );
    convexHull( Mat(contours[i]), hull[i], false );

    // imshow("conturul"+to_string(i), drawing );

    double a=contourArea( hull[i],false);  //  Find the area of contour
    if(a>largest_area){
        largest_area=a;
        largest_contour_index=i;                //Store the index of largest contour
        bounding_rect=boundingRect(hull[i]);}
    }
    cout<<"zaindex "<<largest_contour_index<<endl;

    Scalar color = Scalar( 255,255,255 );
    drawContours( drawing, hull, largest_contour_index, color, 2, 8, hierarchy, 0, Point() );
    namedWindow( "maxim", CV_WINDOW_AUTOSIZE );
    imshow( "maxim", drawing );

    Mat rects=imread( "scene.png", 1 );
    rectangle(rects, bounding_rect,  Scalar(0,255,0),1, 8,0);
    imshow( "maxim2", rects );

    /// Show in a window

}

The problem with it is the definition of a contour. These hand 'contours' are actually made of multiple contours themselves and that image that I showed earlier is actually made of these multiple contours but overlapped with eachother. matchShapes accepts arrays of Points as parameters, but the contours are arrays of arrays of Points.

So my question is, how can I add my contours vector with itself so I can pass it to matchShapes? In other words, how can I make a single contour from multiple overlapped contours?