Connecting model to bounding boxes

Question

First of all, thank you for taking the time to read my question.

I'm trying to predict text in bounding boxes. Somehow it looks like there is no connection between the boxes and the model.

I used EAST text detection to create bounding boxes and trained a CNN model with Keras. Now i'm retrieving input through my webcam with opencv. readNet the EAST model and load_model my trained CNN.

I trained my model with black text words on a white background.

When I'm trying to connect the two it creates the right boxes in the video capture, however it doesn't provide the right prediction class.

This is what the last part of my code looks like:

   # loop over the bounding boxes
    for (startX, startY, endX, endY) in boxes:
        # scale the bounding box coordinates based on the respective
        # ratios
        startX = int(startX * rW)
        startY = int(startY * rH)
        endX = int(endX * rW)
        endY = int(endY * rH)
    
        orig_res =cv2.resize(orig,(200,200))
        orig_res = orig_res.reshape(1,200,200,3)
        
        prediction = model.predict(orig_res)
        max_prediction = np.argmax(prediction[0])
        my_prediction = myList[max_prediction]
            
        cv2.putText(orig,str(my_prediction), (startX, startY),
            cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 0, 255), 3)
        
        # draw the bounding box on the image
        cv2.rectangle(orig, (startX, startY), (endX, endY), (0, 255, 0), 2)
        
    # show the output image
    cv2.imshow("Detect_text", orig)

myList stands for the folder names of the different words. These are the names that the model should predict. As you can see in this screenshot, it doesn't predict "100%" ..

This is a selection of what the dataset looks like:

Can someone provide a solution for this? Many thanks!

Answer 1

It looks like a mismatch between the ordering of the labels you used during training and the list of labels you created at inference from the list of folder names. The reason is that os.listdir returns filenames in an arbitrary order, whereas keras image generator will sort the folder names before mapping them to class indices.

You can ensure to have the same ordering by creating list of labels as follow:

# suppose your training data generator is named train_gen
myList = [item[0] for item in sorted(list(train_gen.class_indices.items()), key=lambda x: x[1])]

This way you've your labels names used during training and during inference aligned