Is there a way to make Colour detector more accurate?

Question

I have created a colour detector for a Rubik's Cube using OpenCV. My plan for this it will then output the solution to the users. So far it detects some colours using OpenCV when the cube is held in the webcam, however some of the colours detect wrongly. There could be a few factors causing this such as lighting or even just the code however so far I've been unable to solve this. As you can see in the image below, the cubies are appearing in the wrong colours and i'm at a loss as to why.

Cube

Any help would be appreciated. Here is the code that I use to detect the colours.

import numpy as np
import cv2

    bounds = {
        "red" : (np.array([160, 75, 75]), np.array([180, 255, 255])),
        "blue" : (np.array([110, 75, 75]), np.array([130, 255, 255])),
        "green" : (np.array([35, 0, 0]), np.array([85, 255, 255])),
        "yellow" : (np.array([20, 75, 75]), np.array([40, 255, 255])),
        "white" : (np.array([0, 0, 20]), np.array([180, 30, 255])),
        "orange" : (np.array([10,100,100]), np.array([20, 255, 255]))
    }
    
    def density(img, color):
        lower = bounds[color][0]
        upper = bounds[color][1]
        mask = cv2.inRange(img, lower, upper)
        return np.sum(mask)/255
    
    def cubestr(data):
        ret = ""
        for i in "URFDLB":
            ret += "".join(data[i])
        for i in "URFDLB":
            ret = ret.replace(data[i][4], i)
        return ret

If my question is missing any details please let me know. Thanks.