How to choose between two overlapping colliders

Question

I am making a rubiks cube generator with unity3D. My generator generates not only the cube, but also "Side objects" for each side of the cube as seen here:

These side objects are just Empty objects with colliders. These colliders could be used to turn the sides of the cube by setting everything inside them except other sides as their children when user clicks on the collider and then just rotating the empty, like this:

Collider[] collides;

void OnMouseDown()
{

    collides = Physics.OverlapBox(transform.position, transform.localScale / 2, transform.rotation);    

    foreach (Collider col in collides)
        {
            if (!col.CompareTag("Side"))
            {
                col.gameObject.transform.parent = this.transform;

                Debug.Log(col.gameObject.transform.name);
            }
        }
}

Here is where my problem comes: I can not figure out the logic to what side to parent.

Let's imagine an event like this:

The user clicks on the area marked with red. Which side reacts? I have tried doing something like calculating the path users cursor has went since first clicked to the current position and comparing it to the orientation of the collider, but I haven't got it working.

Also when I make the cube pieces a child of the collider they become small.

How should I implement this?