How to compare relative rotation between objects?

Question

I have two objects in 3D space (using OpenGL to render it all) of the same type of a class. These objects store xyz offsets and a rotation matrix representing your standard 4x4 rotational matrix.

A stripped down example:

class object {
  float rotMatrix[16];
  float xyzTrans[3];
  //a displaylist for drawing the object
}

I'm using GLUI for UI controls which makes storing the transformations in format pretty simple.

The problem: I need to define a "correct" orientation for one object with respect to the other. For example, if the first object is facing directly down the z-axis, and the second one is the same but also rotated roughly 45deg around the x-axis, this would be deemed "correct" and my functions do what they need to do. This can vary of course, maybe its the same z but rotated on the x and y, or maybe even rotated a little bit around each axis. The definition of "correct" rotations will be stored in the object for comparison.

Ideally I'm hoping to do something like:

bool checkRotation(object* first, object* second) { 
   //do some comparison stuff

   if (eachCheck < someTolerance)
      return true;
   return false;
}

Is this possible to do by comparing two rotational matrices? Do I need to convert to quaternions and use those?

This question is the closest I've found to what I'm asking, but it's just different enough to be confusing.

Answer 1

Not a complete answer, but too long for a comment:

If you have two honest rotation matrices, then they should be invertible, and of determinant 1. Call the matrices A and B. If you want to check that the images A(X) and B(X) of the same object X under the two rotations is "close" in the sense that you can go from A(X) to B(X) by rotation around a specified axis, this is equivalent to checking whether the matrix obtained by taking A times the inverse of B is "nearly" a rotation around that axis. So this is probably the kind of thing you want to look at.

I'm not too familiar with the OpenGL matrix math functions so can't provide any code, sorry.