My graduation project team and I are working on marker-based AR application where in one of the tasks we want to draw some 3d models on markers, we are using opencv to detect markers, and rajawali for drawing the 3d models.
The problem is the tvec/rvec we get from Aruco.estimatePoseSingleMarkers(...) doesn't map correctly to location and rotation on the markers, although we can draw the axis accurately.
image of the 3d model on the marker
So we wanted to ask:
- Is there a processing needed on the tvec/rvec needed before using them to get the position and rotation?
- Is there some alternatives for marker-detection that are more convenient to use than opencv with rajawali?
- What could be the cause for them to be inaccurate?
Code:
- marker-detection
public void markerDetection(Mat frame)
{
ids = new Mat();
corners = new ArrayList<>();
Aruco.detectMarkers(frame, markerDictionary, corners, ids);
List<MarkerData> newDetectedMarkers = new ArrayList<>();
if(ids.size().height > 0)
{
rvecs = new Mat();
tvecs = new Mat();
Aruco.estimatePoseSingleMarkers(corners, markerLength, cameraMatrix, distortionCoef, rvecs, tvecs);
for(int i=0; i<ids.size().height; i++)
{
double[] rvecArray = rvecs.get(i, 0), tvecArray = tvecs.get(i, 0);
Mat rvec = new Mat(3, 1, CvType.CV_64FC1), tvec = new Mat(3, 1, CvType.CV_64FC1);
for (int j = 0; j < 3; ++j) {
rvec.put(j, 0, rvecArray[j]);
tvec.put(j, 0, tvecArray[j]);
}
multiply(rvec, new Scalar(180.0 / Math.PI), rvec); // transform them to degree
MarkerData newMarker = new MarkerData(rvec, tvec, corners.get(i), (int)ids.get(i, 0)[0]);
newDetectedMarkers.add(newMarker);
}
}
updateDetectedMarkers(newDetectedMarkers); // update the detected markers
}
- Rendering
@Override
protected void onRender(long elapsedRealtime, double deltaTime) {
super.onRender(elapsedRealtime, deltaTime);
getCurrentScene().clearChildren();
List<MarkerData> markerData = markerDetector.getDetectedMarkers();
for (MarkerData marker : markerData) {
try {
int id = R.raw.monkey;
LoaderOBJ parser = new LoaderOBJ(mContext.getResources(), mTextureManager, id);
parser.parse();
Object3D object = parser.getParsedObject();
object.setMaterial(defaultMaterial);
object.setScale(0.3);
Mat rvec = marker.getRvec(); // 3x1 Mat
Mat tvec = marker.getTvec(); // 3x1 Mat
object.setRotation(new Vector3(rvec.get(0, 0)[0], rvec.get(1, 0)[0], rvec.get(2, 0)[0]));
object.setPosition(new Vector3(tvec.get(0, 0)[0], tvec.get(1, 0)[0], tvec.get(2, 0)[0]));
getCurrentScene().addChild(object);
} catch (ParsingException e) {
e.printStackTrace();
}
}
}