How can I estimate the pose of an object with tags attached, and always display the entire object, with known rotation and translation vectors?

Question

I have been able to detect multiple apriltags on a board of apriltags like this:

This was done using the AprilTag library: https://github.com/swatbotics/apriltag. I used OpenCV solvePnP() and projectPoints() to get the rotation and translation vectors, and project the 3D points onto the 2D points to display the pose of these apriltags.

See code below:

# I am getting the tag sizes from a json file for all apriltags
tag_sizes, tvecs, rvecs = get_extrinsics(json_file)

# Convert Images to grayscale
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# AprilTag detector options
options = apriltag.DetectorOptions(families='tag36h11',
                                border=1,
                                nthreads=4,
                                quad_decimate=1.0,
                                quad_blur=0.0,
                                refine_edges=True,
                                refine_decode=False,
                                refine_pose=True,
                                debug=False,
                                quad_contours=True)

detector = apriltag.Detector(options)

# Detect the apriltags in the image
detection_results, dimg = detector.detect(gray, return_image=True)

# Amount of april tags detected
num_detections = len(detection_results)
print('Detected {} tags.\n'.format(num_detections))

imgPointsArr = []
objPointsArr = []
opointsArr = []

if num_detections > 0:
    for i, detection in enumerate(detection_results):

        print('Detection {} of {}:'.format(i + 1, num_detections))
        print()
        print(detection.tostring(indent=2))

        if mtx is not None:
            imagePoints = detection.corners.reshape(1,4,2) 

            # Get the tag size from .json file
            tag_size = tag_sizes[detection.tag_id] 

            ob_pt1 = [-tag_size/2, -tag_size/2, 0.0]
            ob_pt2 = [ tag_size/2, -tag_size/2, 0.0]
            ob_pt3 = [ tag_size/2,  tag_size/2, 0.0]
            ob_pt4 = [-tag_size/2,  tag_size/2, 0.0]
            ob_pts = ob_pt1 + ob_pt2 + ob_pt3 + ob_pt4
            object_pts = np.array(ob_pts).reshape(4,3)

            opoints = np.array([
                -1, -1, 0,
                1, -1, 0,
                1,  1, 0,
                -1,  1, 0,
                -1, -1, -2*1,
                1, -1, -2*1,
                1,  1, -2*1,
                -1,  1, -2*1,
            ]).reshape(-1, 1, 3) * 0.5*tag_size
                
            imgPointsArr.append(imagePoints)
            objPointsArr.append(object_pts)
            opointsArr.append(opoints)

            # mtx - the camera calibration's intrinsics
            good, prvecs, ptvecs = cv2.solvePnP(object_pts, imagePoints, mtx, dcoeffs, flags=cv2.SOLVEPNP_ITERATIVE)
            imgpts, jac = cv2.projectPoints(opoints, prvecs, ptvecs, mtx, dcoeffs)

            # Draws the edges of the pose onto the image
            draw_boxes(img, imgpts, edges)

This works perfectly for the apriltag board. However, now I am trying to detect the pose of an object (dodecahedron:Dodecapen) with apriltags attached. These are already calibrated and comes with the json file that lists the size of each tag, and the rotation and translation vectors that corresponds to the correct tag id.

Using the rotation and translation vectors from that json file, I need to overlay all apriltags (even the ones that are not detected within the frame) onto the OpenCV window. To do this, I would have to use the rotation and translation vectors within the object points instead of just using the tag size to get the position of the object points using the vectors from the json, and use those as the 3D points within solvePnP(). I would apply rodrigues to the rotation vectors and then apply that to the object points, but I am unsure of how to apply them to the object points.

To always display all the apriltags onto the screen, I am thinking, that once I get the rotation and translation from the solvePnP(), I can apply that rotation and translation to the predefined rotation and translation vectors that the object came with, and then project those points using projectPoints() and draw it, to show the overlay?

Thus, in conclusion,upon detection of 1 or more apriltags in the frame, I am trying to estimate the pose of entire object (dodecahedron), and for the apriltags that are not detected (such as the ones behind the object), I am to display them onto the screen using the existing starting pose (rotation and translation vectors from the .json file).

Thus, based on this, my questions are:

1. How would I go about doing this, what would be the best method?

2. How can I use the rotation and translation vectors that are provided for the object, within the object points to get that pose in the 3D world and project them into the 2D image in such a way that all apriltags are shown on the openCV window, is the method I am thinking of the best way?

3. If I were to supply the array of object and image points outside of the for loop like this:

   objPointsArr = np.array(objPointsArr).reshape(-1, 3) 
   imgPointsArr = np.array(imgPointsArr).reshape(-1, 2) 
   opointsArr = np.array(opointsArr).reshape(-1, 3) 
   good, prvecs, ptvecs = cv2.solvePnP(objPointsArr, imgPointsArr, mtx, dcoeffs, flags=cv2.SOLVEPNP_ITERATIVE)
   imgpts, jac = cv2.projectPoints(opointsArr, prvecs, ptvecs, mtx, dcoeffs)
   draw_boxes(img, imgpts, edges)
   

These arrays were appended within the loop in the first code sample. Would solvePnP work as it should when inside the for loop?

Any help would be greatly appreciated!