Apriltags3, how can I improve pose ambiguity when a marker faces camera?

Question

I have an c++ application that uses Apriltags3 fiducial markers.

These seem fast and stable, except in the case where the marker is facing flat to the camera, or thereabouts.

in these cases, the pose flips significantly, making the pose data unusable. This flipping, as I understand it, is a known issue with square markers, as there are two possible solutions to estimate the pose.

How can i improve this flipping? I have tried:

1 - Using SolvePnp with the IPPE_SQUARE flag, i am able to return both solutions.

std::vector<cv::Vec3d> rvecsVec, tvecsVec;
std::vector<double> errs;
solvePnPGeneric(markerObjPoints, marker.corners2d, K, D, rvecsVec, tvecsVec, false, cv::SOLVEPNP_IPPE_SQUARE, cv::noArray(), cv::noArray(),errs);

then i choose the pose that is closest to the previous pose. this works well until the rotation of the pose passes zero (positive to negative values) and then the rotation axis flips.

2 - storing the previous few poses and running sliding window bundle adjustment on the 3d points, the 2d corners, and the poses. This seems to have no effect, and I still see the same flipping.

3 - Using the following:

cv::Mat R1;
cv::Rodrigues(rvecsVec[0], R1);
R1 = R1.t();
cv::Mat t1 = -R1 * tvecsVec[0];

cv::Mat R2;
cv::Rodrigues(rvecsVec[1], R2);
R2 = R2.t();
cv::Mat t2 = -R2 * tvecsVec[1];

To get the camera poses relative to the markers. I would have assumed that a camera pose with a positive X value (in front of the marker) would be the correct one, however, when i use that pose:

if (t1.at<double>(0, 0) < 0)
    {
        marker.rvec = rvecsVec[0];
        marker.tvec = tvecsVec[0];
        std::cout << "using pose 0" << std::endl;

    }
    if (t2.at<double>(0, 0) < 0)
    {
        marker.rvec = rvecsVec[1];
        marker.tvec = tvecsVec[1];
        std::cout << "using pose 1" << std::endl;       
    }

I see the same issue as with solution 1.

4 - using a stereo camera, find the markers in each frame, triangulate the points and then use an Eigen SVD function to find the rigid transformation between these points and a set of points created from markerlength at zero.

PointsType p1s, p2s;
p1s.push_back(Eigen::Vector3d(-markerLength / 2.f, markerLength / 2.f, 0));
p1s.push_back(Eigen::Vector3d(markerLength / 2.f, markerLength / 2.f, 0));
p1s.push_back(Eigen::Vector3d(markerLength / 2.f, -markerLength / 2.f, 0));
p1s.push_back(Eigen::Vector3d(-markerLength / 2.f, -markerLength / 2.f, 0));

This works great for translation, but rotation seems to flip even more!

IS there an answer for this? What else can I try?

Note: I have tested with Aruco markers, the same camera and calibration, and I do not see this flipping issue, it seems like a problem with Apriltags3.