PyTorch Camera Projection with Distortion

Question

So I've got some calibrated cameras and I want to implement opencv's projectPoints() and fisheye.projectPoints() in pytorch, so that I can backpropagate through the operations performed. I've done so mostly, but it's ever so slightly off (image linked below). I have 3 fisheye cameras and one normal camera. The normal camera has next to no distortion, so I'm not worried about that. For the fisheye distortion I'm following what's laid out here: https://docs.opencv.org/3.4/db/d58/group__calib3d__fisheye.html

Image: The third pair of rows is opencv's projection of the points while the second pair of rows is my projection. The points being projected are the middle of the arena on the floor, the 4 points in either direction from that point whose offset distance is the radius of the inner cylinder, and one pint directly above the center point a ways up.

Here's my code (everything is within a class Camera that has correct getters and setters for each relevant attribute seen below):

def distort(self, points):
    x = points[:, 0]
    y = points[:, 1]
    r = torch.sqrt(torch.square(x) + torch.square(y))
    theta = torch.atan(r)
    theta2 = theta**2
    theta4 = theta2**2
    theta6 = theta2**3
    theta8 = theta4**2
    D_mx = self.get_D()
    theta_d = theta * (1 + 
                       (D_mx[0].item() * theta2) +
                       (D_mx[1].item() * theta4) +
                       (D_mx[2].item() * theta6) +
                       (D_mx[3].item() * theta8)
                       )
    x_dist = (theta_d / r) * x
    y_dist = (theta_d / r) * y
    dist_pts = torch.stack((x_dist,y_dist), dim=1)
    return dist_pts
    
def project_world(self, points, distort=None):
    utils.check_tensor(points, shape=[points.shape[0],3], dtype=torch.double)
    points = points.view(-1,3,1)
    camera_points = torch.matmul(self.get_R_mx(), points).view(-1,3) + self.get_T_vec().view(3)
    return self.project(camera_points, distort=distort)
        
def project(self, points, distort=None):
    utils.check_tensor(points, shape=[points.shape[0], 3])
    points = points[:, :2] / points[:,2:3]
    if distort is not None:
        points = self.distort(points)
    elif self.should_distort:
        points = self.distort(points)
    points = points.view(-1, 2, 1)
    projected = torch.matmul(self.get_K()[:2,:2], points).view(-1, 2) + self.get_K()[:2,2]
    utils.check_tensor(projected, shape=[projected.shape[0],2])
    return projected

I've tried rewriting it a whole lot. I've checked that I gave it everything in the correct order. It seems to be so close, I'm sure there's gotta be some little thing I'm doing wrong. If not, is it possible that it's something analogous to a rounding difference?

Edit: Added some stuff in comments and want to add it here too.

The difference is very minimal. In fact, I didn't notice it until I started running bundle adjustment on the extrinsic camera matrices to calibrate them. If you look closely, you'll see that the 5 points that make a cross of the bottom row (opencv's projection) are seemingly stretched out compared to the second row (like you shift clicked every point and held shift while expanding it in google docs, if that makes sense).

Here's a github link to a minimal runnable example

If you run that code you'll get only 3 rows of images. The first row is the blank image that I projected onto. The second shows my functions' projections onto those images, while the third shows opencv's projections onto the same images. The second pic of each row (after the first row) is the projection of the top camera. That seems to be fine, and I'm not distorting for that one because it's not a fisheye which leads me to believe the error is related to my distortion function. The error I'm worried about is between the 3 other pics of the second row (my projection) and the 3 other pics of the third row (open cv's projection). These 3 other images are for the fisheye cameras. Sorry if I'm being too verbose and rambling, I want to make it super clear.

P.S. I do distort when I do open cv's projection for the second image, but as you can see the distortion is so minimal it doesn't matter.

Also, I know the cameras aren't well calibrated yet. But I don't think that could change the result of projecting if the functions used to project performed identical operations, but maybe not?

I just tested changing everything to 32 bit floats instead of 64 bit, and that didn't change anything.