How to smooth jagged edges of an image from detectron2 (rcnn) output

Question

We trained a model using Detectron2 for image segmentation. After getting the model outputs, I'm using the mask data to crop out the detected object from the original image. But here’s the hitch - the edges of the mask are pretty jagged, which makes the edges of the cropped image jagged too.

I'm looking to clean up the cropped image in post-processing rather than going back and retraining the model. Since I'm adding padding to the cropped image by dilating it, we can't just use Gaussian blur to smooth out the edges.

I tried getting the moving averages (code below), but because the jaggedness is so fine-scale, it didn’t really help with the jaggedness and just made the image less wavy. Any suggestion to solve this problem would be highly appreciated.

def moving_average(points, window_size=20):
    """Calculate moving average on the contour points."""
    extended = np.vstack([points[-window_size//2:], points, points[:window_size//2]])
    smoothed_points = []
    
    for i in range(len(points)):
        smoothed_points.append(np.mean(extended[i:i+window_size], axis=0))
    
    return np.array(smoothed_points).astype(np.int32)


mask_image = (item_mask * 255).astype('uint8')

contours, _ = cv2.findContours(mask_image, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

smoothed_contours = []
for contour in contours:
  smoothed_contour = moving_average(contour.squeeze())
  smoothed_contours.append(smoothed_contour.reshape(-1, 1, 2))

smoothed_mask = np.zeros_like(mask_image)
cv2.drawContours(smoothed_mask, smoothed_contours, -1, (255), thickness=cv2.FILLED)

cv2_imshow(smoothed_mask)
print(smoothed_mask)