I am doing a project in medical image segmentation.

For example, I have the segmentation mask for the darker region circled in red (already have a NN trained for its segmentation). I would like to segment the lighter region that's surrounding it (circled in green). I prefer not to train a separate NN for that task and in best case scenario not to use DL models at all. Training a new NN is not preferred as it would involve annotating many medical images and DL model is slightly less preferred because of inference speed but I am still ready to explore using pretrained models for segmentation.
Is they a way to leverage the fact that I have darker region segmented and that lighter region surrounds it?