How to Override Django ManyRelatedManager Methods and Enforce Validations

Question

I have the following two models that have a Many-to-Many (m2m) relationship. I want to implement validation rules when creating a relationship, particularly when the Field object is controlled.

class Category(models.Model):
    name = models.CharField(max_length=50, unique=True)
    slug = models.SlugField(max_length=50, unique=True, editable=False)
    fields = models.ManyToManyField("Field", related_name="categories")

class Field(models.Model):
    controller = models.ForeignKey(
        "self",
        models.CASCADE,
        related_name="dependents",
        null=True,
        blank=True,
    )
    label = models.CharField(max_length=50)
    name = models.CharField(max_length=50)

When creating a relationship between a Category and a Field objects via the manager's add() method, I would like to validate that, for a controlled field, the controller field must already be associated with the category.

When creating via the set() method, I would like to validate for any controlled field in the list, the controller field must also be in the list, OR is already associated with the category.

# Category
category = Category.objects.create(name="Cars", slug="cars")

# Fields
brand_field = Field.objects.create(name="brand", label="Brand")
model_field = Field.objects.create(name="model", label="Model", controller=brand_field)

# controller should already be associated or raises an exception
category.fields.add(model_field)

# Controller must be inclded in the list as well,
# OR already associated, or raise an exception
category.fields.set([model_field])

Where can I enforce this constraints? I'm thinking of overriding ManyRelatedManager's add() and set() methods but I do not understand how they're coupled to Django model or field.

Alternatively

If it's not viable to override the ManyRelatedManager methods, where in the ModelAdmin can I intercept objects before they're saved so that I can run these checks?