traefik Ingress Controller wont recognize Ingress Class in multi controller env

Question

I have a cluster with two ingress controllers running. Haproxy and traefik. I am migrating away from haproxy. So I need them to run side by side differentiated by ingress class name.

Haproxy is running as expected and respecting the class name. Traefik is not. I am using the ingress manifest not ingress route or gateway. When I deploy an ingress manifest for traefik everything seems fine with the exception that it doesn't get assigned the LoadBalancers ip.

Here is how traefik is configured:

containers:
        - name: traefik
          image: docker.io/traefik:v2.11.0
          args:
            - '--global.checknewversion'
            - '--global.sendanonymoususage'
            - '--entrypoints.metrics.address=:9100/tcp'
            - '--entrypoints.traefik.address=:9000/tcp'
            - '--entrypoints.web.address=:8000/tcp'
            - '--entrypoints.websecure.address=:8443/tcp'
            - '--api.dashboard=true'
            - '--ping=true'
            - '--metrics.prometheus=true'
            - '--metrics.prometheus.entrypoint=metrics'
            - '--providers.kubernetescrd'
            - '--providers.kubernetesingress'
            - '--providers.kubernetesingress.ingressclass=traefik'
            - '--entrypoints.web.http.redirections.entryPoint.to=:443'
            - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
            - '--entrypoints.websecure.http.tls=true'
            - '--log.level=DEBUG'
            - '--accesslog=true'
            - '--accesslog.fields.defaultmode=keep'
            - '--accesslog.fields.headers.defaultmode=drop'

Here is how I define the ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-tfk
  namespace: fvtt
  labels:
    ingress: demo
    version: 1.0.0
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.priority: '20'
    traefik.ingress.kubernetes.io/router.tls: 'true'
    traefik.ingress.kubernetes.io/router.tls.domains.0.main: college.net
    traefik.ingress.kubernetes.io/router.tls.domains.0.sans: demo.college.net
  
spec:
  ingressClassName: traefik
  tls:
    - secretName: college.net-tls
  rules:
    - host: demo.college.net
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo
                port:
                  number: 80

traefik is being deployed via helm. traefik version 2.11. I know I am missing something for the life of me I can't figure it out.