I have two Kubernetes GCP clusters, and I am using Prometheus operator.
I am using Google Private Service Connect (PSC) to connect both clusters, and this part is working. Prometheus, which is running in one of the clusters, can send alerts to Alert Manager, which is working in the other cluster, using PSC.
If both Alert Manager and Prometheus were in the same cluster, I could run 3 Alert Manager instances, and point all of them from Prometheus. In this way, any alert would be sent to all three Alert managers, achieving redundancy.
With the setup I am describing, I configure Prometheus to send alerts to the PSC address of the Alert Manager in the other cluster. Then I can set up three Alert Manager in the other cluster, and they "clusterize", they recognize each other, but the alert is delivered to only single one Alert manager, not all of them. I would like all Alert Managers to get the same alert.
The three Alert Managers are behind a Load Balancer, so my first approach was to configure the Load Balancer to "do not balance" but send the same request to all the target. After reading people with similar issues ALB forward traffic to two target groups (IP & Lambda) at the same time it looks like is not possible for a Load Balancer to send traffic to all the targets (as far as I know).
Another approach that I investigate was using GCP Packet mirroring, but it is said clearly on the documentation that
Packet Mirroring can't mirror packets for Private Service Connect published services traffic.
So the last solution that I can figure out is having three different PSC, one for each different Alert Manager. I do not like much this solution, since it is hard to scale (you need to add/remove PSCs when adding/removing Alert Manager), but as far as I see, is the only one that can work with some kind of "high availability".
Does anyone have a better solution or idea?
Thanks!