I wanted to create a Grafana dashboard and an alarm on top of that for calculating Average Active concurrent connections per GPU/Node with a specific threshold value for the alarm. There is a Load Balancer in place which distributes the traffic on the nodes(which are dynamic) because of autoscaling. I was looking for the method/metrics to get the average connection count per GPU/Nodes.
I created a Grafana Dashboard with "Sum" as statistics and "ActiveConnectionCount" as metric and got the sum of Active connections count for a particular Load balancer. But as i need to setup a threshold value for the alarm and the calculation/formulae to calculate the threshold value includes the value "no of nodes" which is dynamic in this scenario hence looking for a technique on how to perform these calculations with such dynamic values of Node.
Another option i am trying was to calculate the average concurrent connections per GPU/Node where the threshold calculation for the alarm will not require this dynamic nature/value of Nodes. So just wanted to know if this can be achieved and how.