How to check pod with an IP is actually alive from another pod?

Question

We deployed a Java application with metric job that waits for client applications in the same namespace to push their metrics with http endpoints.We needed some way to uniquely identify each client's metrics. So we attached pod ip addresses to client's metrics and stored them away to report later. What we are struggling to solve now is, because pods are not stateful and at some point pods go unavailable or die and they restart with a new IP address. That means we should dispose stale metric of old pod that was identified by a stale pod IP. We want to create a scheduled job in our metrics application to traverse all metrics and check IP addresses attached to them and dispose stale ones which don't point to any available pod. Is there a way I can query if a pod with an IP address is available or exists from another pod?

We don't know actually what kind of a client pushes metrics to the metric endpoints. There is a wide range of applications deployed in same namespace from nodejs to Java applications. So we can't scrape them for metrics. Kubernetes Client Library is not a viable option by now because we are in an environment that we are forced to avoid making HTTP calls to kubernetes api server.

Answer 1

I needed a simple way to know if an IP address that was attached to a metric stored in my metrics service written in Java , can be resolved to an alive pod.Luckily ,it was easy programmatically in Java .

Example I had a metric for transaction counts flowed from one of our client's service. with cluster ip trx_count{app="10-187-79-75-frictionless-symphony-service"} Just parsed 10-187-79-75 to a correct IP representation like 10.187.79.75 and then in our metrics service just wrote the code below

    public String getHostName(String ipAddr) throws UnknownHostException {
    return Inet4Address.getByName(ipAddr).getHostName();
}

if pod for that IP is not available ,returned string from getHostName method is not a hostname but exactly same IP address that was passed as argument.

private boolean isMetricStale(){
var hostName = getHostName(podIp);
if (podIp.equals(hostName))
       return true;
else
       return false;
}