Micrometer metrics in multiple instances of springboot application

Question

I have a custom micrometer metrics in spring boot application configured with Prometheus which scrapes the metrics every 15s.

The custom metrics is querying the db every 1 min. As I have 2 instances of this service running, both the instances tries to run the same query every 1 minute.

package com.test;

import com.entity.Foo;
import com.repo.FooRepository;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.function.Supplier;

@Component
public class MonitoringService {
    private final MeterRegistry meterRegistry;
    private final Gauge fooCount;
    private final FooRepository<Foo> fooRepository;

    
    @Autowired
    public MonitoringService(final FooRepository<Foo> fooRepository,
                             final MeterRegistry meterRegistry) {
        this.fooRepository = fooRepository;
        this.meterRegistry = meterRegistry;
        fooCount = Gauge.builder("foo_count", checkFooCount())
                .description("Number of foo count")
                .register(meterRegistry);
    }

    @Scheduled(fixedDelayString = "PT1M", initialDelayString = "PT1M")
    public Supplier<Number> checkFooCount() {
        return ()-> fooRepository.getTotalFooCount();
    }

}

Is there anyway I can configure to run this metrics in any 1 instance of my spring boot application?

Answer 1

Depending of your runtime environment you could either use Quartz with a persistent job store and clustering option or use a single App to get this job done. For the latter you might want to use something like Kubernetes Jobs.

• For Quartz see Configure Clustering with JDBC-JobStore.
• For Kubernetes Jobs see Kubernetes documentation for Jobs.

Quartz can schedule the job on just one instance, while all instances are equally configured. The Kubernetes job would be a separate pod next to your application that does just this task. A kubernetes job would start, fetch the data and end itself. (You could set some delay to allow the data to be collected by prometheus before the pod stops.) Both job frameworks work with cron-like configuration.

But wouldn't it be a better solution to use database tools instead of some Java application for counting database rows:

• With mysql-exporter, you'll get at least some table statistics, see info_schema_tablestats.go.
• For custom queries see Have Prometheus send a SQL Query.

Answer 2

Sharing the metric's query result among the instances will most likely require your services to interact with a third party that'll store the latest result. That might be a special table in a DB, a distributed configuration server like etcd or Zookeeper or a distributed cache like Apache Ignite. Obviously anything of this will add to complexity of the system and require more effort to setup and maintain then just letting all the instances run a counting query.

On the other hand you can try to optimize this query itself if it's really slow. For example if insertions to the queried table are relatively seldom you could setup an aggregation on the DB side - with each write to this table also update the total count in a separate table. This way your services will run a simple query that returns a single result of the already prepared total count.