Disable exemplars support in Spring Boot 3.2 to avoid Prometheus problem with scrapping metrics

Question

New Spring Boot added exemplars support: https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.2-Release-Notes#broader-exemplar-support-in-micrometer-112

After upgrading to Spring Boot 3.2.0 Prometheus stopped scrapping a metrics from our apps, because it requires a Prometheus version 2.43.0 or newer:

metric name web_client_requests_seconds_count does not support exemplars

I think there should be a way to disable this until we won't upgrade a Prometheus.

Answer 1

I think this is because you are using an unsupported version of Prometheus server. You will need this change in Prometheus: Exemplars support for all time series #11982. It was introduced in Prometheus 2.43.0. Also see discussion about this: Add exemplar support to _count #3996.

According to the Prometheus support terms, you should use at least 2.45.

If you really want to hack this (please don't and use a supported version of Prometheus instead), I think if you create a SpanContextSupplier @Bean that always says isSampled false and it can return null for the spanId and traceId, you will not see those exemplars.

It is also possible to modify the value of the Accept header and use text/plain instead of application/openmetrics-text in a custom controller or in a gateway or reverse-proxy. This should work because the OpenMetrics format supports exemplars while the Prometheus format does not (Prometheus asks for the OpenMetrics format by default).

Answer 2

Jonatan Ivanov wrote good suggestions. I just want to publish a piece of code that makes it possible to return Prometheus metrics in the text/plain format:

@Component
@WebEndpoint(id = "prometheus-text")
@RequiredArgsConstructor
@ConditionalOnAvailableEndpoint(endpoint = PrometheusPlainScrapeEndpoint.class)
@ConditionalOnEnabledMetricsExport("prometheus")
public class PrometheusPlainScrapeEndpoint {

    private final PrometheusScrapeEndpoint prometheusScrapeEndpoint;

    @ReadOperation(produces = "text/plain")
    public WebEndpointResponse<String> scrape(@Nullable Set<String> includedNames) {
        return prometheusScrapeEndpoint.scrape(TextOutputFormat.CONTENT_TYPE_004, includedNames);
    }

}

Then add prometheus-text in management.endpoints.web.exposure.include property of application.properties (or yaml) file. It would be like management.endpoints.web.exposure.include=healt,prometheus,prometheus-text or just management.endpoints.web.exposure.include=*.

After all changes don't forget to change the scrape endpoint in your Prometheus config file to /actuator/prometheus-text.

Answer 3

Another variant of solving the problem is overriding the ExemplarSampler bean provided by Spring Boot auto-configuration with an implementation that disables exemplars only for counter metrics (histogram metrics will still have exemplars attached).

There are 2 simple ways of doing this.

1. You can define ExemplarSampler bean, that will substitue the default DefaultExemplarSampler implementation. The implementation could just delegate calls to all methods except io.prometheus.client.exemplars.CounterExemplarSampler.sample to the DefaultExemplarSampler instance, and CounterExemplarSampler.sample should just return null.

2. You can define a BeanPostProcessor, that does basically the same, but does not rely on the DefaultExemplarSampler implementation. It will just wrap any found ExemplarSampler bean into the new implementation.

An example of such BeanPostProcessor:

/**
 * A {@link BeanPostProcessor} that wraps any {@link ExemplarSampler} bean with an implementation
 * that delegates all method calls to the wrapped bean, except
 * {@link io.prometheus.client.exemplars.CounterExemplarSampler#sample(double, Exemplar)}.
 * This have an effect of disabled Exemplars for counter metrics, introduced in Prometheus 2.43.
 * Any counter metrics with exemplars are ignored by any older Prometheus version on scrapping,
 * so it is important for counter metrics to <b>not</b> have exemplars, or they will be ignored,
 * if scrapped by Prometheus version less than 2.43.
 *
 * @see <a href="https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.2-Release-Notes#broader-exemplar-support-in-micrometer-112">Spring Boot 3.2 Release Notes</a>
 * @see <a href="https://github.com/GoogleCloudPlatform/prometheus-engine/issues/812">Managed Prometheus on GCP Issue</a>
 * @see <a href="https://stackoverflow.com/questions/77586575/disable-exemplars-support-in-spring-boot-3-2-to-avoid-prometheus-problem-with-sc">Stackoverflow discussion</a>
 * @see <a href="https://github.com/micrometer-metrics/micrometer/pull/3996">Micrometer Metrics PR</a>
 */
@Component
public class ExemplarSamplerBeanPostProcessor implements BeanPostProcessor {

    @Override
    public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
        if (bean instanceof ExemplarSampler delegate) {
            return new IgnoreCountersExemplarSampler(delegate);
        }
        return bean;
    }

    @RequiredArgsConstructor
    static class IgnoreCountersExemplarSampler implements ExemplarSampler {

        private final ExemplarSampler delegate;

        @Override
        public Exemplar sample(double increment, Exemplar previous) {
            // Do not return exemplar for counter metrics to allow them to be scrapped by Prometheus 2.42 and below
            return null;
        }

        @Override
        public Exemplar sample(double value, double bucketFrom, double bucketTo, Exemplar previous) {
            return delegate.sample(value, bucketFrom, bucketTo, previous);
        }

    }

}

Put this component into a component-scanned package and that's all. Counter metrics will not include exemplars anymore and Prometheus older than 2.43 will successfully scrap them.

If you do not want to use BeanPostProcessor, here is the solution with @Bean override:

@Configuration
@ConditionalOnEnabledMetricsExport("prometheus")
public class PrometheusExemplarsConfig {

    @Bean
    IgnoreCountersExemplarSampler exemplarSampler(SpanContextSupplier spanContextSupplier) {
        ExemplarSampler delegate = new DefaultExemplarSampler(spanContextSupplier);
        return new IgnoreCountersExemplarSampler(delegate);
    }

}

The implementation of IgnoreCountersExemplarSampler is the same.